In Differential Privacy, There is Truth: On Vote Leakage in Ensemble Private Learning
Abstract
When learning from sensitive data, care must be taken to ensure that training algorithms address privacy concerns.
The canonical Private Aggregation of Teacher
Ensembles, or PATE, computes output labels by aggregating the predictions of a(possibly distributed) collection of teacher models via a voting mechanism.
Themechanism adds noise to attain a differential privacy guarantee with respect tothe teachers’ training data.
In this work, we observe that this use of noise, whichmakes PATE predictions stochastic, enables new forms of leakage of sensitive information.
For a given input, our adversary exploits this stochasticity to extracthigh-ﬁdelity histograms of the votes submitted by the underlying teachers.
Fromthese histograms, the adversary can learn sensitive attributes of the input such asrace, gender, or age.
Although this attack does not directly violate the differentialprivacy guarantee, it clearly violates privacy norms and expectations, and wouldnot be possible at all without the noise inserted to obtain differential privacy.
Infact, counter-intuitively, the attack becomes easier as we add more noise to provide stronger differential privacy.
We hope this encourages future work to considerprivacy holistically rather than treat differential privacy as a panacea.
Introduction
The canonical Private Aggregation of Teacher Ensembles, PATE, is a model-agnostic approach toobtaining differential privacy guarantees for the training data of ML models [1], [2], that is widelyapplied [3], [4] and adapted [5]–[7] due to its comparatively favorable trade-off between differentialprivacy, utility and ease of decentralization [6].
In PATE, one considers an ensemble of independently trained teacher models.
To generate a prediction, PATE ﬁrst collects the predictions of theseteachers to form a histogram of votes.
It then adds Gaussian noise to the histogram and only revealsthe label achieving plurality.
This label can be used directly as a prediction, or to supervise thetraining of a student model—in a form of knowledge transfer.
Because PATE only reveals the labelreceiving the most votes, it comes with guarantees of differential privacy, i.e., the noisy voting mechanism allows us to bound how much information from the training data is potentially exposed [8].
But PATE does not explicitly protect from leakage of a key element in its inference procedure: thehistogram of votes submitted by teachers.
While the histogram is used internally and not directlyexposed to clients, a careful examination of PATE reveals that information about the histogram leaksto clients via query answers.
The histogram can contain highly sensitive information, not the least of which is membership inminority groups which, if revealed, can be used to discriminate against individuals.
We demonstratethis by showing how attacker, using the vote histogram of a PATE ensemble trained to predict anindividual’s income, can infer wholly different attributes such as their level of education, even whenthe attacker’s instance does not contain any information related to education-level.
Why is thispossible?
At a high level, the histogram of votes can be interpreted as a relatively rich representationof the instance, that reveals attributes beyond what the ensemble was designed to predict
Next, we ask, is this attack a realistic threat?
We answer this in the afﬁrmative by designing anattack that extracts PATE histograms by repeatedly querying PATE, and showing that it reconstructsinternal histograms to near perfection.
Our attack builds on the fact that repeated executions ofthe same query produce the same internal histogram and a consistent distribution of PATE’s noisedanswers corresponding to this histogram.
Our adversary can thus sample this distribution manytimes via querying, and use it to reconstruct the histogram.
This implies that our attack relies on the stochasticity of PATE’s output, which is a product of Gaussian noise, the very mechanism that was intended to protect privacy.
In fact, we ﬁnd that the largerthe variance of noise added to the histogram votes, the more successful our adversary is in reconstructing the histogram.
This is in sharp contrast with the known and expected effect in differentialprivacy, that higher noise scale generally leads to stronger privacy.
Put simply: differential privacymakes our attack possible.
An astute reader may observe that histogram leakage does not violate the differential privacy guarantee, which only protects individual users in the training data, which is not compromised here.
While it is absolutely true that our attack does not violate differential privacy, it clearly violatessocietal norms and user expectations that differential privacy is often incorrectly assumed to protect.
The fact that differential privacy enables the leakage we exploit nicely underscores the distinctionbetween technical deﬁnitions of privacy and common conceptions of privacy.
The attack is difﬁcult to mitigate.
Particularly, we show that it is stealthy in the sense that PATE’sown accounting of “privacy cost” considers our attacker’s set of queries “cheap”, meaning that revealing their answers has a relatively small effect in terms of differential privacy.
Consequently,
PATE’s privacy-spending monitoring does not prevent our attack.
Our attack also performs onlya moderate number of queries in absolute numbers, the same number used by common legitimate
PATE clients, so a hard limit on queries would impede PATE’s utility.
We will discuss other mitigation approraches, which are not robust and/or not always usable.
To summarize, our contributions are as follows:
• We posit the novel threat of extracting PATE’s internal vote histograms.
We observe and show that those contain sensitive information such as minority-group membership.
• We show that differential privacy is the cause for histogram-information leakage to PATE’squerying clients.
• We exploit this leakage to reconstruct the vote histogram.
We achieve this by minimizing the difference between (a) the probability distribution of outcomes observed by repeatedly querying PATE and (b) an analytical counterpart that we derive.
• We experiment with standard PATE benchmarks, showing that the attack can recover high- ﬁdelity histograms while using a low number of queries that remain well within PATE’s budget intended to control leakage.
Vote Histograms are Sensitive Information
We consider an ensemble’s vote histogram, such as those computed internally in PATE.
Clearly,such histograms contain a lot more information on PATE’s innerworkings than simply its revealeddecision, but it is important to clarify that there are common contexts in which this leakage canactually be used to hurt individuals as they contain sensitive information about them.
As a prominent example, minority-group membership often leaks via histograms, and can of coursebe used to discriminate against group members.
To understand this, let’s consider a minority groupthat is under-represented in the training data distributed across PATE’s teachers.
Each teacher observes some outliers and mis-representitive phenomena such as coincidental correlations or out-ofdistribution examples.
When data on group members is scarce, each model will tend to over-ﬁt tothe outlier phenomena within its own data, creating inter-model inconsistencies and resulting in disagreement, or low consensus, when predicting on similar inputs at test time—which readily presentsitself on vote histograms.
Thus, we expect histograms to reveal members of minority group members
Extracting sensitive attributes from UCI Adult-Income histograms.
We now simulate an attack that receives the vote histogram of a salary-predictor ensemble and uses it to detect a smallminority of the population, speciﬁcally, PhD holders.
Following the above observation, our attack will simply classify all highly-consensus (consensus > 75% ) predictions as non-PhD-holders,whereas low-consensus ( < 75% ) predictions will be classiﬁed as PhD holders.
This is a heuristicattack that relies on intuition rather than learning the ensemble’s behavior using a labeled dataset.
On one hand, it may underestimate the attacker’s ability to detect PhD holders; on the other hand, itdoes not require a labeled dataset and only assumes that the attacker sees the votes histogram.
We use UCI Adult-Income dataset [9], containing around 41000 data points with basic personalinformation on people such as age, work hours, weight, education, marital status, and more.
PhDholders form about 1% of this dataset.
We randomly selected 80% of the dataset for training, andheld out the rest for testing.
We randomly partitioned the data into 20 disjoint sets.
For each, we ﬁtteda random forest model (using a hyperparameter grid search, see Appendix D) predicting whetherincome is above or below $50000.
For test data, we removed from the data columns includingeducation levels: that is, test individuals do not contain any feature that directly distinguishes PhDfrom non-PhD holders.
Figure 1 shows the distribution of high-consensus andlow-consensus on the test set (to make the effect clearer,we balanced the minority and majority groups in the testset by randomly removing most of the non-PhD samples).
We observe that low consensus indeed indicates minoritygroup membership.
Our attacker’s precision is not particularly high (75% on the balanced set), but they can stilluse this signal to discriminate against minority groups.
Other attacks.
More sophisticated attackers can lookfor distinctive histogram patterns that characterize certaingroups; the attack should become more accurate as moremodels are added to the ensemble, reﬁning the attacker’shistogram measurement; and precision can be ampliﬁedif the attacker holds multiple samples that are known tobelong to the same group.
Further, we note that sensitiveattribute extraction is not the only example for when votehistograms leak sensitive information: an attack could use votes to try to infer dataset properties [10]or distinguish between different partitions of the data associated with the different teachers in theensemble.
How to Extract PATE Histograms
Having established that vote histogram leakage poses a risk to privacy and fairness, we proceed toprovide a generic method for extracting vote histograms from PATE.
3.1 Problem Formulation and Attack Model
A primer on PATE.
The PATE framework begins by independently training an ensemble of models, called teachers, on partitions of the private data.
There is no particular requirement for thetraining procedure of each of these teacher models; the only constraint is that the partitions be disjoint.
Queries made by clients are answered as follows: (1) each teacher model predicts a label onthe instance, (2) the PATE aggregator builds a histogram of class votes submitted by teachers, (3)
Gaussian noise is added to this histogram, and (4) the client receives the noised histogram’s argmaxclass (henceforth result class ).
This noisy voting mechanism gives labels predicted by PATE differential privacy guarantee, in what is an application of the Gaussian mechanism [11].
To preserve differential privacy, PATE tracks the privacy cost of the set of past queries, and stopsanswering queries once the cost surpasses the privacy budget .
The cost computation is parameterizedby a size  .
The key differential privacy guarantee of PATE can be stated as follows: for a given setof queries with cost " , PATE is ( " ,  ) differentially private.
Put succinctly, " bounds an adversary’sability to distinguish between any adjacent training datasets, whereas  bounds the (usually small) probability, over PATE’s randomness, of this bound not holding.
We defer additional details ondifferential privacy in PATE to the appendix.
[Table]
[Figure]
Attacker’s motivation.
Our attacker’s goal is torecover the histogram of the vote counts when PATElabels an instance.
Formally, given N predictors { P 1 , ..., P N } and a target input a , our attackerwants to infer H ⌘ Count ( P 1 ( a ) , . .
P N ( a )) =[
h 1 , . .
, h m ] where Count counts the number of appearances of each element in [ m ] .
Vote histogramscan be used to extract potentially-sensitive information about an instance, such as its race, gender, or religion (see Section 2).
Attacker’s access and knowledge.
Our attackercan send queries to the aggregator and receive thelabel predicted by PATE (i.e., the output of the noisyvoting mechanism).
This may be possible becausethe aggregator willfully exposes the predictions of
PATE, e.g., through a MLaaS API.
Alternatively,fully-decentralized implementations of PATE havebeen proposed where the central aggregator is replaced with a cryptographic multi-party computationprotocol [6], and its output is exposed directly.
Figure 2 visualizes the workﬂow of our attack.
Input: 1: N 2 N .
total number of teachers (see Section 3.1 for why this is needed) 2: O . PATE instance 3: T,  . optimization termination threshold and learning rate
Output: ˆ H 4: S sample ( O , M ) .
sampling PATE M times and storing into S 2 1 , ..., K M 5: for i = 1 , 2 , . .
, M do 6: for j = 1 , 2 , . . .
, m do 7: q ij = int ( S i == j ) .
q ij = 1 if
S i =
j , 0 otherwise 8: end for 9: end for 10: ¯ q 0 m .
initialization of ¯ q with all 0’s ﬁrst 11: for i = 1 , 2 , . .
. , m do 12: ¯ q [ i ] 1 M
P
Mj =1 q ji 13: end for 14: ˆ H 0 m .
initialization of ˆ H , here we use an all-zero array of length m 15: while
Q ( ˆ H )  ¯ q    2 > T do 16:
ˆ H
ˆ H   r ˆ H
Q ( ˆ H )  ¯ q    2 17: end
while 18: ˆ H
ˆ H + [ N
P
ˆ H m ] m .
shift ˆ H to sum to N return ˆ H
In PATE, the parameters (mean, variance) ofthe noise added during aggregation are public domain [2]; we therefore assume the attacker knows them.
We also assume the attacker knows the number of teacher models N ,which may or may not be public.
This assumption is only necessary to shift the attacker’slearned distribution by a constant to attain alow L 1 approximation error when reconstructing histograms (Section 3).
We note that theattacker could just as easily exploit the leakage(e.g. to learn sensitive attributes or differentiatebetween training sets) without it but we choseto instead make this assumption to simplify result presentation and interpretation.
3.2 Our histogram reconstruction attack
The idea behind the attack, given in pseudocode in Algorithm 2, is as follows: let Q bea function that computes output-class probability distribution of PATE given a vote histogram H .
First, our attacker will sample PATE to ﬁndan estimate for this distribution ¯ q ⇡ Q ( H ) .
Second, the attacker will use gradient descentto ﬁnd ˆ H that minimizes the Euclidean distance between Q ( ˆ H ) and ¯ q .
Finally, they shiftthe estimated histogram ˆ H by a constant to account for the number of teachers (this step assumes the number of teachers is known, but isdone mostly for presentation purposes, see Section 3.1).
We now detail these 3 steps.
Step 1: Monte Carlo approximation.
The ﬁrst step will sample PATE M times and estimate thedistribution over PATE’s outputs ¯ q ⇡ Q ( H ) by setting each class probability as its Monte Carloestimated mean frequency, i.e. ¯ q i 1 M
P Mj =1 q ji where q ji indicates whether class i was sampled inthe j th step.
By the law of large numbers, as M increases, ¯ q i converges to i ’s sampling probability[ Q ( H )]
i , and we can expect the attacker’s estimate produced in the next steps to be more accurate.
Our attacker would want to increase M as much as possible, until they exceed PATE’s privacybudget.
Indeed, in PATE, the privacy leakage expended by each individual query can then be composed overmultiple queries to obtain the total privacy cost " needed to answer the set of queries.
Once the totalprivacy cost " exceeds a a maximum tolerable privacy budget, PATE must stop answering queries topreserve differential privacy.
Section 4 shows that the attack succeeds for values of M that remainwell below PATE’s privacy budget, and are also moderate in absolute value, as they are similar tothe query number of student models that use PATE.
Step 2: constructing the optimization objective.
Our attacker wants to ﬁnd ˆ H such that
Q
[-▷ initialization of ˆ H , here we use an all-zero array of length c 15: while (cid:13)(cid:13)(cid:13) Q ( ˆ H ) − ¯ q (cid:13)(cid:13)(cid:13) 
 > T do 16: ˆ H ← ˆ H − λ ∇ ˆ H (cid:13)(cid:13)(cid:13) Q ( ˆ H ) − ¯ q (cid:13)(cid:13)(cid:13)17: end while 18: ˆ H ←-][Equation]
ˆ
[Equation]
¯
[Equation]
2 is minimized where k · k 2 denotes Euclidean norm.
Given a (differentiable) closedform expression for Q , it becomes natural to program and solve this with modern gradient-basedoptimization frameworks.
Theorem 1 provides a closed form expression; and our attacker will use adifferentiable approximation of this expression, as explained below.
Theorem 1.
Let H =
[ H 1 , . . .
, H m ] be the vote histogram for the m classes, and let PATE’s
Gaussian-mechanism function Agg ( H ) ⌘ argmax { H i + S i } where S =
[ S 1 , . . .
, S m ] is a vec- 212 tor of m samples from a zero-mean normal distribution with variance  2 .
Then the probability that the randomized aggregator outputs the class k is given by [ Q ( H )]
k = P ( Agg ( H ) = k ) = R 1  [Equation]
[Equation]
6 = k i =
i ( ↵ )  k ( ↵ )
d ↵ where  i ( · ) is the cumulative probability distribution (CDF)of N ( H i ,  2 ) (normal distribution with mean H i and variance  ) and  k ( · ) is the probability density function (PDF) of N ( H k ,  2 ) .
Proof.
Q ( H ) =
P ( Agg ( H ) = k ) , is the probability that H k + S k = max { Agg ( H ) } .
For any k , 218 H k + S k is a random variable that follows a normal distribution with mean equal to H k and varianceequal to  2 .
Let g k
= H k
+ S i , then g k ⇠ N ( H k ,  2 ) .
[ Q ( H )] k is the probability of g k is greaterthan g j , 8 j 2 { 1 , . . .
k  1 , k + 1 , . . .
m }
[Equation]
The expression in Theorem 1 is not usable in automatic differentiation and optimization frameworks; we therefore use an approximation of the integral by the trapezoid formula.
We select points with higher probability and sum up their values to get an approximation of the integral with inﬁnite bounds.
Then we decide what values to select.
In the integral R [Equation]
↵ [Equation]
↵
[Equation]
↵ is the value of g k ⇠ N ( H k ,  2 ) .
Therefore ↵ has the highest probability at H k , and has the higher probability closer to H k .
More speciﬁcally, properties of the normal distribution give us that µ ± 6 ⇤  covers 99% of the values of Gaussian random variable z ⇠ N ( µ,  ) .
Therefore values of ↵ between
{+H k ± 
+}
⇤  cover 99% of the integral area.
Therefore,
[Equation]
which is differentiable and is handled well by most automatic differentiation packages.
Step 3: accounting for the number of teachers.
The distribution estimate produced by our optimization may be skewed by a constant, so the attacker shifts each element of ˆ H by ( N
P ˆ H ) /mso that the new histogram ˆ H sums up to
P [Equation] H ) /m
= N .
Theorem 2 in theappendix provides proof that shifting Q ( ˆ H ) by a constant does not affect Q ( ˆ H ) .
Evaluation
We evaluate our attack against instantiations of PATE on common benchmarks.
We will show theattack extracts histograms that only slightly differ from the true ones underlying PATE’s decision,despite the fact that the privacy cost considered by PATE is low, and remains well within budgets thatare enforced in common PATE instantiations.
We also quantify how the impact of the differentialprivacy noise scale, showing that higher noise values result in increased attack success for a givenquery number.
We offer an hypothesis to explain this ostensibly surprising observation.
4.1 Experimental Setup
Data.
We use the experimental results from Papernot et al.
[1] to simulate our attack environment.
Papernot et al.
released the histograms obtained by PATE using 250 teachers for two 10-classcomputer-vision benchmarks, MNIST [12] and SVHN [13].
There are 9,000 histograms generatedby MNIST experiments and 26,032 histograms generated by SVHN experiments, corresponding tothe sizes of these datasets’ test sets.
We deﬁne a histogram’s consensus as its maximum value, and divide each dataset into three equalsized groups corresponding to high consensus, medium consensus, and low consensus.
Figure 3illustrate this.
We sample ﬁve histograms randomly from each group, and mount our attack forvarious noise levels.
Attack parameterization.
We simulated attackers with two types of query limits: ﬁrst, an attackerlimited by PATE’s canonical privacy budget; we used the parameterization from Papernot et al.
[1],i.e.
budgets of 1.97 and 4.96 for MNIST and SVHN and  = 40 .
Second, an attacker with a hardlimit of 10 4 queries; this is a moderate number of queries for clients wishing to train their own“student” model using the aggregator’s labels (see [1], [2]).
We applied this attack against PATEinstantiations for MNIST and SVHN with noise levels  2 { 40 , 60 , 80 , 100 } .
For optimization (see Section 3), we use an adaptive learning rate: at the beginning of training, weuse a learning rate of k [Equation] k 2 , where J = Q ( ˆ H )  ¯ q is the optimization objective.
As the optimization starts to converge, k [Equation] k 2 becomes too large so we switch to a learning rate of [Equation]
This 253 results in changes to the histogram of the magnitude of one vote for each update.
We use 0.01 asa threshold on the loss to establish convergence, and thus when k J k 2 < 0 .
01 , we stop optimizing.
For the attacks against canonical settings, we stopped once estimated histograms started presentingnegative values, which we found to be a slightly better strategy.
(We could also try to constrain it toonly-positive values; we discuss improving this optimization procedure further in Section 5).
Metrics.
For every attack, we measured the error rate and privacy cost .
The error rate is deﬁnedas the normalized L 1 distance between the ground-truth histogram H =
[ H i , . . .
, H m ] and ourattacker’s estimate ˆ H
=
[ ˆ H i , . . .
ˆ H m ] , i.e.,
P i  H i  ˆ H i  / (
P i | H i | ) .
Note that this can beinterpreted as the number of mis-counted votes.
We deﬁne and compute privacy cost using established practices.
At a high level (see detailsin [1]), we model PATE as a R´enyi-differentially-private mechanism and leverage known privacypreserving-composition theorems for such mechanisms; we attain (non-R´enyi) differential privacyvia a known reduction from differential privacy to R´enyi differential privacy.
The parameter  is set as 10  5 for MNIST and 10  6 for SVHN, following Papernot et al.
Implementation Our implementation is provided in Python and the optimization in Jax code,which we make available in the supplementary material.
We ran the optimization on an Intel Xeon
Processor E5-2630 v4; optimization takes about 2.5 hours.
4.2 Results
Our attack has high performance within canonical privacy budget We ﬁrst evaluate our attackon canonical PATE from Papernot et al.
Figure 4a and 4b show our attacker’s error rates forthe different histograms, averaging 0.11 on the MNIST setup and 0.05 on the SVHN setup.
Our attack extracts high-ﬁdelity histograms and has low privacy costs.
Figure 4 shows theprivacy-budget limited attack; Figures 6 and 7 show our hard-query-limit attacker’s error rate andquery costs for different noise levels, i.e. values of  .
We observe that, across attacks, we attain verylow error rates, often as low as 0.03, translating to 3% of the votes being miscounted.
For the hardquery-limit attack, privacy costs roughly range between 1 to 12, which is the order of magnitude forthe budget one would plausibly use, for example to attain guarantees similar to Papernot et al.
[2](which uses budgets of up to 8 in a directly comparable setting to ours) or Abadi et al.
[14] (whichalso employs a (8 , 10  5 ) -differentially private mechanism for MNIST).
[Figure]
Adding more noise helps the attacker.
Perhaps the most surprising result in this work is that thehigher the noise scale, the lower the attacker’s error is.
This is not necessarily aligned with using upmore of the privacy budget.
In fact, in many cases, increasing the noise decreases both the attacker’sprivacy cost and their error; Figure 5 shows the correlation between cost and error.
This is counter-intuitive, as larger Gaussian scales  usually correspond to tighter privacy guarantees, more expected protection against attacks.
Speciﬁcally, our Monte Carlo estimation should beless accurate when higher-variance noise is added, as convergence to the mean is slower.
Nevertheless, our attack actually performs better with higher noise levels.
To explain this fact, consider thedistribution on the aggregator’soutput.
When this distribution isuniform, every class gets sampledby the attack with equal probability, contributing equal informationto its Monte Carlo estimator.
Conversely, when some classes have alower probability than others, theyare seldom encountered, and theirestimator needs more samples toconverge accurately.
Sharp outputdistributions, for example, have apeak that essentially hides other classes from sampling.
To illustrate this, consider the edge casewhere no noise is added at all; here, the output is always the plurality vote, and a black-box querying adversary cannot learn anything about the histogram except its top-voted class, which is knownafter a single query.
Our results thus indicate that this “hiding” effect, which is mitigated by increasing the noise, ismore dominant than the limitation of the Monte Carlo convergence rate, which is exacerbated byincreasing the noise.
Interestingly, this is not always reﬂected in PATE’s privacy-cost score, whichoften produces lower cost scores for setups that leak more information to our attacker.
Technically,there is no contradiction here: privacy cost measures a degree of differential privacy, which does notnecessarily translates to a degree of protection against vote-histogram leakage.
Discussion
Mitigation.
The possibility of this attack is inherent to PATE’s aggregation mechanism, as long asthe attacker can make multiple queries to PATE.
Our experiments in Section 4 show that (1) usingtighter privacy budgets does not necessarily mitigate the attack, as there is no strong correspondencebetween the privacy cost and the attack’s success, and (2) it would be hard to limit the numberof queries some other way without crippling PATE’s utility, because our attack is successful whileusing the same number of queries used in common scenarios from the literature.
Theoretically, the attack would be mitigated if PATE returned a consistent answer for each query.
PATE can thus try to cache answers to past queries and not recalculate them.
Unfortunately, thisdefense would be exposed to adversarial perturbations that try to evade the caching mechanismwithout affecting predictions, and would not be possible for settings that keep queries conﬁdentialand/or include decentralized aggregation [6].
Finally, we can try to prevent sensitive information from leaking onto vote histograms.
Particularly,models that generalize well across subgroups will be more immune to an attacker inferring groupmembership via consensus.
This reduces to the problem of subgroup fairness , an active line of workwith many proposed approaches [15]–[19] but no silver-bullet solutions.
Limitations.
Empirical analysis of sensitive-attribute leakage onto vote histograms (Section 2)can be expanded to improve more sophisticated attackers, other scenarios, and also other formsof sensitive information that can leak onto histograms.
We instead focus our work on extractinghistograms from PATE, noting that this can be used as a foundation for various different attacks.
Our optimization procedure takes a noticeably long time (roughly 10 minutes for a single step and 13hours to convergence on a histogram), which prevented us from fully optimizing its hyper-parameterchoices.
This is however a limitation of our current experimental setup, not of the attack, bearingthe main consequence that we are potentially under-estimating our attack’s capabilities.
Related work.
PATE is a widely-adopted framework for differentially-private ML, with myriadapplications [3], [4] and extensions [5]–[7]; our attack is generally applicable to many of thoseframeworks, which inherit their privacy analysis from PATE.
Another prominent decentralized ML framework, Federated Learning (FL) [20], has been extensively investigated from a privacy perspective.
As we did for PATE in this work, prior work attacking
FL uncovered numerous forms of leakage.
For example, Hitaj et al.
[21] reconstructed the averagetraining set representation of each classes;
Geiping et al.
[22] reconstructed training data with highﬁdelity; Nasr et al.
[23] mounted a membership inference attack against the clients; Wang et al.[24]showed how a malicious server could distinguish multiple properties of data simultaneously; and
Melis et al.
[25] inferred the clients’ training data sensitive properties.
These prior efforts all focuson FL, and are orthogonal to ours.
We are the ﬁrst to evaluate any attack against PATE.
Conclusion We are the ﬁrst to audit the conﬁdentiality of PATE from an adversarial perspective.
Our attack extracts histograms of votes, which can reveal attributes of the input such as race orgender, or help attackers characterize teacher partitions.
The attacker’s success is not highly correlated with their queries’ privacy cost which is monitored by PATE, and mitigations of this attackare nontrivial and/or signiﬁcantly hinder prediction utility.
Particularly, using larger Gaussian noise,even when it fortiﬁes the differential privacy guarantee, actually increases risk to the conﬁdentialityof the vote histogram.
This surprising tension demonstrates that care must be taken to analyze theprotection differential privacy provides within a given threat model, rather than treat it as protectingagainst any form of leakage.
Broader Impact Our work studies information leakage in a widely-adopted system, thus promoting our understanding of its risks.
Our adversarial method can be used by developers and auditorsto evaluate the conﬁdentiality and privacy promises of PATE-based frameworks.
Our observation that differential privacy does not prevent but rather enables the attack is the ﬁrst ofits kind in that it reveals a discrepancy between differential privacy and societal norms of privacy.
Characterizing this distinction is essential to building technology that uses technical deﬁnitions ofprivacy as an instrument to protect privacy norms.
[-h 
-]
[-h 
-]