S CALABLE P RIVATE L EARNING WITH PATE
Anonymous authors
Paper under double-blind review
A BSTRACT
The rapid adoption of machine learning has increased concerns about the privacy implications of machine learning models trained on sensitive data, such as medical records or other personal information.
To address those concerns, one promising approach is Private Aggregation of Teacher Ensembles , or PATE, which transfers to a “student” model the knowledge of an ensemble of “teacher” models, with intuitive privacy provided by training teachers on disjoint data and strong privacy guaranteed by noisy aggregation of teachers’ answers.
However, PATE has so far been evaluated only on simple classiﬁcation tasks like MNIST, leaving unclear its utility when applied to larger-scale learning tasks and real-world datasets. In this work, we show how PATE can scale to learning tasks with large numbers of output classes and uncurated, imbalanced training data with errors.
For this, we introduce new noisy aggregation mechanisms for teacher ensembles that are more selective and add less noise, and prove their tighter differential-privacy guarantees. Our new mechanisms build on two insights: the chance of teacher consensus is increased by using more concentrated noise and, lacking consensus, no answer need be given to a student.
The consensus answers used are more likely to be correct, offer better intuitive privacy, and incur lower-differential privacy cost.
Our evaluation show our mechanisms to improve on the original PATE on all measures, and scale to larger tasks with both high utility and very strong privacy ( ε ≈ 1 . 0 ).
I NTRODUCTION
Many attractive applications of modern machine-learning techniques involve training models using highly sensitive data. For example, models trained on people’s personal messages or detailed medical information can offer invaluable insights into real-world language usage or the diagnoses and treatment of human diseases (McMahan et al., 2017; Liu et al., 2017).
A key challenge in such applications is to prevent models from revealing inappropriate details of the sensitive data—a nontrivial task, since models are known to implicitly memorize such details during training and also to inadvertently reveal them during inference (Zhang et al., 2017; Shokri et al., 2016). Recently, two promising, new model-training approaches have offered the hope that practical, highutility machine learning may be compatible with strong privacy-protection guarantees for sensitive training data (Abadi et al., 2017).
This paper revisits one of these approaches, Private Aggregation of Teacher Ensembles , or PATE (Papernot et al., 2017), and develops techniques that improve its scalability and practical applicability.
PATE has the advantage of being able to learn from the aggregated consensus of separate “teacher” models trained on disjoint data, in a manner that both provides intuitive privacy guarantees and is agnostic to the underlying machine-learning techniques (cf.
the other approach of differentially-private stochastic gradient descent (Abadi et al., 2016)).
The PATE approach assumes multiple teacher are trained on disjoint sensitive data (e.g., different users’ data), and uses the teachers’ aggregate, consensus answers in a black-box fashion to supervise the training of a “student” model.
By publishing only the student model (keeping the teachers private) and by adding carefully-calibrated Laplacian noise to the aggregate answers used to train the student, the original PATE work showed how to establish rigorous ( ε, δ ) differential-privacy guarantees (Papernot et al., 2017)—a gold standard of privacy (Dwork et al., 2006). However, to date, PATE has been applied to only simple tasks, like MNIST, without any realistic, larger-scale evaluation.
The techniques presented in this paper allow PATE to be applied on a larger scale to build more accurate models, in a manner that improves both on PATE’s intuitive privacy-protection due to the teachers’ independent consensus as well as its differential-privacy guarantees. As shown in our experiments, the result is a gain in privacy, utility, and practicality—an uncommon joint improvement.
[Figure] The primary technical contributions of this paper are new mechanisms for aggregating teachers’ answers that are more selective and add less noise.
On all measures, our techniques improve on the original PATE mechanisms when evaluated on the same tasks using the same datasets, as described in Section 4. Furthermore, we evaluate both variants of PATE on a new, large-scale character recognition task with 150 output classes, inspired by MNIST. The results show that PATE can be successfully utilized even to uncurated datasets—with signiﬁcant class imbalance as well as erroneous class labels—and that our new aggregation mechanisms improve both privacy and model accuracy.
To be more selective, our new mechanisms leverage some pleasant synergies between privacy and utility in PATE aggregation. For example, when teachers disagree, and there is no real consensus, the privacy cost is much higher; however, since such disagreement also implies that the teachers may give no good answer, the answer may simply be omitted.
Similarly, teachers may avoid giving an answer where the student already is conﬁdently predicting the right answer. To add less noise, our new PATE aggregation mechanisms sample Gaussian noise, since the tails of that distribution diminish far more rapidly than those of the Laplacian noise used in the original
PATE work.
This reduction greatly increases the chance that the noisy aggregation of teachers’ votes results in a correct consensus answer, which is especially important when PATE is scaled to learning tasks with large numbers of output classes. However, changing the sampled noise requires redoing the entire PATE privacy analysis, from scratch (see Section 3 and details in Appendix C).
Finally, of independent interest are the details of our evaluation extending that of the original PATE work. In particular, we ﬁnd that the virtual adversarial training (VAT) of Miyato et al.
(2017) is a good basis for semi-supervised learning on tasks with many classes, outperforming the improved
GANs by Salimans et al.
(2016) used in the original PATE work.
Furthermore, we explain how to tune the PATE approach to achieve very strong privacy ( ε ≈ 1 . 0 ) along with high utility, for our real-world character recognition learning task. This paper is structured simply, with Section 2 giving an overview of PATE, Section 3 describing our improved aggregation mechanisms, Section 4 detailing our experimental evaluation, and Sectionoffering conclusions, with further details, discussion, and proofs in the Appendices.
O VERVIEW
We introduce essential components of our approach towards a generic and ﬂexible framework for machine learning with provable privacy guarantees for training data.
2. T HE PATE F RAMEWORK
The PATE framework (see Figure 5 in Appendix A) due to Papernot et al.
(2017) comprises three principal components: (1) an ensemble of teachers , (2) an aggregation mechanism , and (3) a student .
Teachers.
Each teacher is a model trained independently on a subset of the data whose privacy one wishes to protect.
The data is partitioned to ensure no pair of teachers will have trained on overlapping data.
Any learning technique suitable for the data can be used for any teacher.
Aggregation Mechanism.
The role of this mechanism is to be the intermediary between the student, who is never directly exposed to sensitive data, and the teachers, each of which has unfettered access to a data partition.
This step enforces privacy by tracking the privacy budget spent (see below).
Student.
The student is another model with access to public unlabeled data.
During learning, the student queries the aggregator with public data and receives private labels.
Yet, each of these labels increases the exposure of the teachers’ sensitive training data.
Hence, the student and aggregator are designed to minimize the querying needed to transfer knowledge from the teachers to the student.
2. D IFFERENTIAL P RIVACY AND R ´ ENYI D IFFERENTIAL P RIVACY FOR PATE
Differential privacy (Dwork et al., 2006) is a gold standard of privacy for statistical algorithms.
It requires that the sensitivity of the distribution of an algorithm’s output to small perturbations of its input be limited.
The following variant of the deﬁnition captures this intuition formally:
Deﬁnition 1.
A randomized mechanism M with domain D and range R satisﬁes ( ε, δ ) -differential privacy if for any two adjacent inputs D, D (cid:48) ∈ D and for any subset of outputs S ⊆ R it holds that:
[Equation]
For our application of differential privacy to ML, adjacent inputs are deﬁned as two datasets that only differ by one training example and the randomized mechanism M would be the model training algorithm.
Composition theorems for differential privacy (Dwork & Roth, 2014) lead to the following natural interpretation of the parameters from its deﬁnition: ε is an upper bound on the loss of privacy that accumulates over time and with each query to the dataset (“the privacy budget”), and δ is the probability with which this guarantee may not hold.
To better facilitate tracking privacy budget across multiple queries, we use R´enyi Differential Privacy (RDP) recently introduced by Mironov (2017).
RDP is closely related to (zero)-Concentrated
Differential Privacy (Dwork & Rothblum, 2016; Bun & Steinke, 2016) and the moments accountant (Abadi et al., 2016)—also used by Papernot et al.
(2017) in the original PATE proposal.
2. PATE A GGREGATION M ECHANISMS
The aggregation step is a crucial component of PATE.
It enables knowledge transfer from the teachers to the student while enforcing privacy.
We improve the LNMax mechanism used by Papernot et al.
(2017) which adds Laplace noise to teacher votes and outputs the class with the highest votes.
First, we add Gaussian noise with an accompanying privacy analysis in the RDP framework.
This modiﬁcation effectively reduces the noise needed to achieve the same privacy cost per student query.
Second, the aggregation mechanism is now selective : teacher votes are analyzed to decide which student queries are worth answering.
This takes into account both the privacy cost of each query and its payout in improving the student’s utility.
Surprisingly, our analysis shows that these two metrics are not at odds and in fact align with each other: the privacy cost is the smallest when teachers agree, and when teachers agree, the label is more likely to be correct thus being more useful to the student.
Third, we propose and study an interactive mechanism that takes into account not only teacher votes on a queried example but possible student predictions on that query.
Now, queries worth answering are those where the teachers agree on a class but the student is not conﬁdent in its prediction on that class.
This third modiﬁcation aligns the two metrics discussed above even further: queries where the student already agrees with the consensus of teachers are not worth expending our privacy budget on, but queries where the student is less conﬁdent are useful and answered at a small privacy cost.
2. D ATA - DEPENDENT P RIVACY IN PATE
A direct privacy analysis of the aggregation mechanism, for reasonable values of the noise parameter, allows answering only few queries before the privacy cost becomes prohibitive.
The original
PATE proposal used a data-dependent analysis, exploiting the fact that when the teachers have large agreement, the privacy cost is usually much smaller than the data-dependent bound would suggest.
In our work, we perform a data-dependent privacy analysis of the aggregation mechanism with
Gaussian noise.
This change of noise distribution turns out be technically much more challenging than the Laplace noise case and we defer the details to Appendix C. This increased complexity of the analysis however does not make the algorithm any more complicated and thus allows us to improve the privacy-utility tradeoff.
An additional challenge with data-dependent privacy analyses arises from the fact that the privacy cost itself is now a function of the private data.
Further, the data-dependent bound on the privacy cost has large global sensitivity (a metric used in differential privacy to calibrate the noise injected) and is therefore difﬁcult to sanitize.
To remedy this, we use the smooth-sensitivity framework, detailed in Section C.3.
I MPROVED A GGREGATION M ECHANISMS FOR PATE
Guarantees provided by PATE stem from the design and analysis of the aggregation step.
Here, we detail our improvements to the mechanism used by Papernot et al.
As outlined in Section 2.3, we ﬁrst replace the Laplace noise added to teacher votes with Gaussian noise, adapting the datadependent privacy analysis.
Next, we describe the Conﬁdent and Interactive Aggregators that select queries worth answering in a privacy-preserving way: the privacy budget is shared between the query selection and answer computation.
The aggregators use different heuristics to select queries: the former does not take into account student predictions, while the latter does.
3. T HE GNM AX A GGREGATOR AND I TS P RIVACY G UARANTEE
This section uses the following notation.
For a sample x and classes 1 to m , let f j ( x ) ∈
[ m ] denote the j -th teacher model’s prediction on x and n i ( x ) denote the vote count for the i -th class (i.e., n i ( x ) =
|{ j : f j ( x ) = i }| ).
We deﬁne a Gaussian NoisyMax (GNMax) aggregation mechanism as:
[Equation]
where N ( µ, σ 2 ) denotes the Gaussian distribution with mean µ and variance σ 2 .
The aggregator outputs the class with noisy plurality after adding Gaussian noise to each vote count.
In what follow, plurality more generally refers to the highest number of teacher votes assigned among the classes.
The Gaussian distribution is more concentrated than the Laplace distribution used by Papernot et al.
This concentration directly improves the aggregation’s utility when the number of classes m is large.
The GNMax mechanism satisﬁes ( λ, λ/σ 2 ) -RDP (see Lemma 8 in Appendix C) which holds for all inputs and all λ ≥ 1 .
A straightforward application of composition theorems leads to loose privacy bounds.
As an example, the standard advanced composition theorem applied to experiments in rows 5 and 6 in Table 1 would give us ε = 8 .
42 and ε = 10 .
14 resp.
at δ = 10 − 8 for the Glyph dataset.
To reﬁne these, we work out a careful data-dependent analysis that yields ε ’s close to 1 for the same δ .
The following theorem translates standard RDP guarantees for higher orders into a data-dependent
RDP guarantee for a smaller order λ .
Its precise statement and proof appear in Appendix C. We use it in conjunction with Lemma 3 to bound the privacy cost of each query to the GNMax algorithm as a function of ˜ q , the probability that the most common answer will not be output by the mechanism.
Theorem 2 (informal) .
Let M be a randomized algorithm with ( µ 1 , ε 1 ) -RDP and ( µ 2 , ε 2 ) -
RDP guarantees and suppose that given a dataset D , there exists a likely outcome i ∗ such that Pr [ M ( D ) (cid:54)
= i ∗ ] ≤ ˜ q .
Then the data-dependent R´enyi differential privacy for M of order λ ≤ µ 1 , µ 2 at D is bounded by a function of ˜ q , µ 1 , ε 1 , µ 2 , ε 2 , which approaches 0 as ˜ q → 0 .
The new bound improves on the data-independent privacy for λ as long as the distribution of the algorithm’s output on that input has a strong peak (i.e., ˜ q (cid:28) 1 ).
Values of ˜ q close to 1 could result in a looser bound.
Therefore, in practice we take the minimum between this bound and λ/σ 2 (the data-independent one).
The theorem generalizes Theorem 3 from Papernot et al.
(2017), where it was shown for a mechanism satisfying ε -differential privacy (i.e., µ 1 = µ 2 = ∞ and ε 1 = ε 2 ).
The ﬁnal step in our analysis uses the following lemma to bound the probability ˜ q when i ∗ corresponds to the class with the true plurality of teacher votes.
Lemma 3.
For any i ∗ ∈ [ m ] , we have Pr [ A σ ( D ) (cid:54) =
i ∗ ] ≤ 12 (cid:80) i (cid:54) = i ∗ erfc (cid:0) n i ∗ − n i 2 σ (cid:1) , where erfc is the complementary error function.
In Appendix C, we detail how these results translate to privacy bounds.
In short, for each query to the
GNMax aggregator, given teacher votes n i and the class i ∗ with maximal support, Lemma 3 gives us the value of ˜ q to use in Theorem 2.
We optimize over µ 1 and µ 2 to get a data-dependent RDP guarantee for any order λ .
Finally, we use composition properties of RDP to analyze a sequence of queries, and translate the RDP bound back to an ( ε, δ ) -DP bound.
Expensive queries.
This data-dependent privacy analysis leads us to the concept of an expensive query in terms of its privacy cost.
When teacher votes largely disagree, some n i ∗ − n i values may be small leading to a large value for ˜ q : i.e., the lack of consensus amongst teachers indicates that the aggregator is likely to output the wrong label.
Thus expensive queries from a privacy perspective are often bad for training too.
Conversely, queries with strong consensus enable tight privacy bounds.
This synergy motivates the aggregation mechanisms discussed in the following sections: they evaluate the strength of the consensus before answering a query.
3. T HE C ONFIDENT -GNM AX A GGREGATOR
In this section, we propose a reﬁnement of the GNMax aggregator that enables us to ﬁlter out queries for which teachers do not have a sufﬁciently strong consensus.
This ﬁltering enables the teachers to avoid answering expensive queries.
We also take note to do this selection step itself in a private manner.
The proposed Conﬁdent Aggregator is described in Algorithm 1.
To select queries with overwhelming consensus, the algorithm checks if the plurality vote crosses a threshold T .
To enforce privacy in this step, the comparison is done after adding Gaussian noise with a large deviation σ 1 .
Then, for queries that pass this noisy threshold check, the aggregator proceeds with the usual GNMax mechanism with a different deviation σ 2 .
For queries that do not pass the noisy threshold check, the aggregator simply returns ⊥ and the student discards this example in its training.
In practice, we often choose signiﬁcantly higher values for σ 1 compared to σ 2 .
The reason for doing is that we pay the cost of the noisy threshold check always , and without the beneﬁt of knowing that the consensus is strong.
We pick T so that queries where the plurality gets less than half the votes (often very expensive) are unlikely to pass the threshold after adding noise, but we still have a high enough yield amongst the queries with a strong consensus.
This tradeoff leads us to look for T ’s between 0 .
6 × to 0 .
8 × the number of teachers.
The privacy cost of this aggregator is intuitive: we pay for the threshold check for every query, and for the GNMax step only for queries that pass the check.
In the work of Papernot et al.
(2017), the mechanism paid a privacy cost for every query, expensive or otherwise.
In comparison, the Conﬁdent
Aggregator expends a much smaller privacy cost to check against the threshold, and by answering a signiﬁcantly smaller fraction of expensive queries, it expends a lower privacy cost overall.
3. T HE I NTERACTIVE -GNM AX A GGREGATOR
While the Conﬁdent Aggregator exclude expensive queries, it ignores the possibility that the student might receive labels that contribute little to learning, and in turn to its utility.
By incorporating the student’s current predictions for its public training data, we design an Interactive Aggregator that discards queries where the student already conﬁdently predicts the same label as the teachers.
Given a set of queries, the Interactive Aggregator (Algorithm 2) selects those answered by comparing student predictions to teacher votes for each class.
Similar to Step 1 in the Conﬁdent Aggregator, Algorithm given query, consensus in way queries where the plurality of these noised differences crosses a threshold are answered with GNMax.
This noisy threshold sufﬁces to enforce privacy of the ﬁrst step because student predictions can be considered public information (the student is trained in a differentially private manner).
For queries that fail this check, the mechanism reinforces the predicted student label if the student is conﬁdent enough and does this without looking at teacher votes again.
This limited form of supervision comes at a small privacy cost.
Moreover, the order of the checks ensures that a student falsely conﬁdent in its predictions on a query is not accidentally reinforced if it disagrees with the teacher consensus.
The privacy accounting is identical to the Conﬁdent Aggregator except in considering the difference between teachers and the student instead of only the teachers votes.
In practice, the Conﬁdent Aggregator can be used to start training a student when it can make no meaningful predictions and training can be ﬁnished off with the Interactive Aggregator after the student gains some proﬁciency.
E XPERIMENTAL E VALUATION
Our goal is ﬁrst to show that the improved aggregators introduced in Section 3 enable the application of PATE to uncurated data, thus departing from previous results on tasks with balanced and wellseparated classes.
We experiment with the Glyph dataset described below to address two aspects left open by Papernot et al.
(2017): (a) the performance of PATE on a task with a larger number of classes (the framework was only evaluated on datasets with at most 10 classes) and (b) the privacy-utility trade-offs offered by PATE on data that is class imbalanced and partly mislabeled.
In Section 4.2, we evaluate the improvements given by the GNMax aggregator over its Laplace counterpart (LNMax) and demonstrate the necessity of the Gaussian mechanism for uncurated tasks.
In Section 4.3, we then evaluate the performance of PATE with both the Conﬁdent and Interactive
Aggregators on all datasets used to benchmark the original PATE framework, in addition to Glyph.
With the right teacher and student training, the two mechanisms from Section 3 achieve high accuracy with very tight privacy bounds.
Not answering queries for which teacher consensus is too low (Conﬁdent-GNMax) or the student’s predictions already agree with teacher votes (InteractiveGNMax) better aligns utility and privacy: queries are answered at a signiﬁcantly reduced cost.
4. E XPERIMENTAL S ETUP
MNIST, SVHN, UCI Adult, UCI Diabetes.
We evaluate with two computer vision tasks (MNIST and Street View House Numbers (Netzer et al., 2011)), census data from the UCI Adult dataset Kohavi (1996), and medical records from the UCI Diabetes Strack et al.
(2014) dataset.
This enables a comparative analysis of the utility-privacy trade-off achieved with our Conﬁdent-GNMax aggregator and the LNMax originally used in PATE.
We replicate the experimental setup and results found in Papernot et al.
(2017) thanks in part to the code and teacher votes made available online.
A detailed description of the experimental setup can be found in Papernot et al.
; we provide here only a brief overview.
For MNIST and SVHN, teachers are convolutional networks trained on partitions of the training set.
For Adult and Diabetes, each teacher is a random forest.
The test set is split in two halves: the ﬁrst is used as unlabeled inputs to simulate the student’s public data and the second is used as a hold out to evaluate test performance.
The MNIST and SVHN students are convolutional networks trained using semi-supervised learning with GANs `a la Salimans et al.
The students for the Adult and Diabetes datasets are fully supervised random forests.
Glyph.
This optical character recognition task has an order of magnitude more classes than all previous applications of PATE.
The Glyph dataset also possesses many characteristics shared by real-world tasks: e.g., it is imbalanced and some inputs are mislabeled.
Each input is a 28 ×grayscale image containing a single glyph generated synthetically from a collection of over 500K computer fonts.
1 Samples representative of the difﬁculties raised by the data are depicted in Figure 2.
The task is to classify inputs as one of the 150 Unicode symbols used to generate them.
This set of 150 classes results from pre-processing efforts.
We discarded additional classes that had few samples; some classes had at least 50 times fewer inputs than the most popular classes, and these were almost exclusively incorrectly labeled inputs.
We also merged classes that were too ambiguous for even a human to differentiate them.
Nevertheless, a manual inspection of samples grouped by classes—favorably to the human observer—led to the conservative estimate that some classes remain 5 times more frequent, and mislabeled inputs represent at least 10% of the data.
To simulate the availability of private and public data (see Appendix A), we split data originally marked as the training set (about 65M points) into partitions given to the teachers.
Each teacher is a ResNet (He et al., 2016) made of 32 leaky ReLU layers.
2 We train on batches of 100 inputs for 40K steps with the Momentum algorithm.
The learning rate, initially set to 0 . 1 , is decayed after 10K steps to 0 .
01 and again after 20K steps to 0 .
These parameters were found with a grid search.
We split holdout data in two subsets of 100K and 400K samples: the ﬁrst acts as public data to train the student and the second as its testing data.
The student architecture is a convolutional network learned in a semi-supervised fashion with virtual adversarial training (VAT) from Miyato et al. Using unlabeled data, we show how VAT can regularize the student by making predictions constant in adversarial 3 directions.
Indeed, we found that GANs did not yield as much utility for Glyph as for MNIST or SVHN.
We train with Adam for 400 epochs and a learning rate of 6 · 10 − 5 .
4. C OMPARING THE LNM AX AND GNM AX M ECHANISMS
Section 3.1 introduces the GNMax mechanism and the accompanying privacy analysis.
With a
Gaussian distribution, whose tail diminishes more rapidly than the Laplace distribution, we expect better utility when using the new mechanism (albeit with a more involved privacy analysis).
To study the trade-off between privacy and accuracy with the two mechanisms, we run experiments training several ensembles of M teachers for M ∈ { 100 , 500 , 1000 , 5000 } on the Glyph data.
Recall that 65 million training inputs are partitioned and distributed among the M teachers with each teacher receiving between 650K and 13K inputs for the values of M above.
The test data is used to query the teacher ensemble and the resulting labels (after the LNMax and GNMax mechanisms) are compared with the ground truth labels provided in the dataset.
This predictive performance of the teachers is essential to good student training with accurate labels and is a useful proxy for utility.
For each mechanism, we compute ( ε, δ ) -differential privacy guarantees.
As is common in literature, for a dataset on the order of 10 8 samples, we choose δ = 10 − 8 and denote the corresponding ε as the privacy cost.
The total ε is calculated on a subset of 4,000 queries, which is representative of the number of labels needed by a student for accurate training (see Section 4.3).
We visualize in
Figure 3 the effect of the noise distribution (left) and the number of teachers (right) on the tradeoff between privacy costs and label accuracy.
Observations.
On the left of Figure 1,we compare our GNMax aggregator to the LNMax aggregator used by the original PATE proposal, on an ensemble of 1000 teachers and for varying noise scales σ .
At ﬁxed test accuracy, the GNMax algorithm consistently outperforms the LNMax mechanism in terms of privacy cost.
To explain this improved performance, recall notation from Section 3.1.
For both mechanisms, the data dependent privacy cost scales linearly with ˜ q —the likelihood of an answer other than the true plurality.
The value of ˜ q falls of as exp (cid:0) − x 2 (cid:1) for GNMax and exp ( − x ) for LNMax, where x is the ratio ( n i ∗ − n i ) /σ .
Thus, when n i ∗ − n i is (say) 4 σ , LNMax would have ˜ q
≈
e
− 4 = 0 .
018 ... , whereas GNMax would have ˜ q ≈ e − 16 ≈ 10 − 7 , thereby leading to a much higher likelihood of returning the true plurality.
Moreover, this reduced ˜ q translates to a smaller privacy cost for a given σ leading to a better utility-privacy trade-off.
As long as each teacher has sufﬁcient data to learn a good-enough model, increasing the number M of teachers improves the trade-off—as illustrated on the right of Figure 3 with GNMax.
The larger ensembles lower the privacy cost of answering queries by tolerating larger σ ’s.
Combining the two observations made in this Figure, for a ﬁxed label accuracy, we lower privacy costs by switching to the GNMax aggregator and training a larger number M of teachers.
4. S TUDENT T RAINING WITH THE GNM AX A GGREGATION M ECHANISMS
As outlined in Section 2, we train a student on public data labeled by the aggregation mechanisms.
We take advantage of PATE’s ﬂexibility and apply the technique that performs best on each dataset: semi-supervised learning with Generative Adversarial Networks (Salimans et al., 2016) for MNIST and SVHN, Virtual Adversarial Training (Miyato et al., 2017) for Glyph, and fully-supervised random forests for Adult and Diabetes.
In addition to evaluating the total privacy cost associated with training the student model, we compare its utility to a non-private baseline obtained by training on the sensitive data (used to train teachers in PATE): we use the baselines of 99 . 2% , 92 . 8% , 85 . 0% , and 93 .
8% reported by Papernot et al.
(2017) respectively for MNIST, SVHN, Adult, and Diabetes,
and we measure a baseline of 82 .
2% for Glyph.
We compute ( ε, δ ) -privacy bounds and denote the privacy cost as the ε value at a value of δ set accordingly to number of training samples.
Conﬁdent-GNMax Aggregator.
Given a pool of 500 to 12,000 samples to learn from (depending on the dataset), the student submits queries to the teacher ensemble running the Conﬁdent-GNMax aggregator from Section 3.2.
A grid search over a range of plausible values for parameters T , σand σ 2 yielded the values reported in Table 1, illustrating the tradeoff between utility and privacy achieved.
We additionally measure the number of queries selected by the teachers to be answered and compare student utility to a non-private baseline.
The Conﬁdent-GNMax aggregator outperforms LNMax for the four datasets considered in the original PATE proposal: it reduces the privacy cost ε , increases student accuracy, or both simultaneously.
On the uncurated Glyph data, despite the imbalance of classes and mislabeled data (as evidenced by the 82.2% baseline), the Conﬁdent Aggregator achieves 73.5% accuracy with a privacy cost of just ε = 1 . 02 .
Roughly 1,300 out of 12,000 queries made are not answered, indicating that several costly queries were successfully avoided.
This selectivity is analyzed in more details in Section 4.4.
Interactive-GNMax Aggregator.
On Glyph, we evaluate the utility and privacy of an interactive training routine that proceeds in two rounds .
Round one runs student training with a Conﬁdent Aggregator.
A grid search targeting the best privacy for roughly 3,400 answered queries (out of 6,000)—sufﬁcient to bootstrap a student—led us to setting ( T = 3500 , σ 1 = 1500 , σ 2 = 100) and a privacy cost of ε ≈ 0 . 65 . [Figure] [Figure] [Figure]
Dataset [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
[Table]
[Figure]
[Figure]
[Table]
[Table]
SVHN
[Table]
[Table]
[Figure]
In round two
[Figure]
this student was then trained with 10,000 more queries made with the InteractiveGNMax Aggregator.
We computed the resulting (total) privacy cost and utility at an exemplar data point through another grid search of plausible parameter values.
The result appears in the last row of Table 1.
With just over 10,422 answered queries in total at a privacy cost of ε = 0 .
99 , the trained student was able to achieve 73.2% accuracy.
Note that this students required fewer answered queries compared to the Conﬁdent Aggregator.
The best overall cost of student training occurred when the privacy costs for the ﬁrst and second rounds of training were roughly the same. (The total ε is less than 0 .
65 × 2 = 1 . 3 due to better composition—via Theorems 6 and 7.)
Baseline.
Note that the Glyph student’s accuracy remains seven percentage points below the nonprivate model’s accuracy achieved by training on the 65M sensitive inputs.
We hypothesize that is due to the uncurated nature of the data considered.
Indeed, the class imbalance naturally requires more queries to return labels from the less represented classes.
For instance, a student trained on 200K queries is only 77% accurate on test data.
In addition, the large fraction of mislabeled inputs are likely to have a large privacy cost: these inputs are sensitive because they are outliers of the distribution, which is reﬂected by the weak consensus among teachers on these inputs.
4. N OISY T HRESHOLD C HECKS AND P RIVACY C OSTS
Sections 3.1 and 3.2 motivated the need for a noisy threshold checking step before having the teachers answer queries: it prevents most of the privacy budget being consumed by few queries that are expensive and also likely to be incorrectly answered.
In Figure 4, we compare the privacy cost ε of answering all queries to only answering conﬁdent queries for a ﬁxed number of queries.
We run additional experiments to support the evaluation from Section 4.3.
With the votes of 5, teachers on the Glyph dataset, we plot in Figure 4 the histogram of the plurality vote counts ( n i ∗ in the notation of Section 3.1) across 25,000 student queries.
We compare these values to the vote counts of queries that passed the noisy threshold check for two sets of parameters T and σ 1 in
Algorithm 1.
Smaller values imply weaker teacher agreements and consequently more expensive queries.
When ( T = 3500 , σ 1 = 1500) we capture a signiﬁcant fraction of queries where teachers have a strong consensus (roughly > 4000 votes) while managing to ﬁlter out many queries with poor consensus.
This moderate check ensures that although many queries with plurality votes between 2,500 and 3,500 are answered (i.e., only 50–70% of teachers agree on a label)
the expensive ones are most likely discarded.
For ( T = 5000 , σ 1 = 1500) , queries with poor consensus are completely culled out.
This selectivity comes at the expense of a noticeable drop for queries that might have had a strong consensus and little-to-no privacy cost.
Thus, this aggressive check answer fewer queries with very strong privacy guarantees.
We reiterate that this threshold checking step itself is done in a private manner.
Empirically, in our Interactive Aggregator experiments, we expend about a third to a half of our privacy budget on this step, which still yields a very small cost per query across 6,000 queries.
C ONCLUSIONS
The key insight motivating the addition of a noisy thresholding step to the two aggregation mechanisms proposed in our work is that there is a form of synergy between the privacy and accuracy of labels output by the aggregation: labels that come at a small privacy cost also happen to be more likely to be correct .
As a consequence, we are able to provide more quality supervision to the student by choosing not to output labels when the consensus among teachers is too low to provide an aggregated prediction at a small cost in privacy.
This observation was further conﬁrmed in some of our experiments where we observed that if we trained the student on either private or non-private labels, the former almost always gave better performance than the latter—for a ﬁxed number of labels.
Complimentary with these aggregation mechanisms is the use of a Gaussian (rather than Laplace) distribution to perturb teacher votes.
In our experiments with Glyph data, these changes proved essential to preserve the accuracy of the aggregated labels—because of the large number of classes.
The analysis presented in Section 3 details the delicate but necessary adaptation of analog results for the Laplace NoisyMax.
As was the case for the original PATE proposal, semi-supervised learning was instrumental to ensure the student achieves strong utility given a limited set of labels from the aggregation mechanism.
However, we found that virtual adversarial training outperforms the approach from Salimans et al.
(2016) for the Glyph dataset.
Thus, future work may continue to investigate other techniques for learning the student in these particular settings with restricted supervision.
A A PPENDIX : B ACKGROUND ON PATE
Here, we provide an overview of the PATE framework.
To protect the privacy of training data during learning, PATE transfers knowledge from an ensemble of teacher models trained on partitions of the data to a student model.
Privacy guarantees may be understood intuitively and expressed rigorously in terms of differential privacy.
Illustrated in Figure 5, the PATE framework consists of three key parts: (1) an ensemble of n teacher models, (2) an aggregation mechanism and (3) a student model.
[Figure]
Teacher models: Training each teacher on a partition of the sensitive data produces n different models solving the same task.
At inference, teachers independently predict labels.
When there is a strong consensus among teachers, the label they almost all agree on does not depend on the model learned by any given teacher.
Hence, this collective decision is intuitively private with respect to any given training point—because such a point could have been included only in one of the teachers’ training set.
Aggregation mechanism: The aggregation mechanism of the original PATE framework counts votes assigned to each class, adds carefully calibrated Laplacian noise to the resulting vote histogram, and outputs the class with the most noisy votes as the ensemble’s prediction.
This mechanism is referred to as the max-of-Laplacian mechanism going forward.
For samples x and classes 1 , . . .
, m , let f j ( x ) ∈
[ m ] denote the j -th teacher model’s prediction and n i denote the vote count for the i -th class (i.e., n i (cid:44) | f j ( x ) =
i | ).
The output of the mechanism is
A ( x ) (cid:44) argmax i ( n i ( x ) + Lap (1 /γ )) .
Through a rigorous analysis of this mechanism, the PATE framework provides a differentially private API: the privacy cost of each aggregated prediction made by the teacher ensemble is known.
Student model: PATE’s ﬁnal step involves the training of a student model by knowledge transfer from the teacher ensemble using access to public—but unlabeled —data.
To limit the privacy cost of labeling them, queries are only made to the aggregation mechanism for a subset of public data to train the student in a semi-supervised way using a ﬁxed number of queries.
The authors note that every additional ensemble prediction increases the privacy cost spent and thus cannot work with unbounded queries.
Fixed queries ﬁxes privacy costs as well as diminishes the avlue of attacks analyzing model parameters to recover training data (Zhang et al., 2017).
The student only sees public data and privacy-preserving labels.
B A PPENDIX : R ELATED W ORK
Differential privacy is by now the gold standard of privacy.
It offers a rigorous framework whose threat model makes few assumptions about the adversary’s capabilities, allowing differentially private algorithms to effectively cope against strong adversaries.
This is not the case of all privacy deﬁnitions, as demonstrated by successful attacks against anonymization techniques (Aggarwal, 2005; Narayanan & Shmatikov, 2008; Bindschaedler et al., 2017).
The ﬁrst learning algorithms adapted to provide differential privacy with respect to their training data were often linear and convex (Pathak et al., 2010; Chaudhuri et al., 2011; Song et al., 2013;
Bassily et al., 2014; Hamm et al., 2016).
More recently, successful developments in deep learning called for differentially private stochastic gradient descent algorithms (Abadi et al., 2016), some of which have been tailored to learn in federated (McMahan et al., 2017) settings.
Differentially private selection mechanisms like GNMax are commonly used in hypothesis testing, frequent itemset mining, and as building blocks of more complicated private mechanisms.
The most commonly used differentially private selection mechanisms are exponential mechanism (McSherry & Talwar, 2007) and LNMax (Bhaskar et al., 2010).
Recent works offer lower bounds on sample complexity of such problem (Steinke & Ullman, 2017; Bafna & Ullman, 2017).
The Conﬁdent and Interactive Aggregator proposed in our work use the intuition that selecting samples under certain constraints could result in better training than using samples uniformly at random.
In Machine Learning Theory, active learning (Cohn et al., 1994) has been shown to allow learning from fewer labeled examples than the passive case (see e.g. Hanneke (2014)).
Similarly, in model stealing (Tram`er et al., 2016), a goal is to learn a model from limited access to a teacher network.
There is previous work in differential privacy literature (Hardt & Rothblum, 2010; Roth &
Roughgarden, 2010) where the mechanism ﬁrst decides whether or not to answer a query, and then privately answers the queries it chooses to answer using a traditional noise-addition mechanism.
In these cases, the sparse vector technique (Dwork & Roth, 2014, Chapter 3.6) helps bound the privacy cost in terms of the number of answered queries.
This is in contrast to our work where a constant fraction of queries get answered and the sparse vector technique does not seem to help reduce the privacy cost.
C A PPENDIX : R ´ ENYI D IFFERENTIAL P RIVACY AND P RIVACY P
ROOFS C. R´ ENYI D IFFERENTIAL P RIVACY
Papernot et al.
(2017) note that the natural approach to bound PATE’s privacy loss—by ﬁrst having a bound for the privacy cost of each label queried and then using strong composition (Dwork et al., 2010) to derive the total cost—yields loose privacy guarantees.
Instead, their approach uses datadependent privacy analysis.
This takes advantage of the fact that when the consensus among the teachers is very strong, the plurality outcome has overwhelming likelihood leading to a very small privacy cost whenever the consensus occurs.
To capture this effect quantitatively, Papernot et al.
(2017) rely on the moments accountant , introduced by Abadi et al.
and building on previous work (Bun & Steinke, 2016; Dwork & Rothblum, 2016).
In this section, we restate the privacy analysis framework of PATE in the language of R ´ enyi Differential Privacy or RDP (Mironov, 2017).
RDP generalizes pure differential privacy ( δ = 0 ) and is closely related to the moments accountant.
We choose to use RDP as a more natural analysis framework when dealing with our mechanisms that use Gaussian noise.
Deﬁned below, the RDP of a mechanism is stated in terms of the R´enyi divergence.
Deﬁnition 4 (R´enyi Divergence) .
The R´enyi divergence of order λ between two distributions P and Q is deﬁned as:
[Equation]
Deﬁnition 5 (R´enyi Differential Privacy (RDP)) .
A randomized mechanism M is said to guarantee ( λ, ε ) -RDP with λ ≥ 1 if for any neighboring datasets D and D (cid:48) ,
[Equation]
RDP generalizes pure differential privacy in the sense that ε -differential is equivalent to ( ∞ , ε ) -RDP.
Mironov (2017) proves the following key facts that allow easy composition of RDP guarantees and their conversion to ( ε, δ )
-differential privacy bounds.
Theorem 6 (Composition) .
If a mechanism M consists of a sequence of adaptive mechanisms
M 1 , . .
, M k such that for any i ∈ [ k ] , M i guarantees ( λ, ε i )
-RDP, then M guarantees ( λ, (cid:80) ki
=1 ε i ) -RDP.
Theorem 7 (From RDP to DP) .
If a mechanism M guarantees ( λ, ε ) -RDP, then M guarantees ( ε + log1 /δ
λ − 1 , δ ) -differential privacy for any δ ∈ (0 , 1) .
While both ( ε, δ ) -differential privacy and RDP are relaxations of pure ε -differential privacy, the two main advantages of RDP are as follows.
First, it composes nicely; second, it captures the privacy guarantee of Gaussian noise in a much cleaner manner compared to ( ε, δ ) -differential privacy.
This lets us do a careful privacy analysis of the GNMax mechanism as stated in Theorem 2.
While the analysis of Papernot et al.
(2017) leverages the ﬁrst aspect of such frameworks with the Laplace noise (LNMax mechanism), our analysis of the GNMax mechanism relies on both.
C. P RIVACY P ROOFS
In this section, we provide the proofs of Theorem 2 and Lemma 3.
Moreover, we present Lemma 9, which provides optimal values of µ 1 and µ 2 to apply towards Theorem 2 for the GNMax mechanism.
We start off with a lemma about the R ´ enyi differential privacy guarantee of the GNMax.
Lemma 8.
The GNMax aggregator A σ guarantees (cid:0) λ, λ/σ 2 (cid:1) -RDP for all λ ≥ 1 .
Proof.
The result follows from observing that A σ can be decomposed into applying the argmax operator to a noisy histogram resulted from adding Gaussian noise to each dimension of the original histogram.
The Gaussian mechanism satisﬁes ( λ, λ/ 2 σ 2 )
-RDP (Mironov, 2017), and since each teacher may change two counts (incrementing one and decrementing the other), the overall RDP guarantee is as claimed.
Lemma 3.
For any i ∗ ∈ [ m ] , we have Pr [ A
[Equation]
Proof.
For any i ∗ ∈ [ n ] , we have
[Equation]
where the last equality follows from the fact that Z i − Z j is a Gaussian random variable with mean zero and variance 2 σ 2 .
We now present a precise statement of Theorem 2.
Theorem 2.
Let M be a randomized algorithm with ( µ 1 , ε 1 ) -RDP and ( µ 2 , ε 2 ) -RDP guarantees and suppose that there exists a likely outcome i ∗ given a dataset D such that ˜ q ≥ Pr [ M ( D ) (cid:54)
= i ∗ ] .
Let λ ≤ µ 1 and ˜ q ≤ e ( µ 2 − 1) ε 2 / (cid:16) µ 1 µ 1 − 1 · µ 2 µ 2 − 1 (cid:17) µ 2 .
Then, for any neighboring dataset D (cid:48) of D , we have:
[Equation]
2) where A (˜ q, µ 2 , ε 2 ) (cid:44) (1 − ˜ q ) / (cid:16) 1 − (˜ qe ε 2 ) µ 2 − 1 µ 2 (cid:17) and B (˜ q, µ 1 , ε 1 ) (cid:44) e ε 1 / ˜ q 1 µ 1 − 1 .
Proof.
Before we proceed to the proof, we introduce some simplifying notation.
For a randomized mechanism M and neighboring datasets D and D (cid:48) , we deﬁne
[Equation]
As the proof involves working with the RDP bounds in the exponent, we set ζ 1 (cid:44) e ε 1 ( µ 1 − 1) and ζ 2 (cid:44) e ε 2 ( µ 2 − 1) .
Finally, we deﬁne the following shortcuts: q i
[Equation]
[Equation]
D )
[Equation]
i ] and q (cid:44) (cid:88)
[Equation]
, p i (cid:44) Pr [ M ( D (cid:48)
[Equation]
and p (cid:44) (cid:88) i (cid:54) =
i
[Equation]
(cid:48) )
(cid:54)
= i ∗ ] , and note that q ≤ ˜ q .
From the deﬁnition of R´enyi differential privacy of order µ 1 , we have:
[Equation]
Since µ 1 > λ , f ( x ) (cid:44) x µ 1 − 1 λ − 1 is convex.
Applying Jensen’s Inequality we have the following:
[Equation]
Next, by the bound at order µ 2 , we have:
[Equation]
By the data processing inequality of R´enyi divergence
[Equation]
we have
[Equation]
which implies p µ 2 q µ 2 − 1 ≤ ζ 2 and thus
[Equation]
Combining (4) and (5) [Equation] we can derive a bound at λ .
[Equation]
Although Equation (6) is very close to the corresponding statement in the theorem’s claim, one subtlety remains.
The bound (6) applies to the exact probability q =
Pr [ M ( D ) (cid:54) =
i ∗ ] .
In the theorem statement, and in practice, we can only derive an upper bound ˜ q on Pr [ M ( D ) (cid:54)
=
i ∗ ] .
The last step of the proof requires showing that the expression in Equation (6) is monotone in the range of values of q that we care about.
Monotonicity of the bound.
Below we demonstrate that for the functions
[Equation]
and
[Equation]
deﬁned on x ∈ [0 , 1] as above, f 1 ( x ) + f 2 ( x ) is increasing in (cid:104)
[Equation]
Taking the derivative of f 1 ( x ) , we have:
[Equation]
We intend to show that:
[Equation]
For x ∈ (cid:104) 0 , ζ 2 / (cid:16) µ 1 µ 1 − 1 · µ 2 µ 2 − 1 (cid:17) µ 2 (cid:105) and y ∈ [1 , ∞ ) , deﬁne g ( x, y ) as:
[Equation]
We claim that g ( x, y ) is increasing in y and therefore g ( x, y ) ≥ g ( x, 1) , and prove it by showing the partial derivative of g ( x, y ) with respect to y is non-negative.
Take a derivative with respect to y as:
[Equation]
To see why g (cid:48)
y ( x, y ) is non-negative in the respective ranges of x and y , note that:
[Equation]
range of
[Equation]
and y )
Consider 1 − x 1 − ( x
[Equation]
2 − 1 ζ 2 ) 1 /µ 2 .
Since ζ 2 ≥ 1 and x ≤ 1 , we have x ≤ ζ 2 and hence
[Equation]
Therefore we can set y =
[Equation]
[Equation] [Equation]
[Equation]
2 and apply the fact that g ( x, y ) ≥ g ( x, 1) for all y ≥ 1 to get
[Equation]
as required by (7).
Taking the derivative of f 2 ( x )
[Equation]
we have:
[Equation]
Combining the two terms together
[Equation]
we have:
[Equation]
For f (cid:48) ( x ) to be non-negative we need:
[Equation]
(cid:19) 1 µ 2 ≥⇐⇒ (cid:18) µ
[Equation]
µ 1 −
[Equation]
· µ
[Equation]
µ
[Equation]
−
[Equation]
(cid:19)
[Equation]
≤ ζ
[Equation]
x .
So f ( x ) is increasing for x ∈ (cid:104) 0 , ζ 2 / (cid:16) µ 1 µ 1 − 1 · µ 2 µ 2 − 1 (cid:17) µ 2 (cid:105) .
This means for q ≤ ˜ q ≤ ζ 2 / (cid:16) µ 1 µ 1 − 1 · µ 2 µ 2 − 1 (cid:17) µ 2 , we have f ( q ) ≤ f (˜ q ) .
This completes the proof of the theorem.
Theorem 2 yields data-dependent R´enyi differential privacy bounds for any value of µ 1 and µ 2 larger than λ .
The following lemma simpliﬁes this search by calculating optimal higher moments µ 1 and µ 2 for the GNMax mechanism with variance σ 2 .
Lemma 9.
When applying Theorem 2 and Lemma 8 for GNMax with Gaussian of variance σ 2 ,
Eq.
(2) is minimized at
[Equation]
Proof.
We can minimize both terms in (2) independently.
To minimize the ﬁrst term in (2), we minimize (˜ qe ε 2 ) 1 − 1 /µ 2 by considering logarithms:
[Equation]
which is minimized at µ 2 = σ · (cid:112) log(1 / ˜ q ) .
To minimize the second term in (2), we minimize e ε 1 / ˜ q 1 / ( µ 1 − 1) as follows:
[Equation]
which is minimized at µ 1 = 1 + σ · (cid:112) log(1 / ˜ q ) completing the proof.
The following corollary offers a simple asymptotic expression of the privacy of GNMax for the case when there are large (relative to σ ) gaps between the highest three vote counts.
Corollary 10.
If the top three vote counts are n 1 > n 2 > n 3 and n 1 − n 2 , n 2 − n 3 (cid:29) σ , then the mechanism GNMax with Gaussian of variance σ 2 satisﬁes ( λ, exp (cid:0) − 2 λ/σ 2 (cid:1) /λ )
-RDP for λ =
( n 1 − n 2 ) / 4 .
Proof.
Denote the noisy counts as ˜ n i = n i + N (0 , σ 2 ) .
Ignoring outputs other than those with the highest and the second highest counts, we bound q =
Pr [ M ( D ) (cid:54) = 1] as Pr [˜ n 1 < ˜ n 2 ] =
Pr [ N (0 , 2 σ 2 ) > n 1 − n 2 ] < exp (cid:0) − ( n 1 − n 2 ) 2 / 4 σ 2 (cid:1) , which we use as ˜ q .
Plugging ˜ q in Lemma 9, we have µ 1 − 1 = µ 2 =
( n 1 − n 2 ) / 2 , limiting the range of applicability of Theorem 2 to λ < ( n 1 − n 2 ) / 2 .
Choosing λ
= ( n 1 − n 2 ) / 4 ensures A (˜ q, µ 2 , ε 2 )
≈ 1 , which allows approximating the bound (2) as ˜ q · B (˜ q, µ 1 , ε 1 ) λ − 1 / ( λ − 1) .
The proof follows by straightforward calculation.
C. S MOOTH S ENSITIVITY AND P UBLISHING THE P RIVACY P ARAMETER
The privacy guarantees obtained for the mechanisms in this paper via Theorem 2 take as input ˜ q , an upper bound on the probability that the aggregate mechanism returns the true plurality.
This means that the resulting privacy parameters computed depend on teacher votes and hence the underlying data.
To avoid potential privacy breaches from simply publishing the data-dependent parameter, we need to publish a sanitized version of the privacy loss.
This is done by adding noise to the computed privacy loss estimates using the smooth sensitivity algorithm proposed by Nissim et al.
This section provides an overview of an algorithm that computes the smooth sensitivity of the privacy loss function.
Any dataset D deﬁnes a histogram ¯ n = ( n 1 , . . . , n m ) ∈
R m of the teachers’ votes.
We have a natural notion of the distance between two histograms dist(¯ n, ¯ n (cid:48) ) and a function q : R m → [0 , 1] on these histograms computing the bound according to Lemma 3.
The value q (¯ n ) can be used as ˜ q in the application of Theorem 2.
Additionally we have n ( i ) denote the i -th highest bar in the histogram.
Recall from Section 3.1 that in calculating the privacy parameter we use the lower of two bounds: a data-dependent and a data-independent one.
Therefore, in what follows, for a given moment λ , we use a (rescaled) bound provided by Theorem 2 and Lemma 8.
[Equation]
where A and B are deﬁned as in the theorem statement, and the parameters µ 1 , µ 2 , ε 1 , ε 2 are selected according to Lemmas 8 and 9.
We aim at calculating the smooth sensitivity of α ( q ( · ))
whose deﬁnition we recall now.
Deﬁnition 11 (Smooth Sensitivity) .
Given the smoothness parameter β , a β - smooth sensitivity of f ( n ) is deﬁned as
[Equation]
where
[Equation]
is an upper bound on the local sensitivity.
We now describe Algorithms 3–5 computing the smooth sensitivity of α ( q ( · )) .
Their correctness is stated in lemmas that follow.
The algorithms assume the existence of the following mathematical constructs, which can be observed to be true for the GNMax algorithm or shown via numerical computations:
• There exist increasing functions B U , B L : [0 , 1] → [0 , 1] that are inverses of each other (i
e., B U (B L ( q )) = q ) and respectively upper bound and lower bound the value of q evaluated at any neighbor of ¯ n given q (¯ n ) .
• α ( · ) increases in the range [0 , q 0 ] for some value q 0 < 1 / 2 and remains ﬁxed at α ( q 0 ) for q > q 0 where the data-independent—and hence q -independent—bound operates.
Additionally, q ([ n, 0 , . . . , 0]) ≤ q 0 .
• ∆ α ( q ) (cid:44) α
(B U ( q )) − α ( q ) is increasing in [0 , B L ( q 0 )] .
• For any given histogram of values, neighboring histograms that maximize the increase and decrease of q can be derived from the given histogram in a straightforward manner.
Lemma 12.
For α ( · ) deﬁned as above and q deﬁned in Lemma 3, the GNMax algorithm satisﬁes the assumptions stated above.
More speciﬁcally, we can set explicitly
[Equation]
where erfc ( · ) is 1 − erf( · ) , the complementary error function.
Algorithm 3 – Local Sensitivity: use the functions B U and B L to compute (an upper bound) of the local sensitivity at a given q value by looking at the difference of α ( · ) evaluated on the bounds.
1: procedure ˜LS q ) 2:
L q q q
3: q ← B L q 0 ) 4: end if 5: return max { α (B U ( q ))
− α ( q ) , α ( q ) − α (B L ( q )) } 6: end procedure
Lemma 13.
Under the assumptions satisﬁed by the GNMax mechanism (Lemma 12), Algorithmcomputes an upper bound on the local sensitivity of α ( · ) .
Moreover, ˜LS( q ) is increasing in [0 , B L ( q 0 )] , constant in (B L ( q 0 ) , q 0 ) , and decreasing in [ q 0 , 1] .
A careful analysis of Algorithms 4 and 5 which we defer to a full version of this paper enables us to conclude the following Theorem.
Theorem 14.
Algorithm 5 computes a β -smooth upper bound on ˜LS .
Summary.
For the privacy values published in this work, we compute the smooth sensitivity and estimate that we need to add typically less than 1% noise to the ﬁnal privacy cost values before publishing them—values that are still conservative and widely used and recommended in practice and in literature.
1: procedure A T D ISTANCE D( ¯ n , d ) 2: q ← q (¯ n ) 3: if q ≤ B L ( q 0 ) then (cid:46) Need to increase q .
4: if ¯ n (1) − ¯ n (2) < 2 d then 5: q (cid:48) ← B L ( q 0 ) 6: return ˜LS( q (cid:48) ) , T RUE 7: else 8: ¯ n (cid:48) ← ¯ n + [ − d, d, 0 , . . . , 0] 9: q (cid:48) ← q (¯ n (cid:48) ) 10: if q (cid:48) > B L ( q 0 ) then 11: return ˜LS( q 0 ) , T RUE 12: else 13: return ˜ LS( q (cid:48) ) , F ALSE 14: end if 15: end if 16: else (cid:46) Need to decrease q .
17: if (cid:80) di
=2 ¯ n ( i ) <
d then 18: ¯ n (cid:48) ← [ n, 0 , . . . , 0] 19: q (cid:48) ← q (¯ n (cid:48) ) 20: return ˜LS( q ) , T RUE 21: else 22: ¯ n (cid:48) ← ¯ n + [ d, 0 , . . .
, 0] 23: for d (cid:48)
= 1 , . . .
, d do 24: ¯ n (cid:48) (2) ← ¯ n (cid:48) (2) −25:
Sort ¯ n (cid:48) in descending order 26: end for 27: q (cid:48)
← q (¯ n (cid:48) ) 28: if q (cid:48) <
q 0 then 29: return ˜LS( q 0 ) , T RUE 30: else 31: return ˜LS( q (cid:48) ) , F ALSE 32: end if 33: end if 34: end if 35: end procedure
1: procedure S MOOTH S ENSITIVITY ( ¯ n , β ) 2:
S ←3: d ←4: repeat 5: c, StoppingCondition ← A T D ISTANCE D (¯ n, d ) 6:
S ← max { S, c · e − βd } 7: d ← d +8: until StoppingCondition 9: end procedure