M AKING THE S HOE F IT : A RCHITECTURES , I NITIALIZATIONS , AND T UNING FOR L EARNING WITH P RIVACY
Anonymous authors
Paper under double-blind review
A BSTRACT
Because learning sometimes involves sensitive data, standard machine-learning algorithms have been extended to offer strong privacy guarantees for training data.
However, in practice, this has been mostly an afterthought, with privacypreserving models obtained by re-running training with a different optimizer, but using the same model architecture that performed well in a non-privacy-preserving setting.
This approach leads to less than ideal privacy/utility tradeoffs, as we show here.
Instead, we propose that model architectures and initializations are chosen and hyperparameter tuning is performed, ab initio , explicitly for privacypreserving training.
Using this paradigm, we achieve new state-of-the-art accuracy on MNIST, FashionMNIST, and CIFAR10 without any modiﬁcation of the fundamental learning procedures or differential-privacy analysis.
I NTRODUCTION
Machine learning (ML) can be usefully applied to the analysis of sensitive data, e.g., in the domain of healthcare (Kononenko, 2001).
However, ML models may unintentionally reveal sensitive aspects of their training data, e.g., due to overﬁtting (Shokri et al., 2017; Song & Shmatikov, 2019).
To counter this, ML techniques that offer strong privacy guarantees have been developed.
Notably, the differentially private stochastic gradient descent, or DP-SGD, of Abadi et al.
(2016) is an easyto-use, generally-applicable modiﬁcation of stochastic gradient descent.
In addition to its rigorous privacy guarantees, it has been empirically shown to stop the leaking of secrets (Carlini et al., 2019).
To strictly bound the impact of any training example, DP-SGD makes two changes to every gradient step: ﬁrst, each example’s gradient contribution is limited to a ﬁxed bound (in practice, by clipping all per-example gradients to a maximum (cid:96) 2 norm); second, random (Gaussian) noise of the scale of the clipping norm is added to each batch’s combined gradient, before it is backpropagated to update model parameters.
Together, these changes create a new, artiﬁcial noise ﬂoor at each step of gradient descent, such that the unique signal of any individual example is below this new noise ﬂoor; this allows differential privacy to be guaranteed for all training examples (Dwork & Roth, 2014).
Training using DP-SGD is eminently practical and in addition to privacy offers advantages such as strong generalization and the promise of reusable holdouts (Google, 2019; Dwork et al., 2015).
Unfortunately, its advantages have not been without cost: empirically, the test accuracy of differentially private ML is consistently lower than that of non-private learning (e.g., see Papernot et al.
Such accuracy loss may sometimes be inevitable: for example, the task may involve heavy-tailed distributions and adding noise will deﬁnitely hinder visibility of examples in the tails (Feldman, 2019; Bagdasaryan & Shmatikov, 2019).
However, this does not explain the accuracy loss of differentially private learning on standard benchmark tasks that are known to be relatively simple: MNIST (Yann et al., 1998), FashionMNIST (Xiao et al., 2017), CIFAR10 (Krizhevsky et al., 2009), etc.
This paper presents several new results for privacy-preserving learning that improve the state-ofthe-art in terms of both privacy and accuracy.
Signiﬁcantly, these new results stem from a single, simple observation: differentially-private learning with DP-SGD is different enough that all aspects of learning—model architecture, parameter initialization, and optimization strategy, as well as hyperparameter tuning—must be reconsidered.
To achieve the best privacy/accuracy tradeoffs, we must tune our learning strategies to the speciﬁcs of privacy-preserving learning; i.e., we must “learn to learn” with privacy.
Conversely, we concretely demonstrate how the architecture, initialization, and optimization strategy that gives the best accuracy for non-private learning can be a poor ﬁt for learning with privacy.
Instead, by revisiting our choices, we can reduce the information loss induced by clipping, limit the impact of added noise, and improve the utility of each gradient step when learning with privacy.
Our contributions facilitate DP-SGD learning as follows:
• We show how simple architecture changes, such as the use of tanh instead of ReLU activations, can improve a model’s private-learning suitability and achievable privacy/accuracy tradeoffs, by eliminating the negative effects of clipping and noising large gradients.
• We explain how high-capacity models can be disadvantageous, as well as the advantages of models with a ﬁnal, fully-connected layer that can be independently ﬁne tuned, and how both help address the curse of dimensionality and high-dimensional noise.
• We demonstrate the importance of ﬁnding good initializations, and show how this can be done with privacy using either transfer learning or weight scaling (Raghu et al., 2019).
• We show that better tradeoffs and increased wall-clock learning speeds can be achieved by tuning hyperparameters and choosing optimizers directly for DP-SGD learning.
By applying the above, we advance the state of the art for MNIST, FashionMNIST, and CIFAR10, signiﬁcantly improving upon the privacy/accuracy tradoffs from prior work.
On MNIST, we achieve 98.1% test accuracy for a privacy guarantee of ( ε, δ ) =
(2 . 93 , 10 − 5 ) , whereas the previous stateof-the-art reported in the TensorFlow Privacy library (Google, 2019) was 96.6%.
On CIFAR10, we achieve 72% test accuracy at ( ε, δ ) =
(2 . 1 , 10 − 5 ) in a setup for which to the best of our knowledge the previous state-of-the-art was achieved by Abadi et al.
at 67% accuracy.
T RAINING - DATA M EMORIZATION , D IFFERENTIAL P RIVACY , AND DP-SGD
Machine-learning models will easily memorize whatever sensitive, personal, or private data that was used in their training, and models may in practice disclose this data—as demonstrated by the attacks of Shokri et al.
(2017), Song & Shmatikov (2019), and Carlini et al.
For reasoning about the privacy guarantees of algorithms such as training by stochastic gradient descent, differential privacy has become the established gold standard (Dwork & Roth, 2014).
Informally, an algorithm can be differentially private if it will always produce effectively the same output (in a mathematically precise sense), when applied to two input datasets that differ by only one record.
Formally, a learning algorithm A that trains models from the set S is ( ε, δ ) -differentiallyprivate, if the following holds for all training datasets d and d (cid:48) that differ by exactly one record:
[Equation]
Here, ε gives the formal privacy guarantee, by placing a strong upper bound on any privacy loss, even in the worst possible case.
A lower ε indicates a stronger privacy guarantee or a tighter upper bound (Erlingsson et al., 2019).
The factor δ allows for some probability that the property may not hold (in practice, this δ is required to be very small, e.g., in inverse proportion to the dataset size).
A very attractive property of differential-privacy guarantees is that they hold true for all attackers— whatever they are probing and whatever their prior knowledge—and that they remain true under various forms of composition.
In particular, the output of a differentially-private algorithm can be arbitrarily post processed, without any weakening of the guarantees.
Also, if sensitive training data contains multiple examples from the same person (or, more generally, the same sensitive group), ε -differentially-private training on this data will result in model with a kε -differential-privacy guarantee for each person, as long as at most k training-data records are present per person.
Abadi et al.
(2016) introduced DP-SGD as a method for training deep neural networks with differential-privacy guarantees that was able to achieve better privacy and utility than previous efforts (Chaudhuri et al., 2011; Song et al., 2013; Bassily et al., 2014).
DP-SGD bounds the sensitivity of the learning process to each individual training example by computing per-example gradients { g i } i ∈ 0 ..
n − 1 with respect to the loss, for the n model parameters { θ i } i ∈ 0 ..
n − 1 , and clipping each per-example gradient to a maximum ﬁxed (cid:96) 2 norm C .
Subsequently, to the average of these perexample gradients, DP-SGD adds (Gaussian) noise that whose standard deviation σ is proportional to this sensitivity.
In this work, we use the canonical implementation of DP-SGD and its associated analysis that has been made available through the TensorFlow Privacy library (Google, 2019).
[Figure]
M ODEL A RCHITECTURES BETTER S UITED TO L EARNING WITH P RIVACY
We show here that learning with differential privacy imposes additional constraints that need to be taken into account when designing neural network architectures.
They help us control the sensitivity of learning to training examples before the clipping operation is performed in DP-SGD, thus reducing the potential negative impact of clipping on the estimated gradient direction.
3. M ODEL C APACITY
The success of neural networks is in part explained by their ability to scale to complex tasks through an increase in model capacity.
ResNets are an illustrative recent examples (He et al., 2016).
Here, we explain how additional capacity may not be beneﬁcial when learning with privacy.
One of the major challenges in training models with differential privacy is the curse of dimensionality (Bassily et al., 2014).
The accuracy of privately trained models typically degrades with the increase in the number of dimensions.
Unfortunately, strong lower bounds suggest that this dependence on dimensionality is necessary (Bassily et al., 2014).
[Table]
= 31) .
Layer
Parameters
Convolution k ﬁlters of 8x8, strides
Max-Pooling 2x
Convolution k ﬁlters of 4x4, strides
Max-Pooling 2x
Fully connected 32 units
Softmax 10 units
Consider the convolutional architecture described to the right.
With all other architectural details being ﬁxed, we can control the model’s capacity by varying the number of ﬁlters k in its two convolutional layers.
While the relationship between generalization performance and the number of parameters is not always monotonic (Neyshabur et al., 2017), we leave as future work a study of how different measures of capacity can inform the design of model architectures for private learning.
We report the model’s accuracy when trained with SGD and DP-SGD in Figure 1, both on MNIST (left) and FashionMNIST (right).
The test accuracy of models trained without privacy monotonically increases with the number of ﬁlters in their convolutional layers.
Instead, we observe an inﬂection point at about 15 ﬁlters for which models trained with privacy achieve their highest test accuracy.
Afterwards, the model’s generalization suffers as more ﬁlters are added.
There are two competing explanations of this behavior, both compatible with the lower bound stated in Bassily et al.
First, recall that DP-SGD performs a clipping operation on each per-example gradient before the average gradients is used to update model parameters; i.e., each gradient is subject to the following transformation
[Equation]
where g i is the gradient corresponding to model parameter i .
For a ﬁxed clipping norm C (corresponding to a certain, ﬁxed privacy guarantee), the quantity
[Equation]
by which individual parameters are multiplied decreases as the number n of parameters in a model increases.
That is, the more
[Figure]
parameters we have, the more likely DP-SGD is to clip the gradient (or signal) at each parameter.
This can explain the presence of an inﬂection point in Figure 1, after which learning with privacy becomes increasingly difﬁcult as capacity is increased.
Second, as the number of parameters (i.e., g i ’s) increases, the norm of the noise vector that DP-SGD must add to the gradient average to ensure privacy also increases.
This noise norm increases as (cid:112) # parameters, and introduces another source of accuracy degradation with an increased number of parameters.
Our observations may seem to contradict some of the ﬁndings in Abadi et al.
However, their limited experimental setup could offer few general lessons.
First, they reduced data dimensionality using PCA to have inputs of only 60 dimensions; second, they explored only a model architectures using a single layer perceptron with between 200 and 2 , 000 units.
Instead, our experiments involve a realistic setting where the full input is passed to a convolutional neural network with a total ofhidden layers and over 26,000 parameters.
A CTIVATION F UNCTIONS
When training a model with differential privacy, gradients computed during SGD are clipped (recall
Equation 2) to control the sensitivity of learning to training examples.
If these gradients take large values, some of the signal will be discarded as gradients are being clipped.
One way to reduce the magnitude (or at least control it), is to prevent the model’s activations from exploding.
However, a common choice of activation function in modern deep neural networks is the ReLU and, unlike other activations functions, ReLUs are unbounded.
Here, we thus test the hypothesis that replacing ReLUs with a bounded activation function prevents activations from exploding and thus keeps the magnitude of gradients to a more reasonable value.
This in turn implies that the clipping operation applied by DP-SGD will discard less signal from gradient updates—eventually resulting in higher performance at test time.
On MNIST and FashionMNIST, we train two models based off the architecture of Table 1: the ﬁrst model uses ReLU whereas the second model uses tanh 1 as the activation for its hidden layers, with other architectural elements kept identical.
In our experiments, we later ﬁne-tuned those architectural aspects (i.e., model capacity, choice of optimizer, etc.) separately for each activation function, to avoid favoring any one choice. In all cases, tanh was an improvement, as summarized in our conclusions (Section 6).
Figure 2 visualizes the privacy-utility Pareto curve (Avent et al., 2019) of the two models trained with DP-SGD.
Rather than plotting the test accuracy as a function of the number of steps, we plot it as a function of the privacy loss ε (but the privacy loss is a monotonically increasing function of the number of steps).
On MNIST, the test accuracy of the tanh model is 98 .
0% compared to 96 .
6% for the ReLU model with an identical privacy loss of ε = 2 .
For comparison, baseline tanh and ReLU models trained without privacy both achieve a test accuracy of 99 . 0% .
Similarly, on FashionMNIST, the tanh model trained with DP-SGD achieves 85 .
5% test accuracy compared to 81 .
9% with ReLUs.
The baselines on FashionMNIST are 89 .
3% for tanh and 89 .
4% with ReLUs.
To explain why a simple change of activation functions has a large impact on the model’s accuracy, we conjecture that the bounded nature of the tanh prevents activations from exploding during [Figure]
training. We thus monitored the (cid:96) 2 norm of the ﬁrst layer’s activations for our MNIST model while it is being trained in three scenarios: (a) without privacy using vanilla SGD and ReLU activations, (b) with ReLU activations and DPSGD, and (c) with tanh activations and DPSGD.
The evolution of activation norms on test data is visualized in Figure 3.
As conjectured, the activations of our ReLU network explode by a factor of 3 when training with privacy when compared to without privacy.
Switching to tanh activations brings down the norms of activations back to levels comparable with the activations of our non-private ReLU network.
I NITIALIZATIONS FOR L EARNING WITH D IFFERENTIAL P RIVACY
Because each gradient step expends some privacy budget, good initialization of learning is important; here, we consider transfer learning (Pratt et al., 1991) and weight scaling (Raghu et al., 2019).
4. I NITIALIZING FROM A P RE - TRAINED M ODEL USING T RANSFER L EARNING
Transfer learning can improve the initialization used when learning with privacy, and allow better privacy/accuracy tradoffs to be achieved.
For example, to reach reasonable accuracy ( > 80% ) on CIFAR10, a convolutional neural network may necessarily include many convolutional layers comprising several hundred-thousand parameters.
However, since convolutional layers for similar image-processing tasks are known to learn similar representations—at least in early layers—it may be possible to transfer most of these parameters from a public model, either as initializations or as frozen parameters, and subsequently train with DP-SGD.
For CIFAR10, the natural choice for such transfer is a CIFAR100 model, and this has been previously explored by Abadi et al.
[Table]
Taking the Abadi et al.
(2016) transfer learning results for CIFAR10 as a baseline, we perform new experiments using much of the same setup and the model architecture of Table 2.
As it is relatively simple, this model is a good candidate for differentially-private learning (although it reaches only 84 .
2% accuracy on CIFAR10 when all its parameters are trained nonprivately, whereas state-of-the-art models can have over 10% higher accuracy).
We performed new transfer-learning experiments based on training this model on CIFAR100 data in three different ways: trained on a total of 5000 examples from 10 classes picked at random ( Minrand-10 ); trained on 25,000 examples from a random half of the CIFAR100 classes, grouped into 10 new, evenly-sized meta classes ( Half-rand-50 ); trained on all examples and all 100 separate classes ( Max-100 ).
From each of these trained models, transfer learning was used to initialize a model to be trained on CIFAR10.
In the subsequent CIFAR10 training, all but the last layer was frozen, which simpliﬁes the learning task to that of logistic regression (but also reduces utility, with the best non-private accuracy reduced to 75% on CIFAR10).
Table 3 shows CIFAR10 privacy and accuracy resulting from ﬁne-tuning of different transferlearning models with DP-SGD.
As shown in Table 4, the results improve on those of Abadi et al.
(2016), even though they performed non-linear ﬁne-tuning of two neural-network layers, and their underlying model was able to achieve higher non-private accuracy (86%).
[Table]
This paper
Abadi et al.
( ε, acc . ) ( ε, acc . )
In addition, the results show the beneﬁts of model architectures whose ﬁnal layer can be ﬁne-tuned using logistic regression training, or other forms of convex optimization.
Such training can be made possible by including a ﬁnal fully-connected layer into a network; in additional experiments (not detailed here), the inclusion of such a layer did not harm the training of the original, source model from which transfer learning was done.
Furthermore, the number of parameters in this layer did not seem to matter much: privacy/accuracy tradeoffs remained the same, even when the layer was grown by an order of magnitude, which is consistent with what is known about differentially-private convex optimization (Jain & Thakurta, 2014).
4. I NITIALIZATION BY W EIGHT S CALING
[Figure] accuracy tradeoff Initialization by transfer learning is only applicable when a suitable public model exists whose weights can facilitate learning with privacy on sensitive data.
But, such model may not exist, and DP-SGD learning may possibly beneﬁt from other, non-standard means of initialization. We consider the Mean Var weightscaling approach of Raghu et al. and initialize
DP-SGD learning with Gaussian random parameter distributions whose layer-wise mean and variance are extracted from a seed model trained on the same sensitive input data. The weight-scaling approach does not directly transfer the parameters of an existing model; instead, just the layer-wise mean and variance are extracted, and those statistics are used to conﬁgure the
Gaussian random distributions from which a second model with the same architecture is initialized.
In the context of learning with privacy, Mean Var weight scaling can improve model initialization by transfer from one differentially-private model to another.
First, DP-SGD can be applied to train a model with standard random initialization.
From this model, per-layer mean/variance statistics can be extracted to initialize a new model of the same architecture, subsequently trained with strong privacy guarantees.
(This extraction can be done privately, although the privacy risk of summary statistics that drive random initialization should be vanishing. Following Bassily et al. ; Papernot et al.
(2018), one can use the formal framework of sub-sample and aggregate in conjunction with Propose-Test-Release (PTR) for this selection.
The algorithm ﬁrst splits the training data into disjoint subsets, and trains models independently on each of the splits. Using these trained models, the parameter is chosen via consensus voting with differential privacy. Notice that if the training data set is large, and there is a strong consensus, then the cost towards
[Figure]
e s
[Figure]
r a c y
[Figure]
[Figure]
s t
[Figure] u r a c y sgd
[Figure]
T e s t
[Figure]
r [Figure] c y
[Figure]
CIFAR privacy is very low.) The idea is that the mean and variance pairs can be obtained quickly at a modest privacy budget, but the faster convergence of the Mean Var initialized model both reduces the overall privacy budget needed for training, and mitigates the increased wall-clock time of DP-SGD.
We experiment with a relatively deep CIFAR10 convolutional model (see Appendix A), since Raghu et al.
found the beneﬁts of Mean Var initialization most pronounced for large models.
We ﬁrst trained a model using random initialization, and then did weight scaling by transferring that model’s statistics to a new model. In this proof-of-concept, both models were trained with the same noise variance ( σ = 0 . 5 ), but one could reserve a larger portion of the privacy budget for the new model. We should note that we did not directly transfer the weight statistics between corresponding layers in the original and new models.
Rather, we used the weight statistics of each of original model’s early layers of the original model for two of the layers in the new model. This gives superior performance to a one-to-one transfer; we conjecture that this is because early layers have higher variance.
Figure 4 shows the results of this experiment for some early training epochs.
Each run that used standard He random initialization (He et al., 2015) gave near identical results, achieving 37% accuracy at epoch 33.
The Mean Var initialization runs showed substantially higher variance, with the best models having 7% better accuracy at epoch 33.
These results are intriguing, and reminiscent of the lottery ticket hypothesis (Frankle & Carbin, 2019); they suggest a strategy of training a collection of Mean Var models and keeping those that show early promise.
T UNING O PTIMIZERS FOR P RIVATE L EARNING
Architectural choices presented in Section 3 control how sensitive learning is to training examples.
This helps us to learn with privacy—because it eliminates the negative effects of clipping and noising large gradients.
We now turn our attention to the training algorithm itself.
We ﬁnd that it is important to tailor algorithm and hyperparameter choices to the speciﬁcities of private learning: a batch size or learning rate that yields good results without privacy may not perform well with privacy.
5. A DAPTIVE O PTIMIZERS PROVIDE M ARGINAL G AINS WHEN L EARNING WITH P RIVACY
We ﬁrst explore the choice of optimizer, and in particular whether adaptive optimizers that leverage the history of iterates help convergence when learning privately.
We compare learning curves for DPSGD and the differentially private counterpart of Adam (Kingma & Ba, 2014), a canonical adaptive optimizer.
A qualitative analysis of Figure 5 leads to the same conclusion for all datasets (MNIST, FashionMNIST, and CIFAR10).
While DP-Adam may converge faster initially, its convergence rate eventually slows down sufﬁciently for DP-SGD to achieve comparable (if not higher) accuracy.
To explain the ineffectiveness of adaptive optimizers, we hypothesize that the iterates they accumulate during training are affected negatively by noise introduced to preserve privacy.
Indeed, while there is enough signal from the training data included in any given batch sampled early in training, later in training most training examples have a loss of zero and do not contribute to the gradients being noised.
Carrying this noise from one gradient descent step to the next to adapt learning rates therefore inadequately slows down training.
To verify this, we track the estimate of the ﬁrst moment in Adam on MNIST.
The mean absolute value of its components converges when learning without privacy (from 0.5 after the ﬁrst epoch to about 0.8 for epochs 45 through 60).
Instead, it increases steadily throughout training with privacy (from 0.5 at the ﬁrst epoch to above 1.
after 60 epochs).
Thus, choosing an adaptive optimizer (e.g., DP-Adam) is not necessary if one is interested in achieving maximal accuracy: given a ﬁxed privacy budget, ﬁne-tuning the learning rate is more important as we conﬁrm in Section 5.2.
Note that this resonates well with recent results questioning the generalization capabilities of adaptive optimizers (Wilson et al., 2017).
5. C HOOSING A ( LARGE ) B ATCH S IZE AND L EARNING R ATE
Having observed that few training examples contribute signal after the ﬁrst epochs, it is natural to ask whether increasing the size of batches could improve the noise-to-signal ratio in DP learning.
To ensure a fair comparison, we ﬁx the privacy budget ε and deduce the number of epochs we can train the model for given a desired batch size.
For instance, in Table 5, we compare models trained for 7 epochs on batches of 1 , 024 examples to models trained for 40 epochs on batches of 256 examples.
In both cases, the total privacy budget for training these models is ε = 2 . 7 .
We run a hyperparameter search to ﬁne-tune the choice of learning rate for both DP-SGD and DP-Adam.
We then compare the test accuracy achieved with small and large batch sizes.
Non-private
Differentially-private
[Table]
Hyperparameters should be tuned for DP-SGD, not SGD.
We conﬁrm that DP-Adam does not improve over DP-SGD.
Yet, this experiment shows how training for a small number of epochs at a large batch size can do comparably well to a large number of epochs at a small batch size: the wall-clock time gain is important (about 5 × ) and the cost in performance is moderate—half a percentage point. This suggests that earlier theoretical analysis (Talwar et al., 2014) also holds in the non-convex setting.
Furthermore, note how learning rates vary across the non-DP and DP settings.
C ONCLUSIONS
Rather than ﬁrst train a non-private model and later attempt to make it private, we bypass nonprivate training altogether and directly incorporate speciﬁcities of privacy-preserving learning in the selection of architectures, initializations, and tuning strategies.
This improves substantially upon the state-of-the-art privacy/accuracy trade-offs on three benchmarks, as summarized below.
Up to now, we evaluated each component (e.g., change of activation function, optimizer, etc.) individually to demonstrate its inﬂuence on private learning. Instead, here this summary table compares each approach after all hyperparameters explored in the paper have been jointly ﬁned tuned. In particular, note how even in their own individually-best setting, tanh continues to consistently outperform
ReLU with for example 98.1% test accuracy (instead of 96.6% for ReLU) on MNIST.
Dataset
Technique
Acc.
ε δ Assumptions MNIST SGD w/ tanh (not private) 99.0% ∞ -
DP-SGD w/ ReLU 96.6% 2.
10 −-
DP-SGD w/ tanh (ours) 98.1% 2.
10 −-
FashionMNIST
SGD w/ ReLU (not private) 89.4% ∞ -
DP-SGD w/ ReLU 81.9% 2.
10 −-
DP-SGD w/ tanh (ours)
[Table]
−
[Table]
CIFAR
Transfer + SGD (not private) 75% ∞ - Transfer + DP-SGD (Abadi et al.) [Table] [Table]
Transfer + DP-SGD (ours) 72%
[Table]
B E XPERIMENTAL DETAILS
We describe hyperparameters used in Sections 3,
[Table]
and 5 because they were omitted from the main body of the paper due to space constraints.
[Table]
See Figure
[Table]
[Table]
[Equation]
[Table]
[Equation]
[Table]
See Table 5 for dataset speciﬁc rates
[Table]
Table 1 (
[Table]
[Table]