L ARGE L ANGUAGE M ODELS C AN B E S TRONG D IFFERENTIALLY P RIVATE L EARNERS
A BSTRACT
Differentially Private (DP) learning has seen limited success for building large deep learning models of text, and straightforward attempts at applying Differentially
Private Stochastic Gradient Descent (DP-SGD) to NLP tasks have resulted in large performance drops and high computational overhead.
We show that this performance drop can be mitigated with (1) the use of large pretrained language models; (2) non-standard hyperparameters that suit DP optimization; and (3) ﬁnetuning objectives which are aligned with the pretraining procedure.
With the above, we obtain NLP models that outperform state-of-the-art DP-trained models under the same privacy budget and strong non-private baselines—by directly ﬁne-tuning pretrained models with DP optimization on moderately-sized corpora.
To address the computational challenge of running DP-SGD with large Transformers, we propose a memory saving technique that allows clipping in DP-SGD to run without instantiating per-example gradients for any linear layer in the model.
The technique enables privately training Transformers with almost the same memory cost as non-private training at a modest run-time overhead.
Contrary to conventional wisdom that DP optimization fails at learning high-dimensional models (due to noise that scales with dimension) empirical results reveal that private learning with pretrained language models tends to not suffer from dimension-dependent performance degradation. Code to reproduce results can be found at https: github .com/lxuechen/private-transformers
I NTRODUCTION
Machine learning systems trained on sensitive user data can be vulnerable to privacy attacks (Shokri et al., 2017; Hayes et al., 2019).
This issue is especially pressing for recent applications of large language models, as these models are capable of memorizing and reconstructing sensitive examples contained in the training data (Zhang et al., 2016; Carlini et al., 2020).
As a result of these concerns, there has been a large interest in developing methods that provide data privacy guarantees for large language models.
The standard paradigm for providing such a guarantee in machine learning is Differential Privacy (DP) (Dwork et al., 2006; 2014).
Unfortunately, DP learning has typically struggled to produce useful models when applied to large language models, resulting in models with either vacuous privacy guarantees (Dupuy et al., 2021) or performance far below non-private baselines.
This is widely attributed to the fact that the core primitive of Differentially Private Stochastic Gradient Descent (DP-SGD) (Song et al., 2013; Bassily et al., 2014;
Abadi et al., 2016) injects noise that must scale with the number of parameters, resulting in large noise levels for large language models (Yu et al., 2021b).
We tackle the problem of building performant DP language models for sentence classiﬁcation and language generation tasks with merely tens to hundreds of thousands of examples.
We pursue this goal by re-examining the performance of the baseline DP optimization algorithm for ﬁne-tuning large language models, and study how choices of hyperparameters, training objective, and pretrained models affect the performance given ﬁxed privacy budgets.
In contrast to the mainstream perception, our empirical results demonstrate that large pretrained models with hundreds of millions of parameters can be effectively and efﬁciently ﬁne-tuned to yield models with high performance with modest privacy leakage.
For text generation, the performance of our models surpasses even strong non-private baselines.
For sentence classiﬁcation, the performance of our ﬁne-tuned models
[Figure]
(Williams et al., 2018)
[Figure]
surpasses those obtained under heuristic privacy notions (Huang et al., 2020a) which do not possess formal privacy guarantees.
Figure 1 illustrates these results.
We summarize our contributions below.
(1) We show that with appropriate hyperparameters and
downstream task
objectives, ﬁne
-tuning pretrained language models with DP-SGD/DP-Adam yields strong performance for a suite of NLP tasks at privacy levels (cid:15) ∈ { 3 , 8 } .
Some of our ﬁne-tuned models outperform strong non-private learning baselines and models obtained under heuristic privacy notions.
(2) Running DP-SGD can be memory-intensive due to clipping per-example gradients.
We present ghost clipping , a memory saving technique that makes ﬁne-tuning large Transformers under DP memory efﬁcient.
Our technique generalizes the Goodfellow (2015) trick to handle sequential inputs and can be combined with a layer-by-layer clipping procedure ( Lee & Kifer, ) to enable privately ﬁtting large Transformers with almost the same memory cost as non-private training—at the cost of one additional backward pass per processed batch.
(3) We show that the dimensionality of gradient updates does not explain private ﬁne-tuning performance.
While there exist dimension-dependent lower bounds for private (convex) optimization (Bassily et al. , we ﬁnd that larger pretrained models lead to better private ﬁne-tuning results.
Moreover, parameter-efﬁcient adaptation methods that reduce the dimensionality of updates do not necessarily outperform a baseline method that ﬁne-tunes all model parameters.
Our empirical studies indicate that directly ﬁne-tuning pretrained models with DP optimization results in performant DP language models under modest privacy budgets.
This enables building practical private NLP models for a range of common tasks where privacy could be at stake.
P ROBLEM S TATEMENT
We build models for sentence classiﬁcation and language generation tasks with datasets of modest sizes under (central/global) approximate-DP (also known as ( (cid:15), δ ) -DP)
(Dwork et al., 2014).
Deﬁnition 1 ( ( (cid:15), δ ) -DP) .
A randomized algorithm M : X → Y is ( (cid:15), δ )-differentially private if for all adjacent datasets X, X (cid:48) ∈ X and all Y ⊂ Y
[Equation]
Two datasets are adjacent if and only if one can be obtained from the other by including an extra record (Mironov et al., 2019).
1 How a record is deﬁned is task dependent and will be made clear below.
Intuitively, DP algorithms ensure that random outputs obtained from similar inputs are difﬁcult to distinguish.
(cid:15) and δ are privacy leakage parameters that measure the loss of privacy and small values imply more privacy.
Unlike heuristic privacy notions (Huang et al., 2020b), DP allows for the tracking of privacy loss through the calculation of leakage parameters, and ensures privacy under composition (Dwork et al., 2014), meaning that the overall privacy loss of multiple DP algorithms releasing multiple statistics can be reasoned in a principled manner.
DP learning typically relies on DP optimizers which privatize gradients before performing updates.
The privatization step ensures that the parameter updates leak limited information about the training examples through their gradients.
Speciﬁcally, this step clips per-example gradients with a norm constraint C , and adds Gaussian noise z ∼ N (0 , C 2 σ 2 I p ) to the sum of clipped gradients.
Here, σ is the noise multiplier determined from the privacy budget ( (cid:15), δ ) , number of gradient updates S , and sampling rate q = BN for a batch size of B and a dataset with N examples.
Intuitively, clipping individual gradients ensures that each example has bounded inﬂuence on the parameter update, whereas noising the gradient prevents exact tracing of particular examples.
The noise being isotropic implies that larger models would experience heavier noise per update, as the norm of the p -dimensional Gaussian (cid:107) z (cid:107) 2 scales as Cσ √ p .
This is widely believed to be the cause for DP optimization performing poorly at training high-dimensional deep learning models (Kamath, 2020).
Our starting point for building DP language models is (public) pretrained models.
Pretrained language models tend to contain general knowledge of language (Manning et al., 2020) and thus should make the downstream private learning problem easier.
We ﬁne-tune these models with DP-Adam (Abadi et al., 2016; Kingma & Ba, 2014) (see Appendix A for details) and track privacy loss through R´enyi
DP (Mironov, 2017), but also report the converted (cid:15) from a Gaussian DP CLT (Dong et al., 2019) and from accurately composing tradeoff functions via fast Fourier transform (Gopi et al., 2021).
We consider privacy levels (cid:15) ∈ { 3 , 8 } and δ = 1 / 2 |D train | throughout for a training set of size |D train | (see Appendix B for further details).
We tune hyperparameters on a text generation task (E2E; introduced below) and transfer these to remaining tasks.
We outline two broad classes of NLP problems considered in this paper and deﬁne what constitutes a record below.
Sentence Classiﬁcation.
The goal is to learn a model that classiﬁes sentences into one of a few categories.
For these tasks, each example/record consists of input sentences and a label to be predicted.
We ﬁne-tune models of various sizes in the BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) families, as these models are known to work well for sentence classiﬁcation.
Language Generation.
The goal is to learn a model that generates natural language sentences given some context.
For table-to-text generation tasks such as E2E (Novikova et al., 2017) and
DART (Nan et al., 2020), each example/record in the training data consists of a pair of table entry and corresponding text description to be predicted.
For a dialogue generation task such as PersonaChat (Zhang et al., 2018), each example/record consists of metadata, a dialogue history, and a response to be predicted.
We ﬁne-tune GPT-2 (Radford et al., 2019) of various sizes for these problems, as this model family is known to work well for text generation.
E FFECTIVE D IFFERENTIALLY P RIVATE F INE -T UNING
By studying the impact of hyperparameters and choice of ﬁne-tuning objective, we demonstrate that the performance of the DP-Adam baseline can be substantially improved, even matching some strong non-private learning results.
Our analyses reveal common failure modes when straightforwardly applying DP optimization and explain poor results reported in past works that consider these baselines.
3. H YPERPARAMETER T UNING
[Table]
[Figure]
s t s e t
B L E U ( = 3 )
DP optimization is sensitive to the choice of hyperparameters (Papernot et al., 2019).
Our experiments suggest that performance can vary from being close to trivial with ill-chosen hyperparameters to near previous state-of-the-art with appropriately chosen ones.
As a consequence, we present simple but effective guidelines on setting the most important hyperparameters.
Unless otherwise stated, the unmentioned hyperparameters are set to defaults documented in Appendix H.
3.1. B ATCH S IZE , L EARNING R ATE & T RAINING E POCHS
Our experiments suggest that batch size is one of the most important hyperparameters to set correctly, and the dependence of the optimal batch size on learning rate and training epochs makes its selection complex.
We ﬁrst describe batch size selection in realistic, compute-bound settings and then describe how the complexity of identifying the optimal batch size in these situations arise due to constraints on the number of training epochs.
Fixed Training Epochs E .
We ﬁrst describe a realistic situation in which there is a constraint on the total amount of compute budget available.
For the case of DP-SGD, this compute budget constraint often corresponds to a constraint on the number of examples that are processed by SGD.
In this ﬁxed training epoch setting, the learning rate and batch size jointly affect performance, since using larger batches implies performing fewer gradient updates.
To study this joint inﬂuence empirically, we ﬁne-tune GPT-2 on the E2E dataset for table-to-text generation with DP-Adam at (cid:15) = 3 with various batch sizes and learning rates.
Figure 2 shows that the best performing models are obtained with both a large batch size and large learning rate.
Using a small learning rate together with a small batch size yields considerably worse results.
Note a seq2seq baseline achieves a test BLEU of ~ 65 without privacy here (Wiseman et al., 2018).
Recall that in the non-private world, pretrained language models are typically ﬁne-tuned with small batch sizes and small learning rates with Adam (bottom left panel in Figure 2).
This implies that na¨ıely ﬁne-tuning pretrained language models privately using the non-private setup would result in more performance degradation than necessary.
Recently, Tram`er & Boneh (2020) studied how the batch size and learning rate jointly affect learning private image classiﬁers while holding other hyperparameters ﬁxed.
They heuristically suggested a linear scaling rule : Scaling the learning rate together with the batch size by the same constant should yield models with almost the same performance.
However, Figure 2 indicates that this fails to hold consistently as it falsely predicts that large batch and high learning rate (top right most entry) would have equal performance to small batch and low learning rate (bottom left entry).
We further explain why linear scaling fails for small batches in Appendix D.
Fixed Update Steps S .
In the ﬁxed epoch setting, we saw that the optimal batch size setting was complex due to the trade-off between batch size and number of gradient updates.
We now show that the complexity of setting batch sizes arises almost entirely from this tradeoff by considering a different setting, where the total number of gradient updates (rather than epochs) is ﬁxed.
In this case, using larger batches implies training for more epochs, and we ﬁnd that using larger batch sizes almost always results in better performance at a given privacy budget (at the cost of processing more examples and using more compute), once other hyperparameters are ﬁxed.
We provide an explanation of this by introducing the idea of an effective noise multiplier σ eff =
σq = σNB .
Recall the noise multiplier σ is determined from the privacy budget ( (cid:15), δ ) , update steps S , and sampling rate q .
In addition, recall the form of the privatized gradient ¯ g in DP-SGD/DP-Adam:
[Equation]
where ∇L i is the gradient of the i th example in a batch of B examples and Clip ( v, K ) clips the vector v by the norm constraint K .
We observe that for moderately large batches, the signal-to-noise ratio r = (cid:107) (cid:101) g (cid:107) 2 / (cid:107) z (cid:107) 2 is mainly controlled by the batch size through the effective noise multiplier:
The signal term (cid:101) g tends to concentrate quickly due to being an average of bounded vectors, whereas the effective noise multiplier σ eff could vary signiﬁcantly as the batch size B changes.
Figure 3 (a) shows that the effective noise multiplier decreases as batch size increases.
In addition, Figure 3 (b) plots the average signal-to-noise ratio r over the ﬁrst 30 gradient updates against the ﬁnal model’s performance on E2E and demonstrates that large batches (up to a threshold) lead to both increased signal-to-noise ratio at the beginning of training and better performance at the end of training.
These ﬁndings additionally resonate with and explains recent empirical successes of private pretraining (Anil et al., 2021).
3.1. C LIPPING N ORM
DP optimization is known to be sensitive to the choice of clipping norm.
Since the scale of noise depends on this clipping norm (recall its standard deviation is Cσ ), picking the threshold C much larger than the actual gradient norm implies more noise is being applied than necessary.
In practice, we have found that a small clipping norm which enforces almost all gradients to be clipped throughout training leads to the best performing models (see Figure 8 in Appendix H).
3. I MPROVING THE TASK ALIGNMENT HELPS PRIVATE LEARNING
Our ﬁne-tuned models on language generation tasks worked well since the pretraining objective and downstream task are aligned : Both involve predicting sequences of tokens.
This alignment simpliﬁed
[Figure]
the task and beneﬁtted private learning.
While pretrained models are naturally aligned for language generation, it is much less so for classiﬁcation tasks.
The standard approach for adapting language models for classiﬁcation involves stacking a freshly initialized network on top of the encoding of the special [CLS] token and jointly optimizing all parameters (Devlin et al., 2018).
This workﬂow introduces a discrepancy between pretraining and ﬁne-tuning:
Pretraining predicts masked out words from a large vocabulary whereas ﬁne-tuning predicts integer labels.
To eliminate the discrepancy, we instead consider learning to predict the missing word during ﬁnetuning for classiﬁcation.
For example, for sentiment classiﬁcation, we reframe the problem as ﬁlling in the [MASK] token in the sequence “ <INPUT> .
It is [MASK] .”
and compare the probabilities of words “awesome” and “terrible”.
This text inﬁlling task is almost exactly the procedure used for pretraining masked language models, and recent works have demonstrated its effectiveness for knowledge probing (Petroni et al., 2019), few-shot learning (Gao et al., 2020) and multi-task ﬁnetuning (Wei et al., 2021).
On SST-2, we found that using the generic template as described above already improved private ﬁne-tuning performance by 3 ∼ 5% across different settings.
Table 1 (to be presented) contains additional results and Appendix E includes analyses on choices of label words.
G HOST C LIPPING : C LIPPING WITHOUT PER - EXAMPLE GRADIENTS
DP-SGD has high memory overhead due to clipping per-example gradients.
Na ¨ ıvely implemented, this step instantiates a giant gradient vector for each example during optimization and can be prohibitively expensive.
For example, Hoory et al.
(2021) pretrained BERT with DP optimization and reported memory issues when using the large batches necessary to achieve high performance.
A time-costly solution to the memory problem is micro-batching: Split large batches into multiple smaller ones and aggregate the results after processing each small batch individually (Tram`er &
Boneh, 2020).
This solution, however, is unlikely to be sufﬁcient as neural language models become larger and ﬁtting a few copies of the gradient in memory can be difﬁcult.
Lee & Kifer (2020) observed that per-example gradients need not be instantiated at all, if the goal is to sum the clipped gradients.
They presented a clipping procedure that only instantiates the per-example gradient for parameters of a single layer in the model one at a time, as opposed to the entire model at once, at the cost of an extra backpropagation pass per processed batch.
Unfortunately, we ﬁnd this trick to be still insufﬁcient for sequence models such as Transformers (Vaswani et al., 2017), as the memory requirement for per-example gradients of embedding layers and language modeling heads can be costly.
We extend the Lee & Kifer (2020) approach such that training Transformers with DP optimization can have almost the same memory consumption as non-private training.
Unlike their approach, our extension avoids instantiating the per-example gradient even for individual linear layers.
We call this approach ghost clipping , as the per-example gradient is the ghost that never explicitly appears.
We anticipate this extension to be useful for both privately ﬁne-tuning and pretraining large Transformers.
4. T HE MEMORY TRICK BY L EE & K IFER (2020)
Per-example gradient clipping is easy if we know per-example gradient norms.
In this case, we ﬁrst compute the scaling factor c i = min(1 , C / (cid:107)∇L i (cid:107) 2 ) , where C is the clipping threshold and L i is the loss associated with the i th example.
Then, we perform the usual backward pass with the reweighted scalar loss (cid:80)
i c i L i .
This procedure gives us the sum of clipped gradients.
Under this setup, the difﬁculty is computing the per-example gradient norm (cid:107)∇L i (cid:107) 2 .
We emphasize two technicalities that enable computing this quantity without instantiating the full per-example gradient ∇L i .
First, for a typical neural net layer l with parameters W ( l ) (without parameter sharing), the perexample gradient w.r.t.
parameters can be easily computed using the input to the layer a ( l ) and the gradient of the loss w.r.t.
the output g ( l ) , both of which are available during backpropagation.
Second, for a large vector formed by concatenating several small vectors u = [ u 1 , . . .
, u k ] , its Euclidean norm is simply the norm of the vector of norms, i.e. (cid:107) u (cid:107)
= (cid:107) ( (cid:107) u 1 (cid:107) 2 , . . .
, (cid:107) u k (cid:107) 2 ) (cid:107) 2 .
The second observation means that computing the per-example gradient norm (cid:107)∇L i (cid:107) 2 can be done by computing the per-example gradient norms for individual layers of the neural net (cid:107)∇ W (1) L i (cid:107) 2 , . . .
, (cid:107)∇ W ( L ) L i (cid:107)one at a time ( L is layer count).
Moreover, the ﬁrst observation implies that the norms for each layer can be computed using quantities freely available to a typical backward pass.
Overall, the per-example gradient norm of any network without parameter sharing can be computed in a layer-by-layer fashion with only one per-example gradient tensor for a single layer being instantiated at any time.
4. G HOST CLIPPING FOR T RANSFORMERS WITH SEQUENTIAL DATA
The trick by Lee & Kifer (2020) still requires instantiating the per-example gradient of individual layers (although not simultaneously).
This can be problematic in terms of memory for Transformers with large embedding layers.
Here, we present a specialized procedure for computing the perexample gradient norm for linear and embedding layers when they are applied to sequential data.
This procedure reduces memory footprint and can be viewed as a generalization of the Goodfellow (2015) trick that additionally handles sequential inputs.
Let a ∈ R B × T ×
d be the input to a linear layer with weight matrix W ∈ R p × d , and s ∈ R B × T × p be the output with s i,j
=
Wa i,j .
Let g ∈ R B × T × p be the gradient of the loss w.r.t.
the output s .
Here,
T is the number of time steps in the input, and we omitted biases for simplicity.
Simple calculation shows that the per-example gradient is the product of two matrices:
[Equation]
Since the per-example gradient norms are the end goal, the per-example gradients {∇ W L i } Bi = themselves need not be instantiated explicitly.
More precisely, we observe that the squared perexample gradient norm for this layer (cid:107)∇
[Equation]
L i
[Equation]
2F obeys the following identity:
[Equation]
See Appendix F for a derivation.
Implemented with common primitives in machine learning libraries, (2) has a memory complexity of order
O
( BT 2 )
when a i a (cid:62)
i , g i g (cid:62) i ∈ R T
× T are instantiated, 56 as opposed to O
( Bpd ) in the na ¨ ıve approach which goes through instantiating (1).
The memory efﬁciency of this procedure is exempliﬁed with off the shelf pretrained language models, most of which have large embeddings.
For instance, for GPT-2, d ≈ 50 , 000 and p = 768 for the embedding layer, and the context window T ≤ 1024 .
8 Our method in theory reduces the memory cost of this large embedding layer by at least a factor of 22 .
In practice, we also observe signiﬁcant savings, since embedding layers can be a major source of memory spending for training large language models.
9 To stress-test ghost clipping, we compare it with 4 baselines:
The PyTorch package
Opacus that implements DP optimization by instantiating per-example gradients, the approach by
Lee & Kifer (2020), non-private training in PyTorch , and na¨ıve DP optimization implemented in 3 For GPT-2, per-example gradients w.r.t.
the embedding for ten examples alone occupy ~ 1.5GB of memory.
An embedding layer is essentially a linear layer:
The embedding lookup operation applied to indices is equivalent to a matrix multiplication of the embedding matrix with one-hot encoded indices.
More sophisticated solutions may even avoid instantiating a i g i i entirely by trading in more run-time.
Custom CUDA kernels are likely needed to make these solutions fast in practice
While there are alternative approaches for reducing the memory footprint of embedding layers during training, these methods tend to introduce extra hyperparameters that require tuning and privacy spending.
JAX with jit and vmap enabled.
We include the JAX baseline since recent studies show that DP optimization can be made cheap through compiler optimization (Subramani et al., 2020).
Figure 4 (a) shows that for typical inputs, our technique is the most memory friendly and allows ﬁtting batches almost as large as those in non-private training.
Since ghost clipping allows us to ﬁt larger batches but with a run-time penalty, a natural question is whether it improves throughput with the use of larger batches.
Figure 4 (b) shows that while ghost clipping only provides minor gains compared to
Opacus for smaller models, it allows processing ~ 10% more examples compared to the approach by Lee & Kifer (2020) for ﬁtting GPT-2-large, a model that neither Opacus or JAX could handle.
See Appendix G for the setup of these experiments.
[Figure]
L OW DIMENSIONAL UPDATES ARE NOT NECESSARILY BETTER
Since the norm of the noise added to gradients in DP scales with dimensionality, it is natural to ask whether privatizing and updating fewer parameters would result in improved performance.
We decompose this question into two aspects: (1) Do smaller pretrained models lead to better private ﬁne-tuned performance, and (2) do parameter-efﬁcient adaptation methods designed with a reduced dimensionality of updates outperform full ﬁne-tuning?
Our experiments below show that the answer to both questions is negative.
Reported numbers in this section are averaged over three seeds.
5. L ARGER P RETRAINED M ODELS RESULT IN BETTER PERFORMANCE
We observe that larger pretrained models lead to better private ﬁne-tuned performance.
Speciﬁcally, we fully ﬁne-tune four sizes of GPT-2 models (for language generation) and three sizes of
BERT/RoBERTa models (for sentence classiﬁcation) at the same privacy budget with DP-Adam and compare their performances.
Since the performance of DP optimization heavily depends on hyperparameter choices, we need to ensure that our hyperparameters are not particularly favoring larger models.
We thus tune hyperparameters on the smallest model and then reuse the same hyperparameters for all ﬁne-tuning workloads.
Figure 1 from earlier demonstrates gains on E2E and MNLI from model scaling, and we ﬁnd similar improvements on 5 additional tasks (deferred to Figure 5 in
Appendix C).
5. F ULL FINE - TUNING WITH DP-A DAM MATCHES STATE - OF - THE - ART
There is a range of lightweight ﬁne-tuning methods that reduce the dimensionality of updates, including some that are designed for DP (Yu et al., 2021c).
Do methods that optimize fewer parameters lead to better results under DP even if they perform similarly non-privately?
Empirical results suggest otherwise and that full ﬁne-tuning is a strong baseline that even matches specialized low-dimensional DP learning methods for both classiﬁcation and generation.
Below, we study the two sets of tasks separately.
For completeness, all experimental details are in Appendix K.
Sentence Classiﬁcation.
We study DP ﬁne-tuning on tasks from the GLUE benchmark that have more than 10 k training examples (MNLI, QQP, QNLI, and SST-2), following the experimental setup of Yu et al.
(2021c).
The associated datasets have modest sizes: SST-2 and QNLI have 60k+ and 100k+ training examples, respectively.
MNLI and QQP each contains less than 400k examples.
Table 1 shows that using larger pretrained models and the text-inﬁlling objective generally improve classiﬁcation accuracy.
We compare full ﬁne-tuning with reparameterized gradient perturbation (RGP)
(Yu et al., 2021c), as it is the state-of-the-art for DP ﬁne-tuning on sentence classiﬁcation at the time of writing.
The method is designed to privatize gradients projected onto low dimensional subspaces and was motivated to reduce DP noise in high-dimensional models.
We note that full ﬁne-tuning with the text inﬁlling objective outperforms well-tuned RGP on all tasks despite being the simplest baseline.
Computationally, while RGP is faster per-update, it requires more than 3 times as many epochs as full ﬁne-tuning – overall, both methods comparable in terms of wall time
[Table] [Table]
Appendix
Tables 7 and 8 in Appendix J contain the full result on E2E and DART and conﬁrm on par with or outperforms others methods that execute gradient update in low dimensional spaces.
Results are on E2E from ﬁne-tuning GPT-2.
[Table]
Chit-Chat Dialog Generation.
We stress-test full ﬁne-tuning under DP with the task of chitchat dialog generation.
This task has the distinct challenge that the response space is intrinsically diverse (Li et al., 2015; Gao et al., 2018) since human conversations can be informal and noisy (Zhang et al., 2019).
Moreover, dialog datasets are usually formed with user data which may contain sensitive information.
We use the Persona-Chat dataset (Zhang et al., 2018) as a testbed and build off a processed version that has ~ 130k training entries.
Each entry contains a dialog history, persona descriptions of the respondent, and the response.
We ﬁne-tune GPT-2, GPT2-medium
[Table]
and DialoGPTmedium on this dataset both privately and non-privately by training to predict the response with the dialog history and persona description.
We report the F1 score and perplexity on the validation split, and human evaluated quality scores of generations.
Table 3 shows that private models have strong performance.
In particular, ﬁne-tuned DialoGPT-medium at (cid:15) = 8 beats the (non-private) winning entry of the ConvAI2 challenge (Dinan et al., 2019) on perplexity and has a human evaluation rating that is close to non-private models.
Samples from our private models can be found in Appendix O
[Table]
[Table]
R ELATED W ORK
Private NLP.
The privacy-preserving NLP space is largely divided by whether or not a formal notion of privacy is considered.
McMahan et al.
(2017) successfully trained small word-level RNNs with 1.35 million parameters in a federated setting with more than 700k users under a global DP guarantee of ( (cid:15), δ ) = (4 . 6 , 10 − 9 ) .
Ramaswamy et al.
(2020) train production grade next-word prediction models using DP-FedAvg with millions of users.
Qu et al.
(2021) studied ﬁne-tuning
BERT for language understanding tasks under local DP.
Kerrigan et al.
(2020) presented initial results that public pretraining is helpful for downstream DP ﬁne-tuning.
However, they did not attempt ﬁne-tuning large pretrained models with DP-SGD.
Bommasani et al.
(2021) brieﬂy commented on the possibility of achieving cheaper private learning by ﬁne-tuning large pretrained language models.
Anil et al.
(2021) pretrained BERT under global DP on datasets with hundreds of millions of examples.
Dupuy et al.
(2021) studied private BERT ﬁne-tuning on datasets of utterances, but reported results with (cid:15) on the order of at least 100.
Orthogonally, many works considered training language models that satisfy empirical notions of privacy (Xu et al., 2021; Coavoux et al., 2018; Mireshghallah et al., 2021; Melamud & Shivade, 2019).
Our work is distinct from all works mentioned above in that we study ﬁne-tuning large language models (with hundreds of millions of parameters) under global DP with stringent guarantees ( (cid:15) ∈ { 3 , 8 } ) on smaller datasets (much less than a million examples).
S COPE AND L IMITATIONS
We presented strategies for ﬁne-tuning large pretrained language models under DP for a wide range of NLP tasks.
For researchers and practitioners working on private NLP, our empirical results suggest that DP ﬁne-tuning with a proper setup is a competitive baseline that is worth trying before prematurely shifting to less formal notions of privacy which have not stood against the test of time.
Below we list some limitations and future directions.
Pretraining vs Private Learning.
Our model scaling results suggest that using larger pretrained models improves performance.
This argument, however, is dependent on the particular choice of pretrained models.
How pretraining helps private learning and whether better pretrained models for private learning could be built are interesting future avenues.
Scaling Laws for Private Learning.
While scaling laws (Kaplan et al., 2020) for non-private deep learning have become prevalent, we are unaware of a case study in the private realm.
Studies on how the dimensionality of models (and pretraining) generally affect private deep learning in precise quantitative terms will likely be a useful tool in trading off compute budget and model quality.
E THICS S TATEMENT
While the present paper studies private NLP, all experiments performed herein are based on publicly available datasets for reproducibility purposes.
The paper empirically studies various behaviors of (mostly existing) algorithms, thus likely does not introduce any new ethical or societal concerns.
The present paper does not introduce any new dataset.
R EPRODUCIBILITY
All experiments in the paper are based on publicly available datasets.
Links to these datasets are included in the main text and appendices.
Hyperparameters necessary for reproducing our experiments are documented in Appendix I and H. Code for the ghost clipping technique can be found at https://github.com/lxuechen/private-transformers . A CKNOWLEDGMENTS
We thank Xiang Lisa Li for help on reproducing non-private preﬁx-tuning results.
We thank Da Yu for discussions and help on reproducing results for the RGP method. We thank Abhradeep Guha Thakurta and Rishi Bommasani for helpful discussions. We thank members of the Stanford statistical machine learning group for comments on early versions of the abstract.
We thank Guodong Zhang and Mayee Chen for comments on an early draft.
We thank Stanford HAI for a Google Cloud Credits
Grant.
XL is supported by a Stanford Graduate Fellowship.
A DP-A DAM
We use DP-Adam throughout.
DP-Adam works just like regular Adam (Kingma & Ba, 2014) but performs updates and moment accumulation with privatized gradients.
The gradient privatization part is the same as that performed in DP-SGD (Song et al., 2013; Abadi et al., 2016).
Seemingly uncommon, DP-Adam is used in many private ML libraries.
10 To determine the noise multiplier, we account privacy through R´enyi differential privacy (RDP) (Mironov, 2017; Mironov et al., 2019).
For completeness, we include the pseudocode for DP-Adam below.
1: Input: Data D = { x i } Ni =1 , learning rate η , noise multiplier σ , batch size B , Euclidean norm threshold for gradients C , epochs E , initial parameter vector θ 0 ∈ initial moment estimates m 0 , v 0 ∈ R p decay rates β β ∈
2: for t ∈
[ E N / B ] do 3:
Draw a batch B t via Poisson sampling; each element has probability B / N of being selected 4: for x
i ∈ B t do 5: g t ( x i ) ← ∇
L (
) , ˜ g t ( i ) ← g t ( x i ) min( , C (cid:107) g t ( x i (cid:107)
) 6: end for 7: z t ∼ N (0 , σ 2 C 2 I p ) 8: ¯ g t = 1 B (cid:16)(cid:80)
Ni
=1 ˜ g t ( x i ) + z t (cid:17) 9: θ t
1 , m t +1 , v t +1 ← AdamUpdate ( θ t , m t v t , ¯ g t , β β γ ) 10: end for 11: return t , β 1 , β 2 , γ 2: m t +1 ← β 1 · m t + (1 − β 1 ) · ¯ g t , v t +1 ← β 2 · v t + (1 − β 2 ) · ¯ g t3: (cid:98) m t +1 ← m t +1 / (1 − β t 1 ) , (cid:98) v t +1 ← v t +1 / (1 − β t 2 ) 4: θ t +1 ← θ t − α · (cid:98) m t [-+
-]+1 / (cid:16)(cid:112)(cid:98) v t +1 + γ (cid:17) 5: return θ t +1 , m t +1 , v t +
B P RIVACY A CCOUNTING
We train all models under approximate-DP (Dwork et al., 2014), and we view two datasets as being adjacent if and only if one can be obtained from the other by including an extra record (Mironov et al., 2019).
Instead of accounting the privacy loss with Moments Accountant (Abadi et al., 2016), we perform computation through ( i ) R
´ enyi DP (Mironov, 2017; Mironov et al., 2019), ( ii ) Gaussian DP with an associated central limit theorem (Dong et al., 2019), and ( iii ) numerically composing tradeoff functions via fast Fourier transform (Gopi et al., 2021; Koskela et al., 2020).
All approaches are improvements over the Moments Accountant.
Accounting loss with R´enyi DP provides strict upper bounds on the actual privacy leakage but may result in loose bounds.
Accounting loss with Gaussian
DP and its central limit theorem, although asymptotically exact, only provides approximations to the actual loss under a ﬁnite number of compositions (Dong et al., 2019, Theorem 3.4).
Gopi et al.
(2021) observed that accounting loss with GDP and its CLT results in underestimation and proposed to numerically compose tradeoff functions resulting in both upper and lower bounds on the actual leakage (cid:15) .
We therefore also report the converted (cid:15) with the approach by Gopi et al.
(2021) using their code.
Given the noise multiplier σ , sampling rate q , number of steps S , and failure constant δ , (cid:15) can be computed via ﬁrst computing the R ´ enyi DP leakage and then converting it to approximate DP.
When a privacy spending (speciﬁed by a set of given (cid:15) and δ ) is prescribed, we can numerically invert the above procedure to obtain a suitable σ for noisy optimization.
This is what we do throughout all experiments.
For completeness, given the σ chosen as above, we also report the leakage estimated by going through the central limit theorem in Gaussian DP (Bu et al., 2020).
Model selection from hyperparameter tuning on private training and validation data incurs extra leakage (Liu & Talwar, 2019; Chaudhuri & Vinterbo, 2013; Papernot & Steinke, 2021).
We perform tuning only on the E2E task and reuse almost the exact hyperparameters for remaining tasks.
Note the general strategy of tuning hyperparameters on a separate (public) dataset and thereafter transfer has been applied to train private production language models (Ramaswamy et al., 2020).
C A DDITIONAL R ESULTS ON M ODEL S CALING
We repeat the model scaling experiments from Figure 1 on the other tasks considered in the paper.
Figure 5 shows that the trend that larger and better pretrained models lead to improved private ﬁnetuned performance holds consistently across all tasks.
TextHide numbers based on the TextHide intra formulation (Huang et al., 2020a).
[Figure]
D W HEN AND WHY DOES LINEAR SCALING FAIL ?
Recall Tram`er & Boneh (2020) suggested that the following simple rule approximately holds in private learning: Scaling the learning rate together with the batch size by the same constant yields models with almost the same performance.
Note that their experiments on MNIST, Fashion-MNIST, and CIFAR-10 used only batch sizes in { 512 , 1024 , 2048 , 4096 } .
These values are fairly large from a non-private learning perspective.
Indeed, our experiments on E2E suggest that this rule does not generalize to batch sizes that are too small (sampling rates q = B / N < 2 − 8 ).
We provide an explanation by noting that a core assumption which the linear scaling rule depends on fails to hold for small batch sizes.
This assumption is that given a privacy budget, a “square-root” relationship holds between the noise multiplier and the sampling rate (see also (Tram`er & Boneh, 2020, Claim D.1)).
For instance, Tram`er & Boneh (2020) showed that σ ≈ c √ q when q ∈ [2 − 7 , 1] for some constant c .
Our numerical estimates show that this relationship fails to hold for small q – it underestimates the true noise multiplier σ that would be obtained with numerical computation.
Figure 6 provides an illustration for ( (cid:15), δ ) =
(3 , 10 − 5 ) when the sample size N = 50 k and number of training epochs E = 50 .
[Figure]
E HOW DOES THE CHOICE OF LABEL WORDS AFFECT SENTENCE CLASSIFICATION
Recall in Section 3.2 we cast sentence classiﬁcation as ﬁlling in the missing word among K candidates for a K -way classiﬁcation problem.
Since a label word could be mapped to multiple possible words, we study how the choice of label words affect performance.
We again use the sentiment classiﬁcation task SST-2 as a testbed.
Figure 7 shows the effect of varying the label word, where we measure the alignment between the label word and the downstream task by the zero-shot performance of the inﬁlling task (x-axis).
We ﬁnd that increasing the task alignment of the label words alone improves performance by 1 − 2% (orange curve), which is in contrast to the non-private setting, where choice of label words do not affect performance in statistically signiﬁcant ways (blue curve).
[Figure]
F D ERIVATION OF THE F ROBENIUS NORM IDENTITY
Recall a ∈ R B × T × d is the input to a linear layer with weight matrix W ∈ R p × d , and g ∈ R B × T × p is the gradient of the loss w.r.t.
the output.
The identity follows from trivial algebra:
[Equation]
Note that when T = 1 , the identity takes the form of
[Equation]
This is exactly what is used in the Goodfellow (2015) trick.
G P ROTOCOL FOR EXPERIMENTS IN S ECTION 4.
For these experiments, we used mock examples with the same format as examples in the E2E dataset.
We created mock input sequences of length 100 , as this length is almost the maximum length of examples in the actual E2E training set.
Our JAX implementation is adapted from a codebase used in the work by Subramani et al.
and utilizes the package flaxmodels for loading pretrained models.
12 The Opacus (Yousefpour et al., 2021) baseline is based on version 0.14.0 of the library, where for a fair comparison, we also optimized the implementation of the privacy engine by replacing all einsum operations with basic primitives that manipulate tensors directly.
We found certain einsum steps to cause large and unnecessary speed and memory overheads.
To identify the maximum batch size that each approach could use, we ran binary search over the range of possible batch sizes until the upper bound matched the lower bound and that OOM did not occur.
To estimate the throughput of each private method, we compared them to non-private training ﬁrst.
Speciﬁcally, for a pairwise comparison, we took the maximum batch size for non-private training and some private method, and computed the least common multiple as a batch size to perform updates with.
Using the least common multiple batch size ensures that both methods will process exactly the same number of examples and perform the same number of updates.
By estimating the time elapse of these methods for performing a ﬁxed number of updates, we obtained throughputs for non-private training and the private method.
This gives us the relative throughput of the private method with non-private training as the reference.
For methods implemented in PyTorch , the time elapse was recorded with torch.profiler .
When estimating the time elapse for a training procedure, we ﬁrst performed 3 gradients updates as a warm up process before taking the actual steps which will be timed.
In particular, this eliminates the time that JAX uses for compiling computation graphs with vmap and jit .
All experimental runs in the section were under full precision.
H D ETAILS AND ADDITIONAL RESULTS FOR S TUDIES IN S ECTION 3.
[Table]
calculated numerically so that a DP budget of ( (cid:15), δ ) is spent after E epochs
[Figure]
[Table]
I H YPERPARAMETER S EARCH R ANGES FOR EXPERIMENTS IN S ECTION
We compare different adaptation methods by reporting task speciﬁc metrics on the test split using hyperparameters that maximize validation BLEU on E2E. For sentence classiﬁcation tasks, we reused the same hyperparameters, except for the number of training epochs and batch size.
For SST-2, we reused the same batch size as for private E2E ﬁne-tuning, and the number of epochs exactly as in typical non-private ﬁne-tuning for SST-2 (number of epochs equals to 3 in this case).
For remaining classiﬁcation tasks, we use a batch size such that the sampling rate is the same as for SST-2, and a number of training epochs that is roughly proportional to the dataset size.
Appendix L outlines why we transfer the sampling rate as opposed to the batch size.
We list the range of hyperparameters that we searched over for each individual adaptation method on E2E considered in the paper.
Preﬁx-tuning has two additional hyperparameters: the length of the preﬁx and the dimensionality of the hidden layer.
We set these to the default used by Li & Liang (2021) (10 for the former and 512 for the latter).
For Adam, we use the default hyperparamaters set by PyTorch (Paszke et al., 2017).
[Table]
[Table]
J F ULL R ESULTS FOR EXPERIMENTS IN S ECTION 5.
[Table] [Table]
K D ETAILS FOR EXPERIMENTS IN SECTION 5.
Sentence Classiﬁcation.
Results for RGP in Table 1 are taken from documented numbers in their released codebase.
These results are under the DP guarantees of ( (cid:15), δ ) =
(3 , 10 − 5 ) or ( (cid:15), δ ) =
(8 , 10 − 5 ) .
These guarantees are strictly looser than our guarantees which are based on δ = 1 / 2 |D train | (recall the smallest dataset in this cohort of tasks has 60 k+ records).
The RGP numbers in Table 1 are higher than those reported in their paper (Yu et al., 2021c), since the latter numbers are not based on ﬁne-tuning the ofﬁcial RoBERTa models.
Table-To-Text Generation.
To evaluate models trained on E2E and DART, we evaluate generations from models obtained with beam search with a beam size of 5 .
For evaluation, we run the ofﬁcial pipeline for E2E, 14 and the pipeline used in the GEM benchmark (Gehrmann et al., 2021) for DART.
Chit-Chat Dialog Generation.
We built off Huggingface’s codebase of the winning entry of the
ConvAI2 competition, 16 17 and used their preprocessed training set with the minor modiﬁcation of truncating the number of training examples to be a multiple of the batch size.
The original ConvAI competition is aimed at advancing research on building engaging chatbots and also requested models to predict the mostly likely response given a list of candidates.
The challenge included hits@ as part of its suite of automatic metrics.
For simplicity, we skip this step of predicting the most likely response for both training and evaluation.
Results for the entry HuggingFace (ConvAI winner) in Table 3 are taken from the ofﬁcial validation set leader board.
18 Our reimplementation of
HuggingFace’s submission uses the released code for their winning entry, ﬁne-tunes GPT with the default hyperparameters, and removes the classiﬁcation loss for learning to predict the most likely response given candidates.
We additionally ﬁne-tuned DialoGPT-medium (Zhang et al., 2019), since the model was pretrained on conversation-like exchanges extracted from Reddit comment chains.
Intuitively, this pretraining corpus is more aligned with the downstream ﬁne-tuning data than WebText (Radford et al., 2019), and thus would likely improve downstream performance.
To evaluate the F1 score, we obtained predicted responses from trained models using beam search with a beam size of 5 .
Since past work found that the F1 score can be gamed by always letting the model predict a predetermined phrase (Dinan et al., 2019), we additionally ask humans to rate generations from the model through Amazon mechanical turk to obtain a more complete picture of model quality.
When sampling responses for human evaluation, we used nucleus sampling with p = 0 .
9 (Holtzman et al., 2019).
For human evaluation, we asked 20 turkers to each rate 5 entries.
For each entry, a turker is asked to rate on a scale of 1 to 5 the quality of predicted responses from privately ﬁne-tuned DialoGPT-medium models, non-privately ﬁne-tuned DialoGPT-medium models, non-privately ﬁne-tuned GPT models (our reimplementation of HuggingFace’s entry), and the reference text, given the history of the dialog.
Since human evaluation can yield noisy results, we also report the 95% asymptotic conﬁdence interval in Table 3.
All models were trained and evaluated on the version of Persona-Chat with the original persona.
All numbers reported in Tableare obtained on the validation split.
L T RANSFERRING H YPERPARAMETERS A CROSS D ATASETS
Our work involved tuning hyperparameter with models trained via DP-Adam on one private dataset and transferring such hyperparameters to other private datasets.
Since different datasets may be of different sizes, transferring the batch size may cause a discrepancy in the effective noise multiplier across workloads with different datasets.
Transferring the batch size based on hyperparameter tuning on small datasets to larger datasets can be particularly problematic, as the effective noise multiplier can be larger than ideal.
In this work, we instead transferred the sampling rate q across different datasets.
M D OES DP F INE -T UNING P REVENT U NINTENDED M EMORIZATION ?
One of the ultimate goals of ﬁtting models under DP is to ensure that training data extraction is unlikely given the trained model.
To empirically evaluate whether DP ﬁne-tuning helps prevent against unintended memorization and related attacks, we follow the secret sharer framework (Carlini et al., 2019) and estimate the exposure of artiﬁcial canaries inserted into the training set used for ﬁne-tuning.
We use the E2E dataset as a testbed.
To create canaries, we ﬁrst form a subvocabulary by randomly sampling V = 10 words in the original vocabulary of GPT-2.
Our canaries have preﬁxes of the form
[Equation]
where <word> is randomly sampled from the subvocabulary.
The sufﬁx which our model should learn to predict consists of randomly sampled words with an average length of l = 5 .
By deﬁnition, canaries with an estimated exposure close to log 2 ( V l ) ≈ 17 can likely be extracted.
We experiment with canary-corrupted datasets for repetition values r ∈ { 1 , 10 , 100 } .
A canary has a higher chance in being extracted when it’s repeated for more than once in the training data.
[Table]
78 ±
[Table]
N T EMPLATES AND L ABEL W ORDS FOR T EXT -I NFILLING -B ASED C LASSIFICATION IN S ECTION 3.
Recall that ﬁne-tuning for classiﬁcation can be reformulated as ﬁlling in the [MASK] token in a template sequence.
Here, we list the templates used for each classiﬁcation task considered in the paper.
These templates are almost generic and are not obtained from expensive manual or automated search.
We anticipate better templates obtained from automated search based on data (Gao et al., 2020) to improve the performance even further.
However, we also expect that such a procedure would lead to some amount of increased privacy spending if it were based on private data.
Task
Template
Label words
SST-2 < S 1 >
It was [MASK] .
positive: great, negative: terrible
MNLI < S 1 > ?
[MASK] , < S 2 > entailment:
Yes, netural: Maybe, contradiction:
No
QNLI < S 1 > ?
[MASK] , < S 2 > entailment:
Yes, not entailment:
No
QQP < S 1 > [MASK] , < S 2 > equivalent: Yes, not equivalent: No
O U NCURATED S AMPLES FROM FINE - TUNED MODELS
[Table]
The Mill is a moderately priced restaurant located in the city centre
[Table]
The Mill is a moderately priced restaurant located in the city centre near Caf´e Rouge.
[Table]
[Table]
= 3)
The Mill is an English restaurant located in the city centre near Caf´e Rouge.
It is moderately priced and has a customer rating of 3 out of 5.
GPT-2-m ( (cid:15) =
The Mill is a moderately priced restaurant located in the city centre near Caf´e Rouge.
It is child friendly and has a customer rating of 3 out of 5.
GPT-2-l ( (cid:15) = 3)
The Mill is a moderately priced English restaurant in the city centre near Caf´e Rouge.
It is child friendly and has a customer rating of 3 out of 5.
GPT-2-l ( (cid:15) =
The Mill is a kid friendly English restaurant in the city centre near Caf´e Rouge.
It has a moderate price range and a customer rating of 3 out of 5.
[Table]
customer
[Table]
The Vaults is a Japanese pub in the city centre near Raja Indian Cuisine.
It has a high price range and a customer rating of 3 out of
[Table]
The Vaults is a Japanese pub located in the city
[Table]
near Raja Indian Cuisine.
It has a high price range and a customer rating of 3 out of
[Table]
The Vaults is a Japanese pub located in the city centre near Raja Indian Cuisine.
It has a high price range and a customer rating of 3 out of
[Table]
= 3)
The Vaults is a Japanese pub in the city centre near Raja Indian Cuisine.
It has a high price range and a customer rating of 3 out of 5.
[Table]
The Vaults is a child friendly Japanese
pub in the city centre near Raja Indian Cuisine.
It has a high price range and a customer rating of 3 out of 5.
Reference
Located near the Raja Indian Cuisine at the city centre , is family - friendly pub
The Vaults .
Serving Japanese food , The Vaults , is high priced with a 3 out of 5 customer satisfaction rating .
[Table]
[Table] [Table]
[Table]
The capital of the United States is Washington,
[Table]
C
[Table]
and is part of Benton County, Oregon
[Table]
The capital of the United States is Washington, D
[Table]
C. and is part of Benton County, Oregon
[Table] [Table]
= 3) Albany, Oregon is part of Benton County, Oregon in the United States where Asian Americans are an ethnic group
GPT-2- [Table] ( (cid:15)
= 8) Albany, Oregon is part of Benton County, Oregon in the United States where Asian
Americans are an ethnic group.
GPT-2-l ( (cid:15) = 3) Albany, Oregon is part of the United States where Asian Americans are an ethnic group and the capital is Washington D.C. GPT-2-l ( (cid:15)
=
8) Albany, Oregon is part of the United States where Asian Americans are an ethnic group and the capital is Washington D.C.
[Table]
[Table]
A Loyal Character Dancer is an English language spoken in the United States where Native Americans are the ethnic group.
[Table]
= 8) A Loyal Character Dancer is written in English and is spoken in Great Britain.
Native Americans are an ethnic group in the United States.
[Table]
= 3) A Loyal Character Dancer is written in English and is written in the United States where Native Americans are an ethnic group.
[Table]
A Loyal Character Dancer is written in English and is written in the United States where Native Americans are an ethnic group.
[Table]
A Loyal Character Dancer is written in English and is written in the United States where Native Americans are an ethnic group.
[Table]
Loyal Character Dancer is written in English, which is spoken in Great Britain and the United States.
Native Americans are an ethnic group in the United States
[Table] [Table]
[Table]
P A DDITIONAL R ELATED W ORK
Differentially Private Deep Learning.
DP-SGD has been viewed as ineffective for large models due to the addition of large Gaussian noise to gradient updates.
Improvements to the learning procedure mostly fall under two distinct camps: ( i ) Simplifying the private learning problem, and ( ii ) reducing the scale of noise.
For instance, Papernot et al.
; Tram`er & Boneh (2020); Abadi et al.
(2016) consider transferring features learned on public datasets to simplify the subsequent private learning task.
On the other hand, Zhou et al.
; Kairouz et al.
(2020) remove the ambient dimension dependence of DP noise by identifying subspaces in which private gradients lie and would be privatized.
Yu et al.
(2021b;c) make such ideas practical and demonstrate improved results on private learning benchmarks.
Zhang et al.
(2021) applied the sparse vector technique to learning wide neural layers to reduce the amount of injected noise.
Our work mostly falls under the ﬁrst camp – improving private learning through simplifying the learning task.
Our work is also distinct from prior works in that we focus on privately ﬁne-tuning large pretrained models.
Lastly, there are alternative solutions in the literature that enforces DP which are not based on gradient perturbation (Papernot et al., 2018; 2016).
These methods typically require extra public data and are not the present focus.
Parameter-Efﬁcient Fine-Tuning.
Recent developments on pretrained model adaptation have produced a wide range of parameter-efﬁcient ﬁne-tuning methods for both vision and language tasks.
We brieﬂy summarize these, grouping by category.
Approaches based on optimizing prompt-like constructions for NLP tasks include preﬁx-tuning (Li & Liang, 2021), P-tuning (Liu et al., 2021), and prompt-tuning (Lester et al., 2021).
Adapter-based methods insert small subnetworks inside pretrained Transformers (Houlsby et al., 2019; R ¨ uckl ´ e et al., 2020; Pfeiffer et al., 2020).
Methods that optimize low-rank matrices include the work by Hu et al.
; Mahabadi et al.
In addition, there are adaptation methods that only optimize biases for vision (Cai et al., 2020) and language tasks (Ben Zaken et al., 2021).
Our evaluation in Section 5.2 covered the most representative methods that generally have state-of-the-art non-private learning performance (at the time of writing) for the range of NLP tasks studied in this paper.
Speeding Up DP-SGD.
Apart from the work by Lee & Kifer (2020) and Subramani et al.
(2020), there is an approach that approximates per-example gradient norms through the combination of random projection and forward-mode autodiff (Bu et al., 2021).
While faster than vanilla private learning, this approach has the drawback of increased privacy spending and having an extra hyperparameter.
Our ghost clipping technique, while only suited for Transformers applied to sequential data, does not introduce new hyperparameters.
Alternative Clipping Strategies.
While there are alternative clipping strategies in the literature that show improvements on simple tasks (Pichapati et al., 2019; Asi et al., 2021), we have opted to study the simplest strategy that clips gradients by their Euclidean norm.
We leave the study of these algorithms for NLP tasks to future work.
Concurrent Work.
We are made aware of a concurrent work that also studies ﬁne-tuning large language models under DP (Yu et al., 2021a).
This work presents initial successes on ﬁne-tuning under DP with low-rank methods such as LoRA.
Our experiments on language generation (see Section 5. and Table 2) demonstrate similar ﬁndings.
Yet, we moreover show that full ﬁne-tuning with good hyperparameters attains similar performance and possesses similar model scaling properties, which was raise by Yu et al. (2021a) as interesting open questions to pursue.
Lastly, our private ﬁne-tuning results for sentence classiﬁcation may be far from optimal, since we used hyperparameters mostly transferred from tuning on the E2E language generation task.
DP Synthetic Data Generation.
Fine-tuning generative language models on private data under
DP can also be viewed as a means of accomplishing DP synthetic data generation – learning generative models from private data so that synthetic examples could be sampled and used for analysis.
Previous work employed generative adversarial networks and focused primarily on image or tabular datasets (Torkzadehmahani et al., 2019;
Neunhoeffer et al., 2020; Chen et al., 2020; Torﬁ et al., 2020).
Perhaps more related is the work by Bommasani et al.
(2019) which attempted ﬁne-tuning
GPT-2 on medical datasets to generate synthetic records but did not report any quantitative results.
Q E STIMATES OF R UN - TIME IN PRACTICE
The actual run-time of algorithms depends on implementation details.
Here, we outline estimates of the run-time for full ﬁne-tuning with DP-Adam on tasks considered in the paper.
These numbers are based on running with a single RTX 3090 with PyTorch==1.9.0 .
Fine-tuning GPT-2 on E2E and DART takes less than 10 minutes per epoch, and ﬁne-tuning for 10 epochs results in reasonably performing models.
The time to ﬁne-tune RoBERTa-base on classiﬁcation tasks depends on the size of the dataset.
It takes less than 10 minutes per epoch on the smallest SST-2, whereas for the largest
MNLI, it takes less than an hour per epoch.
R E FFECTS OF VARYING (cid:15) AND δ ON DP F ULL F INE - TUNING P ERFORMANCE
Figure 9 shows how different values of (cid:15) and δ affect the performance of full ﬁne-tuning under DP.
[Figure]
E 2 E t e s t s e t
B L E U {+=3=         
+}
[Figure]
S DP-A DAM VS DP-SGD
Our work focused on DP-Adam as opposed to DP-SGD, since Adam is more commonly used for non-private language model ﬁne-tuning.
While the two algorithms differ in their parameter update rule, the basic gradient privatization procedure is the same.
We performed additional experiments ﬁne-tuning GPT-2 on the E2E dataset with DP-SGD (with freshly tuned learning rate and clipping norm values) and observed that its performance (test set BLEU 63.175 at ( (cid:15), δ ) =
(8 , 10 − 5 ) ) is on par with DP-Adam (test set BLEU 63.189 at ( (cid:15), δ ) =
(8 , 10 − 5 ) ).
T S UBTLETIES OF IMPLEMENTING DP MIXED PRECISION TRAINING
Mixed precision training (Micikevicius et al., 2017) accelerates updates by storing certain tensors in half-precision.
To mitigate negative effects caused by potential arithmetic underﬂow, usual implementations upscale the loss with an adaptive factor pre-backpropagation and downscale the gradients with the same factor post-backpropagation.
The scaling factor is adapted based on whether underﬂow is observed during training.
Special care needs to be taken when combining gradient privatization with mixed precision training.
One implementation that ensures similar results across full and mixed precision training (1) upscales the loss with the adaptive factor K , (2) clips per-example gradients by CK , (3) adds to the sum of clipped gradients the usual Gaussian noise multiplied by K , and (4) downscales the noisy gradient by
K .
This is the implementation that we adopt, and we were able to obtain similar results on the E2E dataset with and without mixed precision.
One alternative implementation (a) upscales the loss with the adaptive factor K , (b) clips per-example gradients by C , (c) adds the usual Gaussian noise to the sum of clipped gradients, and (d) downscales noisy gradients by K .
The main difference between this procedure and the prior is whether the factor
K is considered during clipping and noising.
This procedure, while having the same DP guarantee, typically does not result in similar results as full precision when the same hyperparameters are used across the two settings (even with an optimizer like Adam which self-adjusts the magnitude of updates with accumulated empirical second moments).
We also identiﬁed that this implementation is also the primary reason that a prior work’s code does not reproduce good results when run in full precision.
U F INE - TUNING WITH RGP AND THE TEXT - INFILLING OBJECTIVE
Recall Section 3.2 and Table 1 showed that private full ﬁne-tuning with a text-inﬁlling objective leads to improved classiﬁcation results.
Here, we show that the inﬁlling objective is also helpful when one privately ﬁne-tunes with the RGP method for a classiﬁcation task.
To study this, we reimplemented the RGP method and tuned the hyperparameters in full precision.
Fixing all hyperparameters, we compared models trained with and without inﬁlling.
Table 14 conﬁrms that ﬁne-tuning with RGP based on inﬁlling is generally helpful across the considered tasks.
Note the aim of this experiment is not obtain state-of-the-art performance, but rather to study the effect of using the text-inﬁlling objective.
Thus, we expect these results could generally be further improved with more extensive hyperparameter search.
[Table]
V W HICH METHOD SHALL I USE FOR PRIVATE FINE - TUNING ?
While we were able to obtain similar results on the E2E and DART datasets with full ﬁne-tuning and
LoRA (Tables 1 and 8), we encountered signiﬁcant difﬁculties in ﬁne-tuning for dialog generation (even non-privately) when not updating the embedding layer and language modeling head – perplexity was much worse, and generations frequently contained seemingly arbitrary characters.
This result suggests that while full ﬁne-tuning may be more computationally intensive than lightweight approaches at times, its simplicity (involving few design decisions) makes it an attractive ﬁrst option when compute resource is sufﬁcient.