Automatic Clipping: Differentially Private Deep Learning Made Easier and Stronger
Author(s)
Abstract
Per-example gradient clipping is a key algorithmic step that enables practicaldifferential private (DP) training for deep learning models.
The choice of clippingthreshold R , however, is shown to be vital for achieving high accuracy under DP.
We propose an easy-to-use replacement, called AutoClipping, that eliminatesthe need to tune R for any DP optimizers, including DP-SGD, DP-Adam, DPLAMB and many others.
The automatic variants are as private and computationallyefficient as existing DP optimizers, but require no DP-specific hyperparametersand thus make DP training as amenable as the standard non-private training.
Wegive a rigorous convergence analysis of automatic DP-SGD in the non-convexsetting, which shows that it can enjoy an asymptotic convergence rate that matchesthe standard SGD, under a symmetric noise assumption of the per-sample gradients.
We also demonstrate on various language and vision tasks that automaticclipping outperforms or matches the state-of-the-art, and can be easily employedwith minimal changes to existing codebases.
Introduction
Deep learning has achieved impressive progress in a wide range of computer vision and naturallanguage processing tasks.
These successes are made available, in part, by the collection of largedatasets, sometimes containing sensitive private information of individual data points (e g., chest scanimages, DNA sequences).
Prior works have illustrated that deep learning models pose severe privacyrisks to individual subjects in the training data and are susceptible to various practical attacks.
Forexample, machine learning services such as Google Prediction API and Amazon Machine Learningcan leak membership information from the purchase records [58]; if one feeds the GPT2 languagemodel with some specific prefix, the model will autocomplete texts that contain the full name, phonenumber, email address etc., from the training data that it memorizes [11].
Differential privacy (DP) [21, 23, 22] is a formal definition of privacy that has been shown to preventthe aforementioned privacy risks in deep learning [1].
On a high level, the key difference between the
DP deep learning and the regular one is whether the gradient is privately released.
In other words,while the standard optimizers update on the summed gradient (cid:80) i g i , and DP optimizers update onthe private gradient :
[Equation]
Here g i ∈
R
d is the per-sample gradient of loss l i , N is the standard normal random variable, σ is thenoise multiplier, and R is the clipping threshold.
The clipping function Clip : R d → R is definedsuch that ∥ g i · Clip ( g i ; R ) ∥ ≤ R .
For instance, the DP-SGD in [1] on batch B t is DP
[Equation]
SGD Abadi : [Equation] In comparison to the regular training (1.2), two additional DP-specific hyperparameters R and σ need to be determined in DP learning (1.1).
On the one hand, setting the noise multiplier σ is easy and can be derived analytically prior to the training.
Whenever the privacy budget ( ϵ, δ ) is determined, one can apply off-the-shelf privacy accounting tools in Section 2.1 to determine σ , based on the subsampling probability p
(e.g. expected batch size over sample size) and the number of iterations T : privacy_accountant ( σ, p, T ; δ ) =
ϵ
On the other hand, the choice of clipping threshold R is crucial to the performance of DP models,yet the hyperparameter tuning is much labor-intensive.
Recent advances of DP deep learning on ImageNet [34] and on E2E datasets [37], using ResNet18 and GPT2 respectively, illustrate that theperformance is very sensitive to R .
We have reproduced their results in Figure 1. Observe that on
ImageNet, ResNet18 can drop from the highest 45% accuracy to 31% if R is chosen 2 times larger,and to 0.1% if R is chosen 4 times larger.
Similar drastic drop can also be observed in [34, Figure 3]even if the noise multiplier σ = 0 .
Unlike the noise multiplier σ , the clipping threshold R cannot beinferred from the privacy budget ( ϵ, δ ) and have to be tuned.
Consequently, DP training necessarilyrequires a 2D grid search for ( R, η ) , like the lower plot of Figure 1, whereas the regular trainingonly requires an easy 1D grid search for η .
Even worse, the difficulty of tuning a per-layer clippingthreshold vector [43], i.e. one clipping threshold for one layer, may increase exponentially as thenumber of layers increases.
To save the effort of tuning R , previous researches have proposed different approaches.
In [3, 51, 25],researchers advocate to use data-adaptive information to select R , such as a specified quantile ofthe gradient norm distribution.
These adaptive clipping methods can be a little ad-hoc: they oftenreplace the the need to tune R by the need to tune one or more new hyperparameters, e.g. thequantile to use and the ratio to split the privacy budget between the quantile decision and the gradientperturbation.
Another approach used by the practitioners is to replace an expensive 2D grid searchby multiple cheaper 1D grid searches.
For example, the researchers propose, in [34, Section 3.3] tofine-tune η with non-DP SGD, fix η and sweep over various values of the clipping threshold R with DP-SGD, then further fix R and do one more grid search on η .
However, tuning R formally in adata-dependent way (e.g. through cross-validation) introduces additional privacy loss [48], and mostexisting empirical work does not privately conduct hyperparameter tuning.
We take a completely different route by proposing a new clipping principle that removes R , insteadof coming up with methods to find the appropriate R .
We term our method as automatic clipping(AutoClipping) and we term the versions of DP optimizers using it as automatic DP optimizers . We summarize our contributions as follows.
We propose the automatic clipping in (4.1) that expunge the clipping threshold from general DPoptimizers, allowing DP learning to be as amenable as regular learning.2.
We show that automatic DP optimizers are as private and efficient as existing DP optimizers.3.
We show that automatic DP-SGD converges in the non-convex setting, at the same asymptoticconvergence rate as the standard SGD.
Our theoretical analysis successfully explains the trainingbehaviors in previous empirical works.4.
We demonstrate the superiority of automatic clipping on a variety of vision and language tasks,especially with large models including ResNet, RoBERTa and GPT2.5.
In Appendix K, we include simple code snippets that demonstrate how easy it is to switch fromAbadi’s clipping to our automatic clipping in popular codebases, that implement DP optimizersfor deep learning, e.g. Opacus and ObJAX.
Preliminaries2.1 Differential Privacy
We consider the ( ϵ, δ ) -DP in Definition 2.1, where smaller ( ϵ, δ ) means stronger privacy guarantee.
Definition 2.1 ([22]) .
A randomized algorithm M is ( ε, δ ) -differentially private (DP) if for any twoneighboring 1 datasets S, S ′ , and for any event E , [Equation]
[Equation]
In words, DP restricts the influence of an arbitrary sample, so that the information contributed by suchsample is limited and less vulnerable to privacy attacks.
In deep learning, DP is generally achieved byapplying the subsampled Gaussian mechanism to privatize the minibatch gradients during training.
As illustrated in Equation (1.1), the subsampled Gaussian mechanism involves (1)
Sampling aminibatch by including each data point iid with probability p (2) per-sample gradient clipping tobound the l 2 norm sensitivity at R and (3) adding independent Gaussian noise proportional to thesensitivity R and σ , which is derived from the privacy loss ϵ .
This can be realized by leveraging avariety of modern privacy accounting tools, such as those based on Renyi DP (or moments accountant)[1, 45, 63], Privacy Loss distribution (Fourier accountants) [33, 27, 70], or Gaussian DP [16, 7].
2.2 Differentially Private optimizers with general clipping operations
Privately released stochastic gradients (through the Gaussian mechanism) can be used to instantiatevarious off-the-shelf optimizers, which gives rise to DP-SGD in (1.3), DP-HeavyBall, DP-AdaGrad, DP-Adam, DP-FedAvg, DP-FedSGD [43], etc.
To improve the performance of DP optimizers,previous researches can be classified into two categories.
The first category, where the majority of researches lie in, works with Abadi’s clipping and focuseson better design of R .
To name a few examples, one can adaptively design R t for each iteration t[3, 51, 25], or design the per-layer clipping threshold vector R ∈ R L for L layers [1, 43] so as toapply a different clipping threshold for each layer.
Much fewer works fall into the second category that proposes new clipping method.
In fact, anyfunction Clip : R d → R satisfying ∥
Clip ( g ) · g ∥ ≤ R can serve as a valid clipping function besides
Abadi’s.
For instance, the global clipping [9] proposes Clip global ( g ) :
=
I ( ∥ g ∥ < Z ) · R/Z to correctthe bias in the private gradient and alleviate the mis-calibration issue of DP classifiers.
Our automaticclipping also belongs to this category.
We note that different clipping methods work orthogonally tooptimizers, network architectures and gradient norm computation (see Section 7).
Motivation
[Figure]
3.1 Small clipping threshold works best
[Figure]
DP-SGD with momentum. One intriguing observation that we can make about the recent studies on DP learning with large models is that thestate-of-the-art (SOTA) results are often achieved with verysmall clipping threshold R . This observation is consistentin both vision and language tasks.
In [37], GPT2 (over 800million parameters) and RoBERTa models (over 400 millions parameters) achieve the best results under DP on QNLI,
MNLI, SST-2, QQP, E2E, and DART datasets, with each persample gradient clipped to length R = 0 .
In [34, 14, 44], ResNets and Vision Transformers achieve the best DP results on ImageNet with R = 1 ; in [60], the best DP resultson CIFAR10 use R = 0 . 1 with ResNeXt-29 and SimCLRv2[12].
The effectiveness of small clipping threshold togetherwith proper learning rate is depicted in Figure 1.
Intuitively, smaller R implies that the Abadi’s clipping (3.1)happens, which means min (cid:0) R/ ∥
g i ∥ , 1 (cid:1) =
R/ ∥ g i ∥ .
Giventhat the clipping threshold R is so small compared to thenumber of parameters in large neural networks, and thatstrong DP is guaranteed when the number of training iterations is small (i.e. ∥ g i ∥ has not converged to small valuesyet), we expect and empirically observe that the clippinghappens on a large proportion of per-sample gradients atall iterations.
For instance, we find in the GPT2 generation experiments in [37] that 100% of per-sample gradients are clipped at all iterations; in classification tasks suchas QQP/QNLI/MNLI, the percentage of clipping is about20 ∼ 60% on average (more details in Appendix H.1).
3.2 Per-sample gradient normalization as new clipping: a similarity viewpoint
In the small clipping threshold regime, we can approximately view
[Equation]
and thus derive a novel private gradient (cid:80)
i R g i ∥
[Equation]
∥ + σR · N (0 , I ) .
Here AUTO-V stands for thevanilla automatic clipping, which essentially performs the gradient normalization on each per-samplegradient.
As a specific example, we can write the R -dependent automatic DP-SGD as
[Equation]
We may view our AUTO-V clipping as to maximize the dot-product similarity between the clippedgradient and the regular gradient, a commonly used similarity measure, e.g. in the attention block intransformers [61].
Suppose we want
[Equation]
Note that the constraint is a sufficient condition for clipping, as discussed in Section 2.2.
It is nothard to see that the optimal clipping factor is
[Equation]
If the per-sample gradients are indeed concentrated in the sense ∀ i, ⟨ g i , (cid:80) j g j ⟩ ≥ 0 , then AUTO-Vis the optimal per-sample gradient clipping.
We compare with Abadi’s clipping in Figure 2, wherethe dot-product similarity is significantly magnified by our AUTO-V clipping.
[Figure]
141 3.3 Stability constant breaks scale-invariance and remains stationary
[Figure]
One potential drawback of AUTO-V clipping is that allgradients lose their magnitudes information completely,since ∥ g i · Clip AUTO-V ( g i ; R ) ∥
=
R, ∀ i .
This scaleinvariance in AUTO-V and partially in Abadi’s clipping(when ∥
g i ∥ > R ) leads to the "lazy region" issue: theparameters will not be updated by DP-GD even if thetrue gradients are non-zero.
In Figure 3, we illustratein a logistic regression 2 that AUTO-V and Abadi’s clipping have zero clipped gradient for the trainable parameterθ ∈ [ − 2 , 2] , as the per-sample gradients from two classes cancel each other.
Another benefit of γ is to remain stationary as g i → 0 , i.e. making the clipped gradient C
i g i
→ g i /γsmall rather than having a magnitude R in AUTO-V.
We elaborate this point in Section 4.3.
To preserve the magnitude information and thus escape the lazy region, we propose the AUTO-Sclipping, with a positive stability constant γ : Clip
[Equation]
We visualize in Figure 4 that AUTO-S allows larger per-sample gradients to have larger magnitudesafter the clipping, while still allowing smaller gradients to vanish after “clipping”. This is critical inour convergence analysis and allows DP-SGD AUTO-S (but not DP-SGD AUTO-V ) to converge to zerogradient norms in Section 5.
Automatic DP Training
One may wonder why our clipping (3.1)(3.3) is automatic at all, if the hyperparameter R is stillpresent and there is an additional parameter γ to choose.
It turns out that any constant choice of R > 0is equivalent to choosing R = 1 , and common deep learning optimizers are insensitive to the choiceof γ (e.g. for any γ > 0 , we show that the gradient norm converges to zero at the same asymptoticrate in Theorem 4; see also the ablation study in Figure 14).
Consequently, we set γ = 0 .
01 as thedefault.
Specifically, let us redefine the R -independent clipping function:
[Equation]
With this clipping, we can design automatic DP optimizers similar to (1.1):
[Equation]
where ˆ g t := (cid:88) i ∈ B
[Equation]
Clearly, the new private gradient ˆ g t from our automatic clipping is R -independent, in contrast to theone used in (1.1).
A concrete example (in the case of γ = 0 ) that is comparable to (3.2) will be
R -independent DP
[Equation]
SGD AUTO [Equation] [Equation] Leveraging the private gradient ˆ g t in (4.2), we can train DP neural networks without tuning DPspecific hyperparamters R and σ , as demonstrated in Algorithm 1.
with DP weights w 0 , learning rate η t , sampling probability p , of iterations T .
We will elaborate two distinct reasons in each section for the following statement:
[Equation] which reduces the hyperparameter tuning of DP training to that of the regular training, i.e. only onlearning rate, weight decay, etc.
The significant save in the tuning effort is illustrated in Figure 15.
4.1 Non-adaptive optimizer couples clipping threshold with learning rate
With R -dependent automatic clipping, DP-SGD becomes
[Equation]
We can view η effective ≡ ηR as a whole: increasing R has the same effect as increasing η , whichexplains the diagonal pattern in Figure 1(lower plot) where DP-SGD Abadi is applied with smallclipping threshold 3 .
We extend to general non-adaptive optimizers in Theorem 1, with proof in Appendix B.1 4 .
Theorem 1.
Non-adaptive R -dependent automatic DP optimizers (including SGD, Heavyball[52]and NAG[47]), with learning rate η and weight decay λ , is equivalent to R -independent automatic
DP optimizers, with learning rate η ′ = ηR and weight decay λ ′ =
λ/R .
4.2 Adaptive optimizer can be insensitive to clipping threshold
Adaptive automatic DP optimizers are different than the non-adaptive ones, as the clipping threshold cancels out instead of being coupled with learning rate.
To see this, we scrutinize DP-Adam Abadi (which is similar to DP-Adam AUTO-V ) in Figure 1(upper plot), where columns to the left are almost identical.
Further evidence is observed in [44, Table 5] that shrinking R has zero effect on LAMB.
We now give a simple explanation using AdaGrad [19]:
[Equation]
where g t =
(cid:80) i g t,i is the gradient sum and G t =
(cid:80) τ<t g 2 τ is sum of gradient square by Hadamard product over the past iterations.
In R -dependent DP-AdaGrad AUTO-V , the private gradient is R ˆ g t in place of the standard gradient sum g t , and ˆ G t =
R 2 (cid:80) τ ≤ t ˆ g 2 τ :
[Equation]
We generalize to the general adaptive optimizers in Theorem 2, with proof in Appendix B.2.
Theorem 2.
Adaptive R -dependent automatic DP optimizers (including AdaGrad[19], AdaDelta[69],
AdaMax/Adam[31], NAdam[17], RAdam[39], LARS[65], LAMB[66]), with learning rate η andweight decay λ is equivalent to R -independent automatic DP optimizers with learning rate η andweight decay λ ′ =
λ/R .
With decoupled weight decay[42], R -dependent automatic DP-AdamW isequivalent to R -independent automatic DP-AdamW with the same η and λ .
Similarly, we demonstrate the automatic DP optimizers with per-layer clipping style in Appendix B.3.4.3 Automatic clipping guarantees the same level of privacy while
[Figure]
utility
[Figure]
In Theorem 3 (proved in Appendix A), we show thatthe new private gradient ˆ g t in (4.2) has the same levelof privacy guarantee as the existing one in (1.1), sincethe global sensitivity remains the same (see Figure 4).
We note that as long as γ > 0 , the magnitude information of per-sample gradients is preserved by AUTOS, in the sense that ∥ g i ∥ >
∥ g j ∥ ⇐⇒ ∥
C i g i ∥ >
∥ C j g j ∥ , whereas this can be violated in both the AUTO-V and Abadi’s clipping (as depicted by theflat curve in Figure 4 when ∥ g i ∥ > 1 ).
Additionally, note that when γ is small, almost all datapoints “max out” the signal relative to the amount of noise we add.
To say it differently, for the sameamount of noise, AUTO-S with small γ allows more signal to be pushed through a differentiallyprivate channel.
Towards the end of the training, i.e., at the limit when ∥ g i ∥ → 0 for all i , then wehave (cid:80) i g i ∥
g i ∥ + γ → 1 γ (cid:80)
i g i .
In words, the clipped gradients become closer to the standard SGD,thus do not suffer from the instability of AUTO-V.
Theorem 3.
Under the noise multiplier σ , number of iterations T , subsampling probability B/n ,
DP optimizers using AUTO-V or AUTO-S clipping satisfy ( ϵ Accountant ( δ, σ, B/n, T ) , δ ) -DP, whereϵ Accountant is any valid privacy accountant for DP-SGD under Abadi’s clipping.
Convergence analysis of DP-SGD with automatic clipping5.1 Convergence theory of DP-SGD to stationary points
We highlight that automatic clipping can be more amenable to analysis than Abadi’s clipping in [13],since we no longer need to decide whether each per-sample gradient is clipped.
To analyze the convergence of automatic DP-SGD (4.2) in the non-convex setting, we follow thestandard assumptions in the SGD literature [24, 2, 6], with one additional symmetry assumption onthe gradient noise.
Assumption 5.1 (Lower bound of loss) .
For all w and some constant L ∗ , we have L ( w ) ≥ L ∗ .
Assumption 5.2 (Smoothness) .
Let g ( w ) denote the gradient of the objective L ( w ) .
Then ∀ w , v ,there is an non-negative constant L such that
[Equation]
Assumption 5.3 (Gradient noise) .
The per-sample gradient noise ˜ g t,i − g t is i.i.d.
from some ditribution such that
[Equation]
and ˜ g t,i is centrally symmetric 5 about g t in distribution:
[Equation]
We show in Theorem 4 that DP-SGD with AUTO-S clipping allows the true gradient norm to convergeto zero, but not so with AUTO-V clipping.
We leave the proof in Appendix C.1.
Theorem 4.
Under Assumption 5.1, 5.2, 5.3, running DP-SGD with automatic clipping for Titerations and setting the learning rate η ∝ 1 / √ T give [Figure]
[Equation]
Here · · · represents the first argument of G , and G is increasing and positive.
As T → ∞ , we havemin t E ( ∥ g t ∥ )
=
O ( T − 1 / 4 ) for AUTO-S, the same rate as the standard SGD given in Theorem 9. [Figure] [Figure] o un d o f m i n t ( || g t || )
r + , r=0.2 r + , r=0.5 r + , r=0.
r + , r=1.1 r + , r=2.0 r + , r=3.
[Figure]
f m i
[Figure]
t
[Figure]
||
[Figure]
Remark 5.4.
In Theorem 4, the upper bound takes an implicit form of G ( · ; ξ, γ ) because it is alower envelope of functions ξr + F ( · ; r, ξ, γ ) over all possible r > 0 , whose forms are detailed in
Theorem 6.
Notice that G results only from the clipping operation, not from the noise addition.
Remark 5.5.
We show in Theorem 6 and demonstrate in Figure 5 that the upper bound (5.2) is alwayslarger than ξ with AUTO-V ( γ = 0 ), and can only be reduced to zero with AUTO-S ( γ > 0 ).
Weprovide real data evidence in Figure 13 that strictly positive γ reduces the gradient norm significantly.
5.2 Analysis of factors affecting the convergence
We now analyze the many factors that affect the convergence in Theorem 4, from a unified viewpointof both the convergence and the privacy.
We start with the stability constant γ and the learning rate η t , both only affect the convergence notthe privacy.
We empirically observe in Figure 7 that small γ benefits the convergence at initialiterations (when the privacy guarantee is strong) but larger γ converges faster asymptotically.
For η t ,the optimal is in fact the miminizer of the hyperbola in (C.4), that is unique and tunable.
Next, we focus on the hyperparameters that affect both convergence and privacy: the batch size B ,the noise multiplier σ , and the number of iterations T .
These hyperparameters have to be consideredalong the privacy-accuracy tradeoff, not just from a convergence perspective.
Recall that given a fixed privacy budget ( ϵ, δ ) , we rely on modern privacy accountant for computingthe appropriate combinations of parameter σ, T, B .
The exact expression of the bound as a function of( ϵ, δ ) is somewhat messy. For this reason, we illustrate our analysis in terms of the surrogate parameterµ for µ -GDP [16]. [7] showed that DP-SGD’s privacy guarantee asymptotically converges to µ -GDP(as T → ∞ ) with µ =
Bn (cid:112) T ( e 1 /σ 2 − 1) . µ -GDP implies ( ϵ, δ ) -DP with ϵ = µ 2 + µ (cid:112) 2 log(1 /δ )) 6 . We can alternatively leverage ρ -tCDP [10] for similar conclusions, using ρ in place of µ 2 in (5.3). Theorem 5.
Under Assumption 5.1, 5.2, 5.3, fixing the asymptotic µ ( ϵ, δ ) -GDP parameter, running DP-SGD with automatic clipping for T iterations and setting the learning rate η ∝ 1 / √ T give
[Equation] To show that our analysis matches the training behaviors observed in SOTA empirical work [37, 34,14, 60, 44, 68], we minimize the first argument of G in (5.3), denoted as X ( B, T, µ, d, L, L 0 ) .
[Train longer with larger noise] Fixing the expected batch size B , we see that X is decreasingin T .
Hence larger T and consequently larger σ are preferred.
[Larger batch size helps] Fixing number of iterations T or epochs E
= BT/n , we see that X isdecreasing in B .
Hence larger B and consequently larger σ are preferred.3.
[Pretraining is critical]
Pretraining can boost the accuracy of DP learning through a much smallerinitial loss L 0 and from a smooth (small L ) and flat (small ξ , c.f. Figure 7(left)) initialization.4.
[Learning rate needs tuning]
The optimal learning rate by minimizing (C.4) is (cid:113) ( L 0 −L ∗ ) µ 2 n 2 L ( µ 2 n 2 + dT )
.This indicates that one should use larger learning rate for smaller model d , weaker privacy (largerµ or small ϵ ), or smaller iteration budget T .
Interestingly, the optimal choice of learning rate isindependent to (expected) batch-size B .
Experiments
We evaluate our automatic DP training on image classification, sentence classification, and table-totext generation tasks.
Detailed settings including hyperparameters can be found in Appendix G.
6.1 Image classification
For MNIST/FashionMNIST, we use the same setup as in [49, 60, 57] with a simple CNN.
For
CIFAR10, we use the same setup as in [60] with pretrained SimCLRv2 [12].
For ImageNette, a10-class sub-task of ImageNet [15], we use the same setup as in [32] without the learning rate decay.
For CelebA [41], the real human face dataset, we train ResNet9 [28] with group normalization toreplace the batch normalization.
Notice that CelebA contains high-resolution (178x218) images, eachwith 40 labels.
We consider CelebA for either multi-class classification on one label, e.g. ‘Smiling’and ‘Male’, or for multi-label/multi-task problem to learn all labels simultaneously.
[Table]
In Table 1, we observe that AUTO-S clipping outperforms existing clipping in all datasets with statistical significance.
Interestingly, the standard deviation from different runs is smaller for automatic
DP optimizers, indicating better reproducibility and stability.
We additionally experiment 40 binaryclassification problems on CelebA with respect to each label, and observe that the mean accuracyfurther improves to 91.63% at ϵ = 8 for AUTO-S (see Appendix J).
6.2 Sentence classification
On five benchmark language datasets (MNLI(m/mm)[64], QQP[30], QNLI[55], SST2[59]), wecompare our automatic DP training with reparameterized gradient perturbation (RGP, [68]) andfull-parameter finetuning (full, [37]) using RoBERTa models [40].
These methods use the sameexperimental setup.
For language models, our automatic training is based on the codebase of [37] 7 .
[Table] [Table]
In Table 2 and Table 3, we note that full parameter finetuning with AUTO-S outperforms or at leastmatches SOTA on all tasks.
We use exactly the same hyperparameters as in [37].
6.3 Table-to-text generation
We compare our automatic DP training with a variety of fine-tuning methods, for table-to-textgeneration task on E2E dataset [20], where the goal is to generate texts about different aspects of arestaurant’s data.
We measure the success on this task by BLEU, ROUGE-L (in Table 4), METEOR,
NIST, CIDEr (extended in Table 7)
[Table]
with higher value meaning better model quality.
[Table]
dataset with GPT2.
Additional performance measures are included in Table 7.
The best two GPT2 models for each row are marked in bold.
Competitive methods include low-rank adaption (LoRA), prefix-tuning (prefix), RGP, only fine-tuningthe top 2 Transformer blocks (top2), and training from scratch (retrain), as were recorded in [37].
Again, we use the exactly the same hyperparameters as in [37].
For GPT2 (163 million parameters),
GPT2 medium (406 million), and GPT2 large (838 million), Table 4 shows that AUTO-S is scalablewith stronger performance on larger models.
Our automatic full-parameter finetuning has the bestoverall performance.
Additionally, we highlight that AUTO-S and methods like LoRA are notmutually exclusive and can be combined to yield strong performance, since AUTO-S modifies theoptimizers and LoRA modifies the architecture.
Discussion
In this work, we proposed AutoClipping as a drop-in replacement to the standard per-exampleclipping differentially private training.
This is the first technique that eliminate the need to tune theclipping threshold R , thus making DP deep learning as easy as regular learning.
Our AUTO-S methodenjoys both theoretical guarantee of convergence in non-convex problems (under various conditions),and strong empirical performance that advances the state-of-the-art (SOTA) of DP learning on bothcomputer vision and language tasks.
We are excited about the future of automatic DP training, especially along with other workingtechniques.
Notably, our automatic clipping applies compatibly with general optimizers (e.g. [8, 18]),clipping styles (all-layer or per-layer), architecture modifications (e.g. LoRA, RGP, prefix), and dataaugmentation (e.g. adversarial training [26] and multiple augmentation [14]).
Thus, we expect toachieve comparable results to all SOTA in a lightweight fashion.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[N/A]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results..
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Assumptions5.1-5.3.
[Table]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A.3.
If you ran experiments...
[Table]
(a) Did you include the code, data, and instructions needed to reproduce the main experimentalresults (either in the supplemental material or as a URL)?
[Yes] We include our code inAppendix K, and cite all datasets.
[Table]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they werechosen)?
[Yes] See all details in Appendix G.
[Table]
(c) Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?
[Yes] See Table 1.
[Table]
(d) Did you include the total amount of compute and the type of resources used (e.g., type ofGPUs, internal cluster, or cloud provider)?
[N/A]
This work is unrelated to computationalefficiency.
[Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[Yes] We cite all codebases,datasets, models clearly.
[Table]
(b) Did you mention the license of the assets?
[N/A] Experiments are conducted on public data. [Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We
[Table]
include code snippet in Appendix K and will include a URL to release the full codebase. [Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re [Table] using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board
(IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent onparticipant compensation?
[N/A]