Automatic Clipping: Differentially Private Deep Learning Made Easy and Stronger
Anonymous Author(s)
AffiliationAddress email
Abstract
Per-example gradient clipping is a key algorithmic step that enables practicaldifferential private (DP) training for deep learning models.
The choice of clippingnorm R , however, is shown to be vital for achieving high accuracy under DP.
Wepropose an easy-to-use replacement, called AutoClipping, that eliminates the needto tune R for any DP optimizers, including DP-SGD, DP-Adam, DP-LAMB andmany others.
The automatic variants are as private and computationally efficientas existing DP optimizers, but require no DP-specific hyperparameters and thusmake DP training as amenable as the standard non-private training.
We give arigorous convergence analysis of automatic DP-SGD in the non-convex setting,which shows that it enjoys an asymptotic convergence rate that matches the standard SGD.
We also demonstrate on various language and vision tasks that automaticclipping outperforms or matches the state-of-the-art, and can be easily employedwith minimal changes to existing codebases.
Introduction
Deep learning has achieved impressive progress in a wide range of computer vision and naturallanguage processing tasks.
These successes are made available, in part, by the collection of largedatasets, sometimes containing sensitive private information of individual data points (e g., chest scanimages, DNA sequences).
Prior works have illustrated that deep learning models pose severe privacyrisks to individual subjects in the training data and are susceptible to various practical attacks.
Forexample, machine learning services such as Google Prediction API and Amazon Machine Learningcan leak membership information from the purchase records [58]; if one feeds the GPT2 languagemodel with some specific prefix, the model will autocomplete texts that contain the full name, phonenumber, email address, etc., from the training data that it memorizes [10].
Differential privacy (DP) [20, 22, 21] is a formal definition of privacy that has been shown to preventthe aforementioned privacy risks in deep learning [1].
On a high level, the key difference between the
DP deep learning and the regular one is whether the gradient is privately released.
In other words,while the standard optimizers update on the summed gradient (cid:80) i g i , and DP optimizers update onthe private gradient :
[Equation]
Here g i ∈
R
d is the per-sample gradient of loss l i , N is the standard normal random variable, σ isthe noise multiplier, and R is the clipping norm.
The clipping function Clip : R d → R is defined such that ∥ g i · Clip ( g i ; R ) ∥ ≤ R .
For instance, the DP-SGD in [1] on batch B t is
[Equation]
In comparison to the regular training (1.2), two additional DP-specific hyperparameters R and σ need to be determined in DP learning (1.1).
On the one hand, setting the noise multiplier σ is easy and can be derived analytically prior to the training.
Whenever the privacy budget ( ϵ, δ ) is determined, one can apply off-the-shelf privacy accounting tools in Section 2.1 to determine σ , based on the subsampling probability p
(e.g. batch size over sample size) and the number of iterations T : privacy_accountant ( σ, p, T ; δ ) =
ϵ
On the other hand, the choice of clipping norm R is crucial to the performance of DP models, yet thehyperparameter tuning is much labor intensive.
Recent advances of DP deep learning on ImageNet[34] and on E2E datasets [36], using ResNet18 and GPT2 respectively, illustrate that the performanceis very sensitive to R , as we have reproduced in Figure 1.
A closer look at Figure 1 shows that, on
ImageNet, ResNet18 can drop from the highest 45% accuracy to 31% if R is chosen 2 times larger,and to 0.1% if R is chosen 4 times larger.
Similar drastic drop can also be observed in [34, Figure3] even if the noise multiplier σ = 0 .
Unlike the noise multiplier σ , the clipping norm R cannotbe inferred from the privacy budget ( ϵ, δ ) and have to be tuned with caution, as R affects both theclipping operation and the noise addition in (1.1).
Consequently, DP training necessarily requires a2D grid search for ( R, η ) , like the right plot of Figure 1, whereas the regular training only requires aneasy 1D grid search for η .
Even worse, the difficulty of tuning a per-layer clipping norm vector [43],i.e. one clipping norm for one layer, may increase exponentially to the number of layers.
To save the effort of tuning R , previous researches have proposed different approaches to efficientlyset R .
In [3, 51, 24], researchers advocate to use data-adaptive information to select R , such as aspecified quantile of the gradient norm distribution.
These adaptive clipping methods can be a little adhoc: they often replace the the need to tune R by the need to tune one or more new hyperparameters,e.g. the quantile to use and the ratio to split the privacy budget between the quantile decision and thegradient perturbation.
Another approach used by the practitioners is to replace an expensive 2D gridsearch by multiple cheaper 1D grid searches.
For example, the researchers propose, in [34, Section3.3] to fine-tune η with non-DP SGD, fix η and sweep over various values of the clipping norm Rwith DP-SGD, then further fix R and do one more grid search on η .
However, tuning R formally in adata-dependent way (e.g. through cross-validation) introduces additional privacy loss [48], and mostexisting empirical work does not privately conduct hyperparameter tuning.
We take a completely different route by proposing a new clipping principle that removes R , insteadof coming up with methods to find the appropriate R .
We term our method as the automatic clippingand summarize our contributions as follows.
We propose the automatic clipping in (4.1) that expunge the clipping norm from general DP optimizers, allowing DP learning to be as amenable as regular learning.
We show that automatic DP optimizers are as private and efficient as existing DP optimizers.
We show that automatic DP-SGD converges in the non-convex setting, at the same asymptoticconvergence rate as the standard SGD.
Our theoretical analysis successfully explains the trainingbehaviors in previous empirical works.4.
We demonstrate the superiority of automatic clipping on extensive vision and language tasks,especially with large models including ResNet, RoBERTa and GPT2.5.
In Appendix K, we publicize the modification that easily switches from Abadi’s clipping to ourautomatic clipping in popular codebases, e.g. Opacus and Objax.
Preliminaries
2.1 Differential Privacy
We consider the ( ϵ, δ ) -DP in Definition 2.1, where smaller ( ϵ, δ ) means stronger privacy guarantee.
Definition 2.1 ([21]) .
A randomized algorithm M is ( ε, δ ) -differentially private (DP) if for any twoneighboring 1 datasets S, S ′ , and for any event E ,
[Equation]
In words, DP restricts the influence of an arbitrary sample, so that the information contributed by suchsample is limited and less vulnerable to privacy attacks.
In deep learning, DP is generally achievedby applying the subsampled Gaussian mechanism in Lemma A.1 with “Amplification by Sampling”to privatize the minibatch gradients during training.
As illustrated in Equation (1.1), the subsampled Gaussian mechanism involves (1)
Sampling aminibatch by including each data point iid with probability p (2) per-sample gradient clipping tobound the l 2 norm sensitivity at R and (3) adding independent Gaussian noise proportional to thesensitivity R and σ , which is derived from the privacy loss ϵ .
This can be realized by leveraging avariety of modern privacy accounting tools, such as those based on Renyi DP (or moments accountant)[1, 45, 63], Privacy Loss distribution (Fourier accountants) [33, 26, 70], or Gaussian DP [15, 7].
2.2 Differentially Private optimizers with general clipping operations
Applying the Gaussian mechanism to deep learning leads to various DP optimizers, including DPSGD in (1.3), DP-HeavyBall, DP-AdaGrad, DP-Adam, DP-FedAvg, and DP-FedSGD [43].
Toimprove the performance of DP optimizers, previous researches can be classified into two categories.
The first category, where the majority of researches lie in, works with Abadi’s clipping and focuseson better design of R .
To name a few examples, one can adaptively design R t for each iteration t[3, 51, 24], or design the per-layer clipping norm vector R ∈ R L for L layers [1, 43] so as to applydifferent clipping norm for each layer.
Much fewer works fall into the second category that proposes new clipping method.
In fact, anyfunction Clip : R d → R satisfying ∥
Clip ( g ) · g ∥ ≤ R can serve as a valid clipping function besides
Abadi’s.
For instance, the global clipping [9] proposes Clip global ( g ) :
=
I ( ∥ g ∥ < Z ) · R/Z to correctthe bias in the private gradient and alleviate the mis-calibration issue of DP classifiers.
Our automaticclipping also belongs to this category.
We note that different clipping methods work orthogonally tooptimizers, network architectures and gradient norm computation (see Section 7).
Motivation
3.1 Small clipping norm works best
[Figure]
Recent studies have witnessed that DP optimizers (1.1) achieve state-of-the-art (SOTA) results withsmall clipping norms R , in both computer vision and language tasks.
In [36], GPT2 (over 800 millionparameters) and RoBERTa models (over 400 millions parameters) achieve the best results under
DP on QNLI, MNLI, SST-2, QQP, E2E, and DART datasets, with each per-sample gradient clippedto length R = 0 . 1 .
In [34, 13, 44], ResNets and Vision Transformers achieve the best DP resultson ImageNet with R = 1 ; in [60], the best DP results on CIFAR10 use R = 0 . 1 with ResNeXt-29and SimCLRv2 [11].
The effectiveness of small clipping norm together with proper learning rate isdepicted in Figure 1.
Intuitively, smaller R implies that the Abadi’s clipping (3.1) happens, which meansmin (cid:0) R/ ∥
g i ∥ , 1 (cid:1) =
R/ ∥ g i ∥ .
Given that the clipping norm R is so small compared to the number of parameters in large neural networks, and that strong DP is guaranteed when the number oftraining iterations is small (i.e. ∥ g i ∥ has not converged to small values yet), we expect and empiricallyobserve that the clipping happens on a large proportion of per-sample gradients at all iterations.
For instance, we find in the GPT2 generation experiments in [36] that 100% of per-sample gradients areclipped at all iterations; in classification tasks such as QQP/QNLI/MNLI, the percentage of clippingis about 20 ∼ 60% on average by Appendix H.1.
normalization as new clipping: a similarity viewpoint
In the small clipping norm regime, we can approximately view
[Equation]
and thus derive a novel private gradient (cid:80)
i R g i ∥
g i ∥ + σR · N (0 , I ) .
Here AUTO-V stands for thevanilla automatic clipping, which essentially performs the gradient normalization on each per-samplegradient.
As a specific example, we can write the R -dependent automatic DP-SGD as R -dependent DP
[Equation]
We may view our AUTO-V clipping as to maximize the dot-product similarity between the clippedgradient and the regular gradient, a commonly used similarity measure, e.g. in the attention block intransformers [61].
Suppose we want
[Equation]
Note that the constraint is a sufficient condition for clipping, as discussed in Section 2.2.
It is nothard to see that the optimal clipping factor is
[Equation]
[Figure] If the per-sample gradients are indeed concentrated in the sense ∀ i, ⟨ g i , (cid:80) j g j ⟩ ≥ 0 , then AUTO-Vis the optimal per-sample gradient clipping.
We compare with Abadi’s clipping in Figure 2, wherethe dot-product similarity is significantly magnified by our AUTO-V clipping.
[Figure]
127 3.3 Stability constant breaks scale-invariance and remains stationary
[Figure]
One potential drawback of AUTO-V clipping is that allgradients lose their magnitudes information completely,since ∥ g i · Clip AUTO-V ( g i ; R ) ∥
=
R, ∀ i .
This scaleinvariance in AUTO-V and partially in Abadi’s clipping(when ∥
g i ∥ > R ) leads to the "lazy region" issue: theparameters will not be updated by DP-GD even if thetrue gradients are non-zero.
In Figure 3, we illustratein a logistic regression 2 that AUTO-V and Abadi’s clipping have zero clipped gradient for the trainable parameter θ ∈ [ − 2 , 2] , as the per-sample gradients from twoclasses cancel each other.
Another benefit of γ is toremain stationary as g i → 0 , i.e. making C
i g i
→ g i /γsmall rather than having magnitude R in AUTO-V.
We elaborate this point in Section 4.
To preserve the magnitude information and thus escape the lazy region, we propose the AUTO-Sclipping, with a positive stability constant γ :
[Equation]
We visualize in Figure 4 that AUTO-S allows larger per-sample gradients to have larger magnitudesafter the clipping, which in turn allows DP-SGD AUTO-S (but not DP-SGD AUTO-V ) to converge to zerogradient norms in Section 5.
Automatic DP Training
One may wonder why our clipping (3.1)(3.3) is automatic at all, if the hyperparameter R is stillpresent and there is an additional parameter γ to choose.
It turns out that any constant choice of R > 0 is equivalent to choosing R = 1 , and common deep learning optimizers are insensitive tothe choice of γ (see ablation study in Figure 13).
Consequently, we choose γ = 0 .
01 in all ourexperiments.
Specifically, let us redefine
[Equation]
With this clipping, we can design automatic DP optimizers similar to (1.1):
Automatic DP Optimizer ( { g i } Bi =1 ) =
Optimizer (ˆ g t ) where ˆ g t := (cid:88) i ∈ B t g t, i [Equation]
[Equation]
Clearly, the new private gradient ˆ g t from our automatic clipping is R -independent, in contrast to theone used in (1.1).
A concrete example (in the case of γ = 0 ) that is comparable to (3.2) will be
R -independent DP-SGD AUTO-V :
[Equation]
Leveraging the private gradient ˆ g t in (4.2), we can train DP neural networks without tuning DPspecific hyperparamters R and σ , as demonstrated in Algorithm 1.
We will elaborate two distinct reasons in each section for the following statement:
DP Optimizer Abadi ≈ R -dependent DP Optimizer AUTO ≡ R -independent DP Optimizer AUTO which reduces the hyperparameter tuning of DP training to that of the regular training, i.e. only onlearning rate, weight decay, etc.
The significant save in the tuning effort is illustrated in Figure 14.
4.1 Non-adaptive optimizer couples clipping norm with learning rate
With R -dependent automatic clipping, DP-SGD becomes
[Equation]
We can view η effective := ηR as a whole: increasing R has the same effect as increasing η , whichexplains the diagonal pattern in Figure 1(right plot) where DP-SGD Abadi is applied with small clippingnorm 3 .
We extend to general non-adaptive optimizers in Theorem 1, with proof in Appendix B.1.
Theorem 1.
Non-adaptive R -dependent automatic DP optimizers (including SGD, Heavyball[52]and NAG[47]), with learning rate η and weight decay λ , is equivalent to R -independent automatic
DP optimizers, with learning rate η ′ = ηR and weight decay λ ′ =
λ/R .
4.2 Adaptive optimizer can be insensitive to clipping norm
Adaptive automatic DP optimizers are different than the non-adaptive ones, as the clipping norm cancels out instead of being coupled with learning rate.
To see this, we scrutinize DP-Adam Abadi (which is similar to DP-Adam AUTO-V ) in Figure 1(left plot), where columns to the left are almost identical.
Further evidence is observed in [44, Table 5] that shrinking R has zero effect on LAMB.
We now give a simple explanation using AdaGrad [18]:
[Equation]
where g t =
(cid:80) i g t,i is the gradient sum and G t =
(cid:80) τ<t g 2 τ is sum of gradient square by Hadamard product over the past iterations.
In R -dependent DP-AdaGrad AUTO-V , the private gradient is R ˆ g t in place of the standard gradient sum g t , and ˆ G t =
R 2 (cid:80) τ ≤ t ˆ g 2 τ :
[Equation]
We generalize to the general adaptive optimizers in Theorem 2, with proof in Appendix B.2.
Theorem 2.
Adaptive R -dependent automatic DP optimizers (including AdaGrad[18], AdaDelta[69],
AdaMax/Adam[31], NAdam[16], RAdam[39], LARS[65], LAMB[66]), with learning rate η andweight decay λ is equivalent to R -independent automatic DP optimizers with learning rate η andweight decay λ ′ =
λ/R .
With decoupled weight decay[42], R -dependent automatic DP-AdamW isequivalent to R -independent automatic DP-AdamW with the same η and λ .
Similarly, we demonstrate the automatic DP optimizers with per-layer clipping style in Appendix B.3. Automatic clipping guarantees the same level of privacy while
[Figure]
utility
[Figure]
In Theorem 3 (proved in Appendix A), we show thatthe new private gradient ˆ g t in (4.2) has the samelevel of privacy guarantee as the existing one in (1.1),since the global sensitivity remains the same (see Figure 4).
We note that as long as γ > 0 , the magnitude information of per-sample gradients is preservedby AUTO-S, in the sense that ∥ g i ∥ >
∥ g j ∥ ⇐⇒∥ C
i g i ∥ >
∥ C j g j ∥ , whereas this can be violated inboth the AUTO-V and Abadi’s clipping (as depictedby the flat curve in Figure 4 when ∥ g i ∥ > 1 ).
Additionally, note that when γ is small, almost all datapoints “max out” the signal relative to the amountof noise we add.
To say it differently, for the sameamount of noise, AUTO-S with small γ allows more signal to be pushed through a differentiallyprivate channel.
Towards the end of the training, i.e., at the limit ∥ g i ∥ → 0 for all i , then wehave (cid:80) i g i ∥
g i ∥ + γ → 1 γ (cid:80)
i g i .
In words, the clipped gradients become closer to the standard SGD,thus do not suffer from the instability of AUTO-V.
Theorem 3.
Under the noise multiplier σ , number of iterations T , subsampling probability B/n ,
DP optimizers using AUTO-V or AUTO-S clipping satisfy ( ϵ Accountant ( δ, σ, B/n, T ) , δ ) -DP as muchas those using Abadi’s clipping, regardless of the choice of privacy accountant.
Convergence analysis of DP-SGD with automatic clipping5.1 Convergence theory of DP-SGD to stationary points
We highlight that automatic clipping can be more amenable to analysis than Abadi’s clipping in [12],since we no longer need to decide whether each per-sample gradient is clipped.
To analyze the convergence of automatic DP-SGD (4.2) in the non-convex setting, we follow thestandard assumptions in the SGD literature [23, 2, 6], with one additional symmetry assumption onthe gradient noise.
Assumption 5.1 (Lower bound of loss) .
For all w and some constant L ∗ , we have L ( w ) ≥ L ∗ .
Assumption 5.2 (Smoothness) .
Let g ( w ) denote the gradient of the objective L ( w ) .
Then ∀ w , v ,there is an non-negative constant L such that
[Equation]
Assumption 5.3 (Gradient noise) .
The per-sample gradient noise ˜ g t,i − g t is i.i.d.
from some ditribution such that
[Equation]
and ˜ g t,i is centrally symmetric 4 about g t in distribution:
[Equation]
We show in Theorem 4 that DP-SGD with AUTO-S clipping allows the true gradient norm to convergeto zero, but not so with AUTO-V clipping.
We leave the proof in Appendix C.1.
Theorem 4.
Under Assumption 5.1, 5.2, 5.3, running DP-SGD with automatic clipping for Titerations and setting the learning rate η ∝ 1 / √ T give
[Equation]
Here · · · represents the first argument of G , and G is increasing and positive.
As T → ∞ , we havemin t E ( ∥ g t ∥ )
=
O ( T − 1 / 4 ) , the same rate as the standard SGD given in Theorem 8.
[Figure]
Remark 5.4.
In Theorem 4, the upper bound takes an implicit form of G ( · ; ξ, γ ) because it is alower envelope of functions ξr + F ( · ; r, ξ, γ ) over all possible r > 0 , whose forms are detailed in
Theorem 5.
Notice that G results only from the clipping operation, not from the noise addition.
Remark 5.5.
We show in Theorem 5 and demonstrate in Figure 5 that the upper bound (5.2) is alwayslarger than ξ with AUTO-V ( γ = 0 ), and can only be reduced to zero with AUTO-S ( γ > 0 ).
Weprovide real data evidence in Figure 12 that strictly positive γ reduces the gradient norm significantly.
5.2 Analysis of factors affecting the convergence
We now analyze the many factors that affect the convergence in Theorem 4, from a unified viewpointof both the convergence and the privacy.
We start with the stability constant γ and the learning rate η t , both only affect the convergence notthe privacy.
We empirically observe in Figure 7 that small γ benefits the convergence at initialiterations (when the privacy guarantee is strong) but larger γ converges faster asymptotically.
For η t ,the optimal is in fact the miminizer of the hyperbola in (C.4), that is unique and tunable.
Next, we focus on the hyperparameters that affect both convergence and privacy: the batch size B ,the noise multiplier σ , and the number of iterations T .
These hyperparameters have to be consideredalong the privacy-accuracy tradeoff, not just from a convergence perspective.
We denote the firstargument of G in Theorem 4 as X := 4 √ T (cid:113)
( L 0 − L ∗ )
L (cid:0) 1 + σ 2 dB 2 (cid:1) , which is to be minimized.
Given a fixed privacy budget ( ϵ, δ ) , we leverage µ -GDP [15, 7] with µ · n = B (cid:112) T ( e 1 /σ 2 − 1) fortheoretical analysis only 5 .
We summarized the following guidelines for DP training, which matchthe training behaviors observed in SOTA empirical work [36, 34, 13, 60, 44, 68].
[Train longer with larger noise] Fixing B , we have X ∝ (cid:113) 1 + σ 2 dB 2 / √ T ≈ σ √ d B √ T ∝
σ (cid:112) ( e 1 /σ 2 − 1) , which is decreasing in σ .
Hence larger σ and larger T are preferred.2.
[Larger batch size helps] Fixing iterations T or epochs E
= BT , we again have X ∝σ √ e 1 /σ 2 − 1 .
Hence larger σ and larger B are preferred.
[Pretraining is key]
Pretraining can boost the accuracy by starting with a much smaller initialloss L 0 and from a smooth (small L ) and flat (small ξ , c.f. Figure 7(left)) initialization.4.
[Learning rate needs tuning]
The optimal learning rate by minimizing (C.4) is(cid:113) 2( L 0 − L ∗ ) /L/ (cid:0) 1 + σ 2 dB 2 (cid:1) .
We should use larger learning rate for smaller model, weakerprivacy (larger ϵ or small σ ), or larger batch size (c
f. [36, Figure 2]).
Experiments
We evaluate our automatic DP training on image classification, sentence classification, and table-totext generation tasks.
Detailed settings including hyperparameters can be found in Appendix G.
6.1 Image classification
For MNIST/FashionMNIST, we use the same setup as in [49, 60, 57] with a simple CNN.
For
CIFAR10, we use the same setup as in [60] with pretrained SimCLRv2 [11].
For ImageNette [28], a10-class sub-task of ImageNet [14], we use the same setup as in [32] without the learning rate decay.
For CelebA [41], the real human face dataset, we train ResNet9 [27] with group normalization toreplace the batch normalization.
Notice that CelebA contains high-resolution (178x218) images, eachwith 40 labels.
We consider CelebA for either multi-class classification on one label, e [Table] g. ‘Smiling’and ‘Male’, or for multi-label/multi-task problem to learn all labels simultaneously.
[Table]
In Table 1, we observe that AUTO-S clipping outperforms existing clipping in all datasets with statistical significance.
Interestingly, the standard deviation from different runs is smaller for automatic
DP optimizers, indicating better reproducibility and stability.
We additionally experiment 40 binaryclassification problems on CelebA with respect to each label, and observe that the mean accuracyfurther improves to 91.63% at ϵ = 8 for AUTO-S (see Appendix J).
6.2 Sentence classification
On five benchmark language datasets (MNLI(m/mm)[64], QQP[30], QNLI[55], SST2[59]), wecompare our automatic DP training with reparameterized gradient perturbation (RGP, [68]) andfull-parameter finetuning (full, [36]) using RoBERTa models [40].
These methods use the sameexperimental setup.
For language models, our automatic training is based on the codebase of [36]
[Table] [Table]
In Table 2 and Table 3, we note that full parameter finetuning with AUTO-S outperforms or at leastmatches SOTA on all tasks.
We use exactly the same hyperparameters as in [36].
6.3 Table-to-text generation
We compare our automatic DP training with a variety of fine-tuning methods, for table-to-textgeneration task on E2E dataset [19], where the goal is to generate texts about different aspects of arestaurant’s data.
We measure the success on this task by BLEU, ROUGE-L (in Table 4), METEOR,
NIST, CIDEr (extended in Table 7)
[Table]
with higher value meaning better model quality.
[Table]
dataset with GPT2.
Additional performance measures are included in Table 7.
The best two GPT2 models for each row are marked in bold.
Competitive methods include low-rank adaption (LoRA), prefix-tuning (prefix), RGP, only fine-tuningthe top 2 Transformer blocks (top2), and training from scratch (retrain), as were recorded in [36].
Again, we use the exactly the same hyperparameters as in [36].
For GPT2 (163 million parameters),
GPT2 medium (406 million), and GPT2 large (838 million), Table 4 shows that AUTO-S is scalablewith stronger performance on larger models.
Our automatic full-parameter finetuning has the bestoverall performance.
Additionally, we highlight that AUTO-S and methods like LoRA are notmutually exclusive and can be combined to yield strong performance, since AUTO-S modifies theoptimizers and LoRA modifies the architecture.
Discussion
In this work, we proposed AutoClipping as a drop-in replacement to the standard per-exampleclipping differentially private training.
This is the first technique that eliminate the need to tune theclipping norm R , thus making DP deep learning as easy as regular learning.
Our AUTO-S methodenjoys both theoretical guarantee of convergence in non-convex problems (under various conditions),and strong empirical performance that advances the state-of-the-art (SOTA) of DP learning on bothcomputer vision and language tasks.
We are excited about the future of automatic DP training, especially along with other workingtechniques.
Notably, our automatic clipping applies compatibly with general optimizers (e.g. [8, 17]),clipping styles (all-layer or per-layer), architecture modifications (e.g. LoRA, RGP, prefix), and dataaugmentation (e.g. adversarial training [25] and multiple augmentation [13]).
Thus, we expect toachieve comparable results to all SOTA in a lightweight fashion.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[N/A]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Assumptions [Table]
[Table]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A.
If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experimental
[Table]
results (either in the supplemental material or as a URL)?
[Yes] We include our code in [Table] Appendix K, and cite all datasets.
[Table]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they werechosen)?
[Yes] See all details in Appendix G.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments
[Table]
multiple times)?
[Yes] See Table 1.
[Table]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of [Table] GPUs, internal cluster, or cloud provider)?
[N/A]
This work is unrelated to computational [Table] efficiency.
[Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite all codebases,datasets, models clearly.
[Table]
(b) Did you mention the license of the assets?
[N/A] Experiments are conducted on public data.(c)
Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We
[Table]
include code snippet in Appendix K and will include a URL to release the full codebase.(d)
Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[Table]
[N/A]
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review Board [Table]
(IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A
[Table]