Differentially Private Adaptive Optimization with Delayed Preconditioners
Abstract
Privacy noise may negate the benefits of using adaptive optimizers in differentially private model training.
Prior works typically address this issue by using auxiliary information (e.g., public data) to boost the effectiveness of adaptive optimization.
In this work, we explore techniques to estimate and efficiently adapt to gradient geometry in private adaptive optimization without auxiliary data .
Motivated by the observation that adaptive methods can tolerate stale preconditioners, we propose differentially private adaptive training with delayed preconditioners ( DP 2 ), a simple method that constructs delayed but less noisy preconditioners to better realize the benefits of adaptivity.
Theoretically, we provide convergence guarantees for our method for both convex and non-convex problems, and analyze trade-offs between delay and privacy noise reduction.
Empirically, we explore
DP 2 across several real-world datasets, demonstrating that it can improve convergence speed by as much as 4 × relative to non-adaptive baselines and match the performance of state-of-the-art optimization methods that require auxiliary data.
1. Introduction
Adaptive optimizers such as AdaGrad [9, 22] and RMSProp [15] are commonly used to improve convergence speed in machine learning training.
However, in privacy-sensitive applications, the benefits of adaptivity may degrade as a result of noise added to the preconditioners to guarantee differential privacy [19].
Prior works typically address this issue by using non-sensitive auxiliary data to approximate the underlying structures of private gradients [3, 17, 19].
While this can boost performance, assuming access to informative public data may be unrealistic in many privacy-sensitive applications.
In this work, we instead ask: Can we improve privacy/utility trade-offs in private adaptive optimization without accessing auxiliary data ?
A key insight we have in addressing this question is that for many machine learning problems, the gradient geometry may not change drastically during successive steps of optimization (e.g., see Fig.
1, which plots successive distributions of preconditioner values).
This presents an opportunity to estimate the preconditioners used by adaptive optimizers with smaller noise, by averaging across previous iterates.
[Figure]
To this end, we propose DP 2 , a differentially private adaptive method that uses historical gradients to construct delayed preconditioners with reduced noise.
Despite the simplicity of this approach, we find that it can significantly improve performance in practice—improving convergence speed by as much as 4 × relative to non-adaptive baselines, all without the need to access auxiliary data.
To better understand these performance gains, we theoretically and empirically analyze the method to study the effect of using delayed preconditioners, including trade-offs that emerge between the noise reduction and staleness.
Contributions.
We propose DP 2 as a method for differentially private adaptive optimization with delayed preconditioners.
Unlike prior work, DP 2 does not rely on auxiliary data to improve privacy/utility trade-offs in private training.
We provide convergence guarantees for DP 2 in both convex and non-convex settings, and analyze the trade-offs between delay and privacy noise.
We conduct extensive experiments to showcase the effectiveness of DP 2 , which can significantly improve model utility for a given privacy budget across text and recommendation benchmarks.
2. Background and Related Work
In this section we discuss closely related works.
We defer full related works to Appendix A. We consider the classic framework of ( ε, δ ) -differential privacy (DP) [10], and the popular subsampled
Gaussian mechanism [11, 23] to perturb gradients to ensure DP.
To reduce privacy cost in iterative
DP algorithms, it is natural to consider applying adaptive optimizers (e.g., AdaGrad [9, 22], RMSProp [15], AMSGrad [25], and Yogi [30]) to speed up convergence.
A straightforward approach is to first privatize mini-batch gradients and then plug in noisy gradients to any adaptive updating rules [32].
However, estimating gradient moments in this way may yield preconditioners with too much noise, resulting in adaptive methods that may not have meaningful improvements over
DP-SGD [19].
As we discuss in Section 1, more recent works suggest the use of non-sensitive public information to estimate the preconditioners (or other gradient structures) [3, 17, 19], which may not always be available in practice.
In Section 5.2, we empirically benchmark two baselines along this line of work and demonstrate that DP 2 can perform comparably to these state-of-the-art methods, even though it does not require access to auxiliary data.
Notation.
We aim to use adaptive optimization methods to solve the classic empirical risk minimization objective, i.e., min w F ( w ) =
1 n (cid:80) ni
=1 f ( x i ; w ) , where w ∈ R d and { f ( x i ; w ) } i ∈
[ n ] are individual loss functions on training sample i ∈ [ n ] .
For vectors u, v ∈ R d , we use u + v for coordinate-wise addition, and uv for coordinate-wise division.
For any vector v , v j denotes the j -th coordinate of v .
Finally, | v | ∈ R d denotes taking coordinate-wise absolute values, and ∥ · ∥ M denotes the matrix norm defined as ∥ ·
∥ M := (cid:112) ⟨· , M ·⟩ for a symmetric and positive definite matrix
M ∈ R d × d , or a diagonal matrix with non-negative diagonal entries populated by a vector M ∈ R d .
3. DP 2 : Delayed Preconditioners for Differentially Private Adaptive Optimization
We now introduce our DP 2 framework.
While we discuss DP 2 in the context of a particular adaptive method (RMSProp), we note that the approach is method-agnostic in that it can generally be applied to any private adaptive optimization method where preconditioners are calculated at each iteration (Appendices D.5 and E).
As an initial step towards understanding the algorithm, we first investigate the effects of delayed preconditioners in non-private training.
3.1. Delayed preconditioners in non-private settings
Adaptive methods use preconditioners to adapt to gradient geometry, effectively resulting in coordinatewise learning rates.
This can be advantageous for many applications, especially those with sparse gradients or non-uniform stochastic noise [e.g., 15, 22, 24, 31].
One of the key design choices of DPis to update preconditioners less frequently and use the average of past gradients to reduce noise.
Our observation is that a wide range of learning problems are tolerant to the staleness of preconditioners.
In this subsection, we validate this empirically on the benchmark datasets considered throughout this paper.
There are potentially many ways that one could instantiate the idea of delayed preconditioner computation in adaptive optimization.
Here we consider a specific algorithm, which is the exact non-private version of our proposed DP 2 framework (Algorithm 1) introduced in later sections.
As motivation for DP 2 , we empirically show that RMSProp with delayed preconditioners achieves almost the same optimization performance as RMSProp (Figure 2).
[Figure]
3.2. Constructing delayed preconditioners with reduced noise
Without access to public data or other side information, prior works typically update preconditioners based on noisy gradients at each iteration [32].
For instance, a natural way to privatize RMSProp is to update the preconditioner v ∈ R d as v ← βv
+ (1 − β )(˜ g ) 2 where β ∈ (0 , 1) is a moving average constant, and ˜ g ∈ R d is the noisy gradient output by some standard privacy mechanism (e.g., the Gaussian mechanism).
However, a drawback to this is that the noise gets accumulated at each iteration, making adaptive methods significantly less effective [19].
Inspired by the observation that problems can be tolerant to the staleness of preconditioners (Figure 2), we propose to update the preconditioners less frequently to reduce noise.
For instance, we update v every s steps using some aggregate function of s recent private gradients from DP-SGD.
During iterations where v is not updated, we simply apply the most recent (stale) v to precondition the gradients.
In order to mitigate the noise, we average over these s gradients to form a pseudo-gradient g , which can be plugged into arbitrary adaptive optimization algorithms.
Note that the privacy noise variance will be reduced s times if we average s Gaussian random variables (i.e., the DP noise).
[-While in principle, non-adaptive and adaptive updates can take different numbers of consecutive iterations, in our empirical evaluation, we simply set s 
-]
Algorithm
Input: T , batch size b , noise multiplier σ , clipping thresholds C , initial model w 0 ∈ R d , v = 0 , constant ϵ ∈ R + , learning rate schedule α t , moving average parameter β , SGD cumulative aggregation step s 1 , RMSProp cumulative step s1 for t = 0 , · · · , T − 1 do if t mod ( s 1 + s 2 )
=
then
Reset accumulator G t ← end if t mod ( s 1 + s 2 ) =
s 1 then
Update moment estimates as v ← βv
+ (1 − β ) (cid:0) G t /s 1 (cid:1) 2 , reset accumulator G t ← end
Uniformly randomly sample a mini-batch B with size b from private training data
Get individual gradients for sample i ∈ B : g i,t ← ∇ f ( x i ; w t )
Privatize the (preconditioned) gradients using the Gaussian mechanism: ˜ g t ← 1 b (cid:32)(cid:88) i ∈ B clip (cid:18) g
i,t
D t , C (cid:19) + N (cid:0) 0 , σ 2 C 2 (cid:1)(cid:33) where
D t ← (cid:40)if t mod ( s 1 + s 2 ) <
s√ v + ϵ otherwise.
Accumulate the private gradients ˜
g t : G t +1 ← G t + ˜ g t
Update model parameters w : w t +1 ← w t − α t ˜ g t 13 end 14 return w T DP 2 is summarized in Algorithm 1. For simplicity of presentation, we assume RMSProp as the adaptive method (denoted as DP 2 -RMSProp) throughout this section. The high-level idea is to average noisy gradients to construct a cleaner estimate of the preconditioner (Line 6 and 11 in
Algorithm 1) and apply it for adaptive optimization (Line 10).
As these two steps access raw gradients twice, we need to privatize them separately. Unfortunately, the privacy budget would accumulate with each query to the raw training data. Hence, we use the private SGD gradients for both the model update and the preconditioner estimation. This results in a hybrid method that alternates between private SGD and private adaptive optimization steps.
{+While in principle, non-adaptive and adaptive updates can take different numbers of consecutive iterations, in our empirical evaluation, we simply set s 
+}
= s 2 , and find that this works reasonably well across all datasets (Section 5).
Privacy guarantees.
From Algorithm 1, we see that at each iteration, we access raw data and pass them through the privacy barrier once (Line 10) to generate private gradients ˜
g t with the same noise multiplier σ and batch size b , and the preconditioner only accumulates already differentially private gradients.
Since the final model is a composition of these private releases (noisy gradients),
Algorithm 1 (or DP 2 in general) achieves the same privacy guarantees as standard DP-SGD training under the same training settings.
We formally state the privacy guarantee in Appendix B.
4. Convergence Analysis
In this section, we analyze Algorithm 1 for convex problems, and defer readers to Appendix B for non-convex analysis.
We aim to study the convergence properties of DP 2 and investigate the trade-offs between delay and privacy noise.
In doing so, key challenges are introduced by alternating between adaptive and non-adaptive updating and through the staleness of preconditioners.
For convex functions, we define the optimal model w ∗ as w ∗ ∈ arg min w F ( w ) .
First we state some assumptions (apart from convexity) that are used in the analysis.
Assumption 1 There exists a constant R such that ∥ w t − w ∗ ∥ 2 ≤ R for any iteration t.
Assumption 2 There exists a constant C such that (cid:13)(cid:13)
g i,t (cid:13)(cid:13) 2 ≤ C for any i ∈ [ n ] and iteration t .
Assuming bounded stochastic gradient norm is standard in prior works on convex and non-convex private optimization [e.g., 17, 19, 32].
Under this assumption, suppose the clipping does not happen, we have ˜ g t ← g t + N (0 , σ 2 C 2 /b 2 ) , where g t := 1 b (cid:80) i ∈
B g i,t .
Without loss of generality, let s 1 = sin Algorithm 1.
Our main convergence result is stated as follows (assuming t starts from 1).
Theorem 1 (Convergence of Algorithm 1 for convex problems) Let Assumptions 1 and 2 hold.
Assume F is a convex function.
Let the learning rate α t be set as α t ← α
⌊ t 2 s ⌋ + ⌊ t + s 2 s ⌋ +1 √ t .
After running Algorithm 1 for T iterations with s = υT for a small constant υ ∈ (0 , 1] , we obtain min t ∈
[ T ] E
[Equation]
√ t
E [ ∥ N t ∥
[Equation] D t ] [Equation] where T υ denotes the iteration indices where we switch from private RMSProp steps to private SGD steps plus the last iteration, with cardinality | T υ | = ⌈ 12 υ ⌉ , N t ∼ N ( 0 , σ 2 C 2 /b 2 )
[Equation]
and κ ≥ max
[Equation] where M :=
C 2 + σ 2 Csb 2 .
We defer all proofs to
Appendix B.
The above convergence rate relies on a critical metric h ( s ) which is related to temporal gradient similarity and the amount of staleness s , formally defined as:
[Equation]
[Figure]
where the expectation is taken with respect to all randomness in the algorithm, and G ⌊ ts ⌋ s ∈ R d refers to the latest accumulator that is used to update v (Line 6 in Algorithm 1).
A smaller h ( s ) indicates better convergence.
We see that the denominator of h ( s ) can be decomposed into the average of past raw gradients and the average of random Gaussian noise.
In Appendix B.3, we show that an upper bound of h ( s ) can be expressed as c 1 + c 2 s where c 1 , c 2 are two constants.
We also visualize the value of h ( s ) on the IMDB dataset in Figure 3, and show that (1) the values of h ( s ) are consistently small across all delays, and (2) h ( s ) increases as the s gets larger, which is consistent with the expression of s .
Trade-offs between delay and noise.
Intuitively, larger s (larger delay) results in staler preconditioners, but introduces less noise due to private gradient averaging.
In our convergence bound, there are several terms that depend on s (or υ ).
Although this makes it difficult to derive a closed-form characterization of an optimal s , we can analyze the effects of s in simplified settings.
In particular, examine the first term of the RHS of the convergence bound, let α =
[Equation]
M + ϵ = √ c + c 4 υ
+ ϵ (where c 3 , c 4 are two constants), and assume (cid:4) 12 υ (cid:5) + (cid:4) 1+ υ 2 υ (cid:5) = 12 υ + 1+ υ 2 υ = 2+ υ 2 υ .
Combined with h ( s ) , the dependence on υ in
[-R 
-][Equation] ⌋ can be expressed as ( c [Equation]
+ c [Equation] υ ) (cid:0)(cid:112) [Equation]
+ [Equation]
(cid:1) 2+ [Equation] 2 υ .
This suggests that there exists an optimal υ that achieves the minimal value.
In Section 5.1, across real-world datasets, we demonstrate that there exist specific ranges of s that provide favorable trade-offs between delay and noise (Figure 5).
Discussions.
Similar to previous results on private adaptive optimization [3, 19], Theoremshows that when the gradients are sparse and the delay parameter s is not too small (so that (cid:13)(cid:13)
D t (cid:13) (cid:13) 1 < d in adaptive iterations), we are able to reduce the amount of added noise added compared with DPSGD.
If s = T (i.e., running DP-SGD all the time), our result reduces to the convergence rate of
DP-SGD in convex settings with learning rates α t =
O (1 / √ t ) .
5. Empirical Evaluation
In all experiments, we use R´enyi differential privacy (RDP) accountant for the subsampled Gaussian mechanism [23] for privacy accounting.
We focus on the RMSProp optimizer [15] and provide results relating to other adaptive methods such as AdaGrad [9, 27] in Appendix D. See also Appendix D. for implementation details and hyperparameter values.
Tuning s .
In all experiments, we tune the delay parameter ( s ) via grid search.
For convex tasks, we choose s from { 0 . 025 , 0 . 5 , 0 . 1 , 0 . 5 , 1 , 2 } epochs.
For the non-convex model, we choose s from { 0 . 5 , 3 , 10 , 25 } epochs.
We explore the sensitivity of DP 2 to s in Section 5.2, and show that there exist a wide range of s parameters that result in superior performance compared with baselines.
5.1. DP 2 compared with DP-SGD and vanilla DP adaptive methods
We consider two popular baselines: DP-SGD [1] and vanilla DP-RMSProp [32].
In vanilla DP adaptive methods, private gradients are plugged into adaptive updating rules to approximate the preconditioners at each iteration.
Privacy/utility trade-offs.
Here, we explore privacy/utility trade-offs across a range of privacy parameters, where ε ranges are consistent with prior works [e.g., 18].
Results are shown in Figure 4.
We observe that, DP 2 significantly outperforms DP-SGD and DP-RMSProp under each privacy budget.
For reference, the non-private RMSProp method achieves 87% accuracy, 62% accuracy, and 0.88 mean square error (MSE) on IMDB, StackOverflow, and MovieLens, respectively.
Indeed, with weaker privacy (larger ε ), we expect smaller utility gaps between private and non-private optimization.
See Figure 6 in the appendix for optimization curves under one ε value for each dataset.
In Appendix D.6, we additionally explore how increasing the computational budget may affect the privacy-utility trade-off.
Effects of s .
We empirically study the effect of the delay parameter s .
Intuitively, there exists a trade-off between the amount of delay and the privacy noise in the preconditioner: averaging over
[Figure]
[Figure]
more historical gradients (larger s ) could yield less noisy preconditioners, while introducing more staleness.
In Figure 5, we report test performance versus the delay s across all datasets on the first three subplots.
In the last subplot, we additionally show the convergence behavior under different values of s .
These results suggest that there is a “sweet spot” for s to yield good performance— small delays are gradually improving over DP-RMSProp; moderate delays perform best in terms of convergence and absolute performance; and large delays may slow down convergence (although it is possible to reach similar performance with sufficient training).
5.2. DP 2 compared with recent methods for private optimization
As discussed in Section 2, beyond DP-SGD and vanilla DP adaptive methods, another line of work uses auxiliary, public data to improve private (adaptive) optimization.
While not directly comparable to DP 2 since DP 2 does not require any side/public information, we compare DP 2 to two state-ofthe-art methods along this direction : (1) AdadPS [19] which uses public data or their statistics to estimate gradient geometry, and (2) PDA-DPMD [2], which uses the loss on public data as a mirror map to learn the underlying gradient geometry.
Results are reported in Table 1, which show that DPhas comparable performance to state-of-the-art baselines, but without the need to access auxiliary data.
See Appendix D.8 for full details and convergence curves.
[Table]
5.3. Ablation Studies
Finally
[Table]
we also study the effectiveness of different components of DP 2 .
Due to space constraints [Table] we defer interested readers to Appendix D.7 for the results.
Appendix A. Related Works
Definition 2 (Differential privacy [10])
A randomized algorithm M is ( ε, δ ) -differentially private if for all neighboring datasets D, D ′ differing by one element, and every possible subset of outputs
O ,
[Equation]
Differentially Private SGD.
Informally, DP in machine learning offers protection by masking the influence of individual examples (example-level DP, e.g. [1, 4, 26]) or all of the examples from one user (user-level DP, e.g. [18, 21]) on the trained model.
In this work, we consider example-level DP using the popular subsampled Gaussian mechanism [11, 23] to perturb gradients to ensure DP.
Unless much larger batch sizes and possibly larger datasets are used, DP mechanisms often lead to a significant utility drop. So extensive research has investigated improving privacy/utility/computation trade-offs for DP-SGD, including various training techniques (e.g., data augmentation and large-batch training)
[7], leveraging public data [2, 33], and releasing gradient statistics via tree aggregation to reduce the amount of noise [6, 8, 18].
These prior works are orthogonal to and could be applied in conjunction with our proposed method, which focuses specifically on privacy in the context of adaptive optimization.
Differentially Private Adaptive Optimization.
To reduce privacy cost in iterative DP algorithms, it is natural to consider applying adaptive optimizers (e.g., AdaGrad [9, 22], RMSProp [15], AMSGrad [25], and Yogi [30]) to speed up convergence.
A straightforward approach is to first privatize mini-batch gradients and then plug in noisy gradients to any adaptive updating rules [32].
However, estimating gradient moments in this way may yield preconditioners with too much noise, resulting in adaptive methods that may not have meaningful improvements over DP-SGD [19].
As we discuss in Section 1, more recent works suggest the use of non-sensitive public information to estimate the preconditioners (or other gradient structures) [3, 17, 19], which may not always be available in practice.
In Section 5.2, we empirically benchmark two baselines along this line of work and demonstrate that DP 2 can perform comparably to these state-of-the-art methods, even though it does not require access to auxiliary data.
Finally, we note that previous works have explored the high-level direction of delayed preconditioners, but mainly as a compromise for computational considerations in non-private training [12].
In this work, we instead show that staleness can be leveraged to improve privacy/utility trade-offs in private adaptive optimization, and propose and analyze a novel method for delaying preconditioner computation in the context of private training.
Appendix B. Theorems and Proofs
Theorem 3 (Privacy guarantee of Algorithm 1 [1])
There exist constants c 1 and c 2 such that for any ε < c 1 b 2 T/n 2 , Algorithm 1 is ( ε, δ ) -differentially private for any δ > 0 if σ ≥ c 2 b √ T log(1 /δ ) nε .
Lemma 4 Under Assumption 2, let s
= s
= s in Algorithm 1, we have for any j ∈ [ d ] ,
E [ v j ] ≤ C 2 + σ 2 C 2 sb 2 .
Proof Recall that C is the gradient norm bound (Assumption 2).
Let the clipping threshold be C as well.
We have for j ∈ [ d ] ,
[Equation]
where { i 1 , . .
, i s } denotes the indices of s noisy gradients used to obtain G j , and { N i 1 j , . . .
, N i s j } are random zero-mean Gaussian variables with variance σ 2 C 2 bunder noise multiplier σ , clipping threshold C , and mini-batch size b .
Hence for any j ∈ [ d ] and t ∈ [ T ] ,
[Equation]
B.1. Convergence for Non-Convex Cases
In non-convex cases, we make the following additional assumptions, which are common in nonconvex convergence analyses.
Assumption 3 (Smoothness)
Each f ( x i ; w ) ( i ∈ [ n ]) is L -smooth with respect to w ∈ R d .
Assumption 4 Stochastic gradient variance is bounded, i.e., E [ ∥ g i,t − E [ g i,t ] ∥ 22 ] ≤ τ 2 for all i, t .
Theorem 5 (Convergence of Algorithm 1 for non-convex problems.)
Let Assumptions 1-4 hold.
Define constant M as M :
=
C 2 + σ 2 C 2 sb 2 .
Under any delay parameter s , after running Algorithmwith constant learning rates α t = α such that Lαϵ ≤ 1 , we have
[Equation]
The proof is deferred to Appendix C. Compared with Theorem 1, here we do not have constraints on s .
Note that to guarantee ( ε, δ ) -DP by running T iterations, we can set σ 2 = O (cid:16) b 2 T log(
[Equation]
n 2 ε(cid:17) ,
α = O (cid:16) 1 √ d (cid:17) , and T =
O (cid:16) nε log(1 /δ )
(cid:17) , to arrive at a convergence bound O (cid:16) √ dnε + τ 2 √ db (cid:17) .
Under any s , our rate (with and without noise) is the same as previous results on DP-SGD and (DP) adaptive methods for non-convex problems [19, 30].
We note that our non-convex analysis does not directly highlight the benefits of adaptivity or trade-offs around s ; hence the optimal choice of s according to this result is s = T , to maximize the goal of reducing privacy noise.
B.2. Proof of Theorem
Based on the updating rule
[Equation]
we have
[Equation]
Rearranging terms gives
[Equation] [Equation]
Taking the expectation on both sides conditioned on w t ,
[Equation]
where we have used the fact that N is a zero-mean Gaussian variable independent of g t , w t .
Taking the expectation on both sides and using the convexity of F ( · ) :
[Equation]
Applying telescope sum
[Equation]
we have
T
[Equation]
Hence, we need to bound the RHS:
[Equation]
where the vector D t ∈ R
d satisfies that D t = 1 when running private SGD steps, and D t = √ v
+ ϵ when running private RMSProp steps.
Let the delay parameter to be scheduled as
[Equation]
) (17) and the learning rate α t be
[Equation] [Equation]
where α = min (cid:110) ϵ,
[Equation]
(cid:111) , and M is the upper bound of E [ v j ] for j ∈ [ d ] , as defined and proved in Lemma 4.
We next consider the T 1 term.
There are four cases.
DP-SGD at the t − 1 -th iteration, and DP-SGD at the t -th iteration :
As D t = D t − 1 there is
not much requirement other that the learning rates need to satisfy α t ≤ α t − 1 , which holds for our choice.
Private RMSProp at the t − 1 -th iteration, and private RMSProp at the t -th iteration : Similar to previous case, the learning rates need to satisfy α t ≤ α t − 1 , which holds for our choice.
DP-SGD at the t − 1 -th iteration, and private RMSProp at the t -th iteration : We require
[Equation]
But in this case we must have t % s = 0 .
So this is satisfied by our choice as long as α ≤ ϵ .
4. Private RMSProp at the t − 1 -th iteration, and DP-SGD at the t -th iteration
The first three cases form an updating pattern of DP-SGD → · · · → DP-SGD → DP-RMSProp → · · · → DP-RMSProp, where every pattern takes 2 s iterations, except for the first pattern, because the telescope sum starts from t = 2 .
For the first pattern, we have
[Equation]
where D 2 s = √ v
+ ϵ .
For k ≥
[Equation]
we have
[Equation]
23) where D 2 sk +2 s = √ v + ϵ belong to DP-RMSProp updates.
We look at the second T 2 term
[Equation]
and prove by induction that there exists a constant κ such that
[Equation]
When T = 1 ( α
=
α and D 1 = 1 ), α 2 E [ ∥ g 1 ∥ 2 ] ≤ κdα holds if κ ≥ α 2 C 2 .
At each step t , the goal is to get
[Equation]
DP-SGD at the t − 1 -th iteration, and DP-SGD at the t -th iteration : We require
[Equation]
which would hold for choice of α t as gradients are bounded and κ ≥ α 2 C 2 .
We need
[Equation] [Equation]
Let
[Equation]
Based on our updating rule,
[Equation]
Note that
[Equation]
where we have used the assumption that ∥ g t ∥ ≤ C .
Combining the above two,
[Equation]
This implies the condition holds as long as κ satisfies
[Equation] [Equation]
3. DP
-SGD at the t − 1 -th iteration, and private RMSProp at the t -th iteration.
We want to prove
[Equation] [Equation]
As (cid:13) (cid:13) g t (cid:13) (cid:13) ≤ C , it holds that
[Equation]
Therefore,
[Equation]
Based on our learning rate set in Eq.
[Equation]
Hence,
Ch ( s )
[Equation]
where we require
[Equation]
Private RMSProp at the t − 1 -th iteration, and DP-SGD at the t -th iteration.
We need
[Equation]
Plug in E (cid:104) ∥√ v t − 1 ∥ 1 (cid:105) ≤ d √ M (Lemma 4) and ∥
g t ∥ 2 ≤ C 2 , we have
[Equation]
Based on our learning rate set in Eq.
(18), for some constant γ ,
[Equation]
Therefore
[Equation]
48) holds as long as κ ≥ α 2 C 2 .
To sum up, the requirement on κ is
[Equation]
Final convergence results:
[Equation]
where T v denotes the iteration indices where we switch from private RMSProp steps to private
SGD steps plus the last iteration, and its cardinality is | T υ | = ⌈
[Equation]
⌉ , and κ ≥ max (cid:110)
α
[Equation] [Equation]
α = min
[Equation]
(cid:111)
[Equation]
B.3. A closer look at h ( s )
We closely examine h ( s ) , defined as
[Equation]
Let us assume mini-batch gradients on consecutive time steps are not very different, i.e. ∥ g t − g t −∥ 1 ≤ M .
This means each gradient norm cannot be too far away from each other, which can be used to show the dependence of h ( s ) on the delay parameter s .
Denote the gap between the current iteration t and the iteration where v gets updated as k , i
[Equation]
e., k :=
t − ⌊ ts ⌋ s .
Hence,
[Equation]
dϵ
[Equation]
Denote a := 1 s (cid:0) N t − k − 1 + · · · + N t − k − s (cid:1) , and b := 1 s (cid:0) g t − k − 1 + · · · + g t − k − s (cid:1) .
Then
[Equation]
In the special case where gradients are sparse, i.e., E [ ∥ b ∥ 1 ] < E [ ∥ a ∥ 1 ] , we have
[Equation]
It is easy to see that the RHS is O ( s ) , and it increases as s .
We can informally express it as c 1 s + c 2 , where c 1 and c 2 are two constants.
Appendix C. Proof of Theorem
First we introduce a result that will be used in this section.
Under the bounded stochastic gradient variance assumption (Assumption 4), we have that conditioned on w t ,
[Equation]
where b refers to the mini-batch size to obtain gradient g t , i.e., g t ← 1 b (cid:80) i ∈
B g i,t .
This lemma is proved in Zaheer et al.
The per-coordinate version of this result is that for j ∈ [ d ] ,
[Equation] [Equation]
and (cid:80) j ∈ [ d ] τ 2 j = τ 2 .
As we assume F ( w ) is L -smooth, at each iteration t ,
[Equation]
Based on the updating rule of Algorithm 1, we have
[Equation]
where N ∈ R d and N j ∼ N
[Equation] [Equation]
b
[Equation]
with noise multiplier σ and clipping threshold C , and D t satisfies that
[Equation]
Take expectation with respect to samples at the t -th iteration and N t ,
[Equation]
where we have used the fact that N t is a zero-mean random variable independent of w t , and D t is independent of samples at time t .
We need to consider two cases.
1. DP-SGD at the t -th iteration
In this case, D t = 1 .
Hence plugging in
[Equation]
we have
[Equation]
Under constant learning rate
[Equation]
let α t = α ≤ 1 L ,
[Equation]
Taking expectation on both sides gives
[Equation]
2. Private RMSProp at the t -th iteration
We have
[Equation]
Plugging in E t (cid:104) ( g tj ) 2 (cid:105) ≤ τ 2 j b
+ (cid:0) ∇ j F ( w t ) (cid:1) 2 results in
E t [ F ( w t +1 )] (72) ≤
[Equation]
≤
[Equation]
( w
[Equation]
−
[Equation]
Taking expectation on both sides yields
[Equation]
We need to lower bound (cid:80) j ∈
[ d ] E
[Equation]
+
[Equation]
(cid:35) .
We know from Holder’s inequality that E [ ⟨ u, v ⟩ ] ≤
E [ ∥ u ∥ 1 ] E [ ∥ v ∥ ∞ ] .
Now note that
[Equation]
Hence (cid:88)
j ∈ [ d ]
E
[Equation]
and
[Equation]
Let α t = α ≤ ϵL , we obtain
[Equation]
Combining the two cases, for any t
[Equation]
we have
[Equation]
Taking a telescope sum results in
[Equation]
where M :=
C 2 + σ 2 C 2 sb 2 .
Appendix D. Experimental Details and Additional Results D.1. Datasets and Tasks
IMDB [20] is a binary classification dataset on sentiment analysis for movie reviews that includes 25,000/25,000 training/test samples.
Each sample is a review under a vocabulary size of 10,000.
We train a logistic regression model with 10,001 parameters.
StackOverflow [16, 29] is a large-scale text dataset containing questions and answers from
Stack Overflow.
We focus on the task of classifying the tag(s) of a given sentence described in [29], though we focus on the usual centralized training setting instead of a federated setting.
We randomly sample 246,092 sentences for training and 61,719 for testing, where each sentence is described by 10,000 features.
We format the task as a 500-class classification problem, and the resulting model has roughly 5 million parameters.
MovieLens-100k [13] is a movie review dataset commonly used for recommendation systems.
It contains 1,000,000 movie ratings from 943 users on 1,682 items ( ≈ 6% non-zero entries).
We study a (non-convex) matrix factorization task with embedding size 100, thus totaling 262,500 parameters.
We treat each non-zero entry as a ‘record’ for differential privacy, and randomly partition them for training and evaluation.
D.2. Implementation
Our experiments are implemented in JAX [5] with Haiku [14] to auto-vectorize over the per-example operations (e.g. per-example clipping) for substantial speedups [28].
Unless explicitly stated, we report results with the best grid-searched hyperparameters, which include the learning rate, perexample clipping threshold, adaptivity ϵ for preconditioned methods, and the delay parameter s .
Note that for DP 2 we tune the learning rates and clipping thresholds separately for private SGD iterations and private adaptive (RMSProp) iterations.
For all datasets, we explore the effects of several noise multiplier ( σ ) values, and set δ = 10 − k where k is the smallest integer that satisfies 10 − k ≤ 1 /n for the training dataset size n .
D.3. Hyperparameters
Unless otherwise stated, we fix the following hyperparameters in our experiments: for IMDB,
StackOverflow, and MovieLens respectively, we train for 100/50/50 epochs with batch size 64 and privacy δ = 10 − 5 / 10 − 6 / 10 − 6 .
We then perform a grid search on other hyperparameters:
• Learning rates : We grid search over { 0.03, 0.1, 0.3, 1, 3, 5 } for SGD / AdaGrad update rules and from { 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3 } for the RMSProp update rule.
• Per-example clipping thresholds : We grid search over { 0.1, 0.25, 0.5, 1 } when performing perexample clipping on clean gradients without preconditioning (e.g. for DP-SGD updates), and over { 0.5, 1, 2, 3, 5 } when clipping preconditioned clean gradients (e.g. for DP 2 updates in adaptive iterations).
The rationale is that, in general, the preconditioned gradient norms are usually larger than those without preconditioning (recall from Section 3.2 that we apply preconditioning before privatization in DP 2 ).
• Delay parameter s :
For all datasets, s (i.e., the number of optimization steps) is chosen heuristically as a function of the number of steps in an epoch.
When reporting the best results (e.g. Figure 6, Figure 4), we search over s ∈ { 195, 390, 780 } (roughly 0.5, 1, 2 epochs respectively) for IMDB
(390 steps/epoch); s ∈ { 100, 300, 1000, 3000 } for StackOverflow (3845 steps/epoch); and s ∈ { 1250, 15625, 31250, 50000 } for MovieLens (1250 steps/epoch).
• Adaptivity ϵ :
In our settings, the adaptivity parameter ϵ for RMSProp/AdaGrad (in the denominator D t = √ v + ϵ ) would affect the amount of adaptivity as well as the norms of preconditioned gradients, which may in turn influence the privacy-utility trade-off under per-example clipping.
We tune ϵ over a small grid of { 10 − 3 , 10 − 5 , 10 − 7 } .
All reported results use the best hyperparameter configurations, which are selected using training set metrics (as overfitting generally does not occur under strong DP noise).
We summarize the tuned hyperparameters in Table 2 below to facilitate reproducibility
[Table]
DPMD, the values refer to (learning rate, DP clipping threshold); for
DP-RMSProp and AdaDPS, the values refer to (learning rate, DP clipping threshold, adaptivity ϵ ); and for DP 2 , the values refer to (LR for SGD iters, LR for RMSProp iters, clip for SGD iters, clip for RMSProp iters, adaptivity ϵ , delay s ).
D.4. Training curves for specific ε values
Figure 6 compares DP 2 -RMSProp with DP-SGD and DP-RMSProp.
We observe that across all datasets, DP 2 consistently and substantially outperforms the baselines in terms of both convergence and absolute performance.
[Figure]
D.5. Results for DP 2 -AdaGrad
The DP 2 framework can be applied to a range of adaptive methods beyond RMSProp mostly discussed in the main text.
We extend DP 2 to the AdaGrad update rule (with only one line of code change, see
Section E)
[Figure]
and benchmark its convergence and privacy-utility trade-offs.
In Figure 7 and Figure 8, the results indicate that DP 2 -AdaGrad, like DP 2 -RMSProp, can consistently and substantially improve over the baselines in terms of both convergence and absolution performance, demonstrating the generality of DP 2 to other adaptive optimizers
[Figure]
[Figure]
D.6. Effects of Increasing Computational Budgets
When differential privacy introduces a large utility gap between private and non-private training, one approach to improving the privacy-utility trade-off is to increase computational costs by using larger
[Figure]
.t
e s t a cc
IMDB ( ε = 3 . 04 , δ = 10 − 5 )
[Figure]
( ε =
[Figure] [Figure]
δ =
[Figure] [Figure]
batch sizes under fixed numbers of steps.
The noise multiplier needs to increase to achieve the same privacy target, while the overall privacy noise may still be reduced due to the larger batch size.
This technique may be adopted in practice when we want to prioritize the utility of private optimization under fixed privacy budgets.
In Figure 8 (right), we explore the effect of such increased computation on StackOverflow.
With a 4x factor increase in computational cost (4x larger batch sizes with the same number of training iterations), we observe that the privacy/utility trade-off of all methods can be substantially improved, narrowing the utility gap to non-private training.
In particular, observe that the absolute performance improvement of DP 2 over the vanilla DP baselines remains similar.
D.7. Additional Results for Ablation Studies
Recall that in Algorithm 1, we use noisy gradients from DP-SGD iterations to update both the model parameters and the preconditioner such that the total privacy cost is identical to that of DP-SGD.
The first variant considers accumulating DP-SGD gradients in the same way, but it runs private adaptive methods using delayed preconditioner in almost all iterations.
This requires us to add independent noise twice at most iterations (when accumulating the preconditioner and when noising the preconditioned update), thus increasing the total privacy budget.
The second variant is identical to DP 2 except that it applies the delayed preconditioner after noising the clean gradient; this is to study the order of preconditioning as discussed in Section 3.
As illustrated in Figure 9, both variants indeed significantly underperform our proposed method, thus validating the design choices of DP 2 .
In particular, note that the utility drop of variant 2 (adding noise before preconditioning) on StackOverflow is more significant compared to that on IMDB; we argue that this is due to
StackOverflow being a high-dimensional learning task (roughly 5 million model parameters) and thus the detrimental effect of preconditioning per-coordinate noise is larger.
We defer complete results Table 3.
See also Appendix E for the exact algorithms of both variants.
D.8. Additional Results for Comparison with Public Data-Assisted Methods
Figure 10 extends the results in Section 5.2 with convergence plots on IMDB and StackOverflow.
On IMDB, we observe that despite not using any auxiliary information, the convergence of DP 2RMSProp is comparable with that of AdaDPS-RMSProp [19] which uses 1% of training data as
[Figure]
[Table]
[Figure]
the public data (250 examples) to approximate the preconditioner.
On StackOverflow where the same public split of 1% corresponds to 2460 examples, we observe that AdaDPS-RMSProp can outperform DP 2 .
On the other hand, the extra public data do not help PDA-DPMD outperform DP 2 .
[Figure]
ε =
[Figure]
δ
[Figure]
In Figure 11, we additionally implement a private AdaGrad method proposed in [17] that also leverages public data.
Specifically, in each iteration, the algorithm clips and adds independent noise to both the clean gradients and the preconditioner estimated using clean gradients; it then uses public data to estimate a gradient subspace onto which to project the clipped/noised preconditioner in order to reduce the effect of noise; finally, it preconditions the noisy gradient with the noisy preconditioner and takes an update step.
Our implementation differs from [17] in that we use the diagonal form of the preconditioner instead of the full matrix form.
To estimate the gradient subspace, we follow the approach described in [33] where the projection matrix
V ∈ R d × k where d is the number of parameters and k is the dimension of the subspace is obtained by taking the top- k eigenspace of M t with
[Equation]
where X pub is the set of public examples.
Unfortunately, we have not obtained a satisfactory result for this noisy AdaGrad algorithm.
We remark that since the method is extremely computationally expensive (involves computing the eigendecomposition of a d × d matrix with d = 10001 at every iteration), further hyperparameter tuning may help improve the performance.
However, our ablation studies (Section 5.3 and Appendix D.7) may shed light on the current observations since this method privatizes gradients before preconditioning.
Appendix E. Algorithms
For completeness, we present all algorithms mentioned in the main text in detail.
• Non-private version of DP 2 : only changing Line 10 in Algorithm 1 to
[Equation]
• DP 2 with the AdaGrad update rule ( DP 2 -AdaGrad) : only changing Line 6 in Algorithm 1 to
[Equation]
• DP 2 with Yogi’s additive update rule ( DP 2 -Yogi) : only changing Line 6 in Algorithm 1 to
[Equation]
• Ablation variant 1 (extra query) with delayed preconditioners : see Algorithm 2.
Observe that the clean batch gradients { g i,t } i ∈ B get privatized twice in most iterations (when ( t − 1) mod s ̸ = 0 ), increasing the total privacy cost.
• Ablation variant 2 (noise before preconditioning) with delayed preconditioners : in Line 10 of Figure 1, privatize the batch gradients with the following replacement:
[Table]
[Equation]
Input: T , batch size b , noise multiplier σ , clipping thresholds C 1 , C 2 , initial model w 0 ∈ R d , v = 0 , constant ϵ ∈ R + , learning rate schedule α t , moving average parameters β , delay steps s
15 Set accumulator G 0 ←16 for t = 1 , · · · , T do
Uniformly randomly sample a mini-batch B with size b from private training data Get individual gradients for sample i ∈ B : g i,t ← ∇ f ( x i ; w t − 1 ) Privatize the gradients using the Gaussian mechanism:
˜ g t ← 1 b (cid:32)(cid:88) i ∈ B clip (cid:0) g
i,t , C 1 (cid:1) + N (cid:0) 0 , σ 2 C 21 (cid:1)(cid:33)
Accumulate the private gradients ˜
g t : G t ← G t − 1 + ˜ g t if ( t − 1) mod s = 0 then
Update moment estimates: v ← βv + (1 − β ) (cid:0) G t /s (cid:1) 2 Reset accumulator: G t ← 0 Set final gradient : ¯ g t ← ˜ g t end else
Privatize the clean, preconditioned gradients using the Gaussian mechanism: ˆ g t ← 1 b (cid:32)(cid:88) i ∈ B clip (cid:18) g
i,t √ v + ϵ, C(cid:19)
+ N (cid:0) 0 , σ 2 C 22 (cid:1)(cid:33)
Set final gradient : ¯ g t ← ˆ g t end
Update model parameters w : w t ← w t − 1 − α t ¯ g t 25 end 26 return w T