Private Stochastic Optimization With Large Uniform Lipschitz Parameter: Optimal Rates for (Non-Smooth) Convex Losses & Extension to Non-Convex
author names withheld
Under Review for OPT
Abstract
We study differentially private (DP) stochastic optimization (SO) with loss functions whose worstcase Lipschitz parameter over all data points may be extremely large.
To date, the vast majority of work on DP SO assumes that the loss is uniformly Lipschitz continuous over data (i.e. stochastic gradients are uniformly bounded over all data points).
While this assumption is convenient, it often leads to pessimistic excess risk bounds. In many practical problems, the worst-case (uniform)
Lipschitz parameter of the loss over all data points may be extremely large due to outliers.
In such cases, the error bounds for DP SO, which scale with the worst-case Lipschitz parameter of the loss, are vacuous.
To address these limitations, this work provides near-optimal excess risk bounds that do not depend on the uniform Lipschitz parameter of the loss.
Building on a recent line of work [36, 62], we assume that stochastic gradients have bounded k -th order moments for some k ě 2 .
Compared with works on uniformly Lipschitz DP SO, our excess risk scales with the k -th moment bound instead of the uniform Lipschitz parameter of the loss, allowing for significantly faster rates in the presence of outliers and/or heavy-tailed data.
For convex and strongly convex loss functions, we provide the first asymptotically optimal excess risk bounds (up to a logarithmic factor).
In contrast to [36, 62], our bounds do not require the loss function to be differentiable/smooth.
We also devise an accelerated algorithm for smooth losses that runs in linear time and has excess risk that is tight in certain practical parameter regimes.
Additionally, our work is the first to address non-convex non-uniformly Lipschitz loss functions satisfying the Proximal-PL inequality ; this covers some practical machine learning models.
Our Proximal-PL algorithm has near-optimal excess risk.
1. Introduction
As the use of machine learning (ML) models in industry and society has grown dramatically in recent years, so too have concerns about the privacy of personal data that is used in training such models.
It is well-documented that ML models may leak training data, e.g., via model inversion attacks and membership-inference attacks [17, 28, 30, 49, 53].
Differential privacy (DP) [27] ensures that data cannot be leaked, and a plethora of work has been devoted to differentially private machine learning and optimization [7, 9, 10, 18, 21, 25, 29, 44, 57, 60].
Of particular importance is the fundamental problem of DP stochastic (convex) optimization (S(C)O): given n i.i.d.
samples
X “ p x 1 , . . .
, x n q P X n from an unknown distribution D , we aim to privately solve
[Equation]
where f : W ˆ X Ñ R is the loss function and W Ă R d is the parameter domain.
Since finding the exact solution to (1) is not generally possible, we measure the quality of the obtained solution © .
via excess risk (a.k.a.
excess population loss): The excess risk of a (randomized) algorithm A for solving (1) is defined as E F p
A p X qq ´ min w P W F p w q , where the expectation is taken over both the random draw of the data X and the algorithm A .
A large body of literature is devoted to characterizing the optimal achievable differentially private excess risk of (1) when the function f p¨ , x q is uniformly L f - Lipschitz for all x P X —see e.g., [7, 10, 13, 29, 44].
In these works, the gradient of f is assumed to be uniformly bounded with sup w P W ,x P X } ∇ w f p w, x q} ď L f , and excess risk bounds scale with L f .
While this assumption is convenient for bounding the sensitivity [27] of the steps of the algorithm, it is often unrealistic in practice or leads to pessimistic excess risk bounds.
In many practical applications, data contains outliers, is unbounded or heavy-tailed (see e.g. [22, 46, 63] and references therein for such applications).
Consequently, L f may be prohibitively large.
For example, even the linear regression loss f p w, x q “ 12 px w, x p 1 q y ´ x
p 2 q q 2 with compact W and data from X “ X p 1 q ˆ X p 2 q , leads to L f ě diameter p X p 1 q q 2 , which could be huge.
Similar observations can be made for other useful ML models such as deep neural nets [42], and the situation becomes even grimmer in the presence of heavy-tailed data.
In these cases, existing excess risk bounds, which scale with L f , becomes vacuous.
While L f can be very large in practice (due to outliers), the k -th moment of the stochastic gradients is often reasonably small for some k ě 2 (see, e.g., Example 1).
This is because the k -th moment r r k : [Equation]
[Equation]
{ k depends on the average behavior of the stochastic gradients, while L f depends on the worst-case behavior over all data points.
Motivated by this observation and building on the prior results [36, 62], this work characterizes the optimal differentially private excess risk bounds for the class of problems with a given parameter r r k .
Specifically, for the class of problems with parameter r r k , we answer the following questions:
• Question I: What are the minimax optimal rates for (strongly) convex DP SO?
• Question II: What utility guarantees are achievable for non-convex DP SO?
Prior works have made progress in addressing the first question above: 1 The work of [62] provided the first excess risk upper bounds for smooth DP (strongly) convex SO.
[36] gave improved, yet suboptimal, upper bounds for smooth (strongly) convex f p¨ , x q , and lower bounds for (strongly) convex SO.
In this work, we provide optimal algorithms for convex and strongly convex losses, resolving Question I up to logarithmic factors.
Our bounds hold even for non-differentiable/nonsmooth F .
Regarding Question II, we give the first algorithm for DP SO with non-convex loss functions satisfying the Proximal-Polyak-Łojasiewicz condition [37, 50].
We provide a summary of our results for the case k “ 2 in Figure 1, and a thorough discussion of related work in Appendix B.
1.1. Preliminaries
Let }¨} be the ℓ 2 norm.
Let W be a convex, compact set of ℓ 2 diameter D .
Function g : W Ñ R is µstrongly convex if g p αw `p 1 ´ α q w 1 q ď αg p w q`p 1 ´ α q g p w 1 q´ α p 1 ´ α q µ} w ´ w 1 } 2 for all α P r 0 , 1 s and all w, w 1 P W .
If µ “ 0 , we say g is convex.
For convex f p¨ , x q , denote any subgradient of f p w, x q w.r.t.
w by ∇ f p w, x q P B w f p w, x q : i.e. f p w 1 , x q ě f p w, x q ` x ∇ f p w, x q , w 1 ´ w y for all w 1 P W .
Function g is β -smooth if it is differentiable and its derivative ∇ g is β -Lipschitz.
For β -smooth, µ -strongly convex g , denote its condition number by κ “ β { µ .
For functions a and b of
[Figure]
input parameters, write a À b if there is an absolute constant A such that a ď Ab for all feasible values of input parameters.
Write a “ r O p b q if a À ℓb for a logarithmic function ℓ of input parameters.
We assume that the stochastic gradient distributions have bounded k -th moment for some k ě 2 :
Assumption 1 There exists k ě 2 and r r p k q ą 0 such that E “ sup w P W } ∇ f p w, x q} k‰ ď r r p k q for all ∇ f p w, x i q P B w f p w, x i q .
Denote r r k : “ p r r p k q q 1 { k .
Clearly, r r ď L f “ sup t ∇ f p w,x qPB w f p w,x qu sup w,x } ∇ f p w, x q} , but this inequality is often very loose:
Example 1 For linear regression on a unit ball W with 1 -dimensional data x p 1 q , x p 2 q P r´ 10 6 , 10 6 s having truncated Normal distributions and Var p x p 1 q q “ Var p x p 2 q q ď 1 , we have L f ě 10 12 .
On the other hand, r r k is much smaller than L f for small to moderate k : e.g., r r 2 ď 5 , r r 4 ď 8 , and r r 8 ď 14 .
For finite k , Assumption 1 is weaker than assuming that f p¨ , x q is uniformly Lipschitz for all x since it allows for the p -th moments of sup w P W } ∇ f p w, x q} to be unbounded for 8 ě p ą k .
Differential Privacy [27] ensures that no adversary—even one with enormous resources—can infer much more about any person who contributes training data than if that person’s data were absent.
Definition 1 (Zero-Concentrated Differential Privacy (zCDP) [16])
A randomized algorithm A :
X n Ñ W satisfies ρ -zero-concentrated differential privacy ( ρ -zCDP) if for all X, X 1 P X n differing in a single entry and all α P p 1 , 8q , we have D α p
A p X
q|| A p X 1 qq ď ρα , where D α p
A p X
q|| A p X 1 qq is the α -Rényi divergence 2 between the distributions of A p X q and A p X 1 q .
zCDP lives between pure DP and approximate DP [16]: see Appendix D for the precise statement.
2. Optimal Rates for Non-Smooth (Strongly) Convex Losses (see Appendix F)
We establish asymptotically optimal (up to logarithms) excess risk bounds for DP SCO under Assumption 1, without requiring differentiability of f p¨ , x q :
Theorem 2 (Informal, see Theorem 10, Theorem 18, Theorem 21, Theorem 23) Let f p¨ , x q be convex.
Grant Assumption 1.
Then, there is a polynomial-time ϵ 2 {+
+} -zCDP algorithm
A such that E F p
A p X qq ´
F ˚ “ r O ˆ r r 2 k D ˆ 1 ?
n ` ´ ?
d ϵn ¯ p k ´ 1 q{ k ˙˙ .
If f p¨ , x q is µ -strongly convex, then E F p
A p X qq ´
F ˚ “ r O ˆ r r 22 k µ ˆ 1 n ` ´ ?
d ϵn ¯ p 2 k ´ 2 q{ k ˙˙ .
Further, these bounds are minimax optimal up to a factor of r O p r r 22 k { r r 2 k q .
The works [36, 62] make a slightly different assumption than Assumption 1: they instead assume that the k -th order central moment of each coordinate ∇ j f p w, x q is bounded by γ 1 { k for all j P r d s , w P W .
We also provide asymptotically optimal excess risk bounds for the class of problems satisfying the coordinate-wise moment assumption of [36, 62] and having subexponential stochastic subgradients: see Appendix F.3.
The previous state-of-the-art convex upper bound was suboptimal: O ´r rD b dn ¯ for ϵ « 1 [36,
Theorem 5.4].
Their result also required f p¨ , x q to be β f -smooth for all x P X , which can be restrictive with outlier data: e.g. this implies that f p¨ , x q is L f -Lipschitz with L f ď
β f D if ∇ f p w ˚ p x q , x q “ 0 for some w ˚ p x q P W .
Our µ -strongly convex bound also improves over the best previous upper bound of [36, Theorem 5.6], which again required β f -smoothness of f p¨ , x q .
In fact, [36, Theorem 5.6] was incorrect as stated in the ICML 2022 version of their paper, as we explain in Appendix C. 4 However, after communicating with the authors of [36], they updated their result and proof in the arXiv version of their paper.
The corrected version of [36, Theorem 5.6]—which we derive in Appendix C for completeness—is suboptimal by a factor of r O pp β f { µ q 3 .
In practical applications, the condition number β f { µ can be very large, especially in the presence of outliers or heavy-tailed data.
Our near-optimal excess risk bounds remove this dependence on β f { µ .
Our Algorithm 3 combines the iterative localization technique of [7, 29] with a clipped noisy subgradient method 5 run on a regularized empirical objective.
In each iteration, we use a geometrically decreasing step size, which shrinks the effective variance of the privacy noise over time.
We also geometrically decrease the number of samples used, and we increase the regularization parameter over time as our iterates get closer to the optimum to improve the stability [14, 38, 41] of our algorithm.
With clipped (hence biased ) stochastic gradients and non-smooth/non-uniformly
Lipschitz loss, the excess risk analysis of our algorithm is harder than in the smooth and uniformly
Lipschitz settings.
Instead of the uniform convergence analysis used in [36, 62], we derive new results about the stability and generalization error of (regularized) ERM with non-smooth, non-Lipschitz losses.
In a bit more detail, our upper bound proof consists of the following steps: i)
We bound the excess empirical risk of our noisy clipped subgradient subroutine (Lemma 12).
ii)
We bound the on-average model stability [41] of our algorithm (Proposition 16).
On-average model stability is a weaker notion than uniform stability [10, 14]; this is necessary to obtain our learnability guarantees for non-Lipschitz f p¨ , x q . iii)
We prove that on-average model stable algorithms generalize well (Proposition 14), strengthening and extending results from [41, 52] to non-smooth/non-Lipschitz f p¨ , x q .
We obtain our strongly convex bound (Theorem 18) by a reduction to the convex case, ala [29, 32]. by r r ?
d . 4. In short, the mistake is that Jensen’s inequality is used in the wrong direction to claim that the T -th iterate of their algorithm w T satisfies E r} w T ´ w ˚ } 2 s ď p E } w T ´ w ˚ }q 2 , which is false.
Clipped noisy subgradient method (Algorithm 2) works as follows:
In each round t , we: draw a random minibatch of samples; compute and then clip (i.e. project onto an ℓ 2 ball centered around 0 ) their stochastic subgradients; average the clipped stochastic subgradients and add noise; and then take a noisy minibatch stochastic subgradient step.
We also refine (to describe the dependence on r r k D, µ ), extend (to k " 1 ), and tighten (for µ “ 0 ) the lower bounds of [36]: see Theorems 21 and 23.
3. Linear Time Algorithms for Smooth (Strongly) Convex Losses (see Appendix G)
For convex, β -smooth F , we provide a novel accelerated DP algorithm (Algorithm 4), building on the work of Ghadimi and Lan [31].
Our algorithm is linear time and attains excess risk that improves over the previous state-of-the-art ( not linear time ) algorithm
[36, Theorem 5.4] in practical parameter regimes (e.g. d Á n 1 { 6 ).
The excess risk of our algorithm is tight in certain cases: e.g., d Á p ϵn q 2 { 3 or “sufficiently smooth” F (see Remark 27).
To prove our upper bound, we give the first analysis of accelerated SGD with biased stochastic gradients (Proposition 25).
For µ -strongly convex, β -smooth losses, acceleration results in excessive bias accumulation, so we propose a simple noisy clipped SGD.
Our algorithm builds on [36], but uses a lower-bias clipping mechanism from [8] and a tighter analysis.
We attain excess risk that is near-optimal up to a r O pp β { µ q p k ´ 1 q{ k q factor: see Theorem 30.
Our linear-time bound strictly improves over the best previous bound of [36, Theorem 5.6].
Our analysis leverages a novel bound (Proposition 31) for
SGD with biased stochastic gradients; this bound is tighter than existing results of [3, 6, 36], which is needed to obtain near-optimal excess risk.
4. First Algorithm for Non-Convex (Proximal-PL) Losses (see Appendix H)
We consider losses satisfying the Proximal Polyak-Łojasiewicz (PPL) inequality [37, 50] (Definition 34), an extension of the classical PL inequality to the proximal setting.
Many practical ML models satisfy the PPL condition, such as some classes of neural nets, linear/logistic regression, and
LASSO [37, 42].
We propose a DP proximal clipped SGD to attain near-optimal excess risk that almost matches the strongly convex rate: see Theorem 35.
The proof of this result is difficult because it is unclear how to separate the privacy noise from the non-private terms in the proximal/non-convex setting.
We prove Proposition 36 by building on [45], which analyzed uniformly Lipschitz PPL losses with unbiased stochastic gradients using techniques from the analysis of objective perturbation [19].
5. Concluding Remarks and Open Questions
This paper was motivated by practical problems in which data contains outliers and potentially heavy tails, causing the worst-case Lipschitz parameter of the loss over all data points to be prohibitively large.
In such cases, existing bounds for DP SO that scale with the worst-case Lipschitz parameter become vacuous.
Thus, we operated under the more relaxed assumption of stochastic gradient distributions having bounded k -th order moments.
The k -th moment bound is often much smaller than the worst-case Lipschitz parameter in practice.
For (strongly) convex loss functions, we established the asymptotically optimal rates (up to logarithms), even with non-differentiable losses.
We also provided linear time algorithms for smooth losses that are optimal in certain practical parameter regimes, but suboptimal in general.
An interesting open question is: does there exist a linear time algorithm with optimal excess risk?
We also initiated the study of non-convex nonuniformly Lipschitz DP SO, showing that the optimal strongly convex rates can nearly be attained without convexity, via the proximal-PL condition.
We leave the treatment of general non-convex losses for future work.
Appendix Appendix A. Other Bounded Moment Conditions Besides Assumption
In this section, we give the alternate bounded moment assumption made in [36, 62] and a third bounded moment condition, and discuss the relationships between these assumptions.
The notation presented here will be necessary in order to state the sharper versions of our linear-time excess risk bounds and the asymptotically optimal excess risk bounds under the coordinate-wise assumption of [36, 62] (which our Algorithm 3 also attains).
First, we introduce a relaxation of Assumption 1:
Assumption 2 There exists k ě 2 and r p k q ą 0 such that sup w P W E “ } ∇ f p w, x q} k‰ ď r p k q , @ ∇ f p w, x i q P B w f p w, x i q .
Denote r : “ p r p k q q 1 { k .
Assumption 1 implies Assumption 2 for r ď r r .
Next, we precisely state the coordinate-wise moment bound assumption that is used in [36, 62] for differentiable f :
Assumption 3 (Used by [36, 62] 7 , but not in this work)
There exists k ě 2 and γ ą 0 such that sup w P W E |x ∇ f p w, x q ´ ∇ F p w q , e j y| k ď γ , for all j P r d s , where e j denotes the j -th standard basis vector in R d .
Also, L fi sup w P W } ∇ F p w q} ď ?
dγ 1 { k .
Lemma 3 allows us compare our results in Section 3 obtained under Assumption 2 to the results in [36, 62], which require Assumption 3.
Lemma 3 Suppose Assumption 3 holds.
Then, Assumption 2 holds for r ď 4 ?
dγ 1 { k .
Since Assumption 2 is implied by Assumption 3, the upper bounds that we obtain under Assumption 2 also hold (up to constants) if we grant Assumption 3 instead, with r Ø ?
dγ 1 { k .
Also, in Appendix F.3, we will use Lemma 3 to show that our optimal excess risk bounds under Assumption 1 imply asymptotically optimal excess risk bounds under Assumption
[Equation]
Proof [Proof of Lemma 3]
We use the following inequality, which can easily be verified inductively, using Cauchy-Schwartz and Young’s inequalities: for any vectors u
[Equation]
v P R d , we have
[Equation]
Therefore,
[Equation]
where we used convexity of the function ϕ
p y q “ y k { 2 for all y ě 0 , k ě 2 and Jensen’s inequality in the last inequality.
Now using linearity of expectation and Assumption 3 gives us r k ď 2 k ´
L k ` d k { 2 γ ¯ ď 2 k ` 1 d k { 2 γ, since L k ď d k { 2 γ by hypothesis.
Appendix B. Additional Discussion of Related Work
DP SCO Without Uniform Lipschitz Continuity: The study of DP SCO without uniformly
Lipschitz continuous loss functions was initiated by [62], who provided upper bounds for smooth convex/strongly convex loss.
The work of [36] provided lower bounds and improved, yet suboptimal , upper bounds for the convex case.
Both of the works [36, 62] require f to be β f -smooth.
It is also worth mentioning that [36, 62] restricted attention to losses satisfying ∇ F p w ˚ q “ 0 for w ˚ P W , i.e. W is a compact set containing the unconstrained optimum w ˚ “ argmin w P R d F p w q P W .
By comparison, we consider the more general constrained optimization problem min w P W F p w q , where
W need not contain the global unconstrained optimum.
Here we provide a brief discussion of the techniques used in [36, 62].
The work of [62] used a full batch (clipped, noisy) gradient descent based algorithm, building on the heavy-tailed mean estimator of [33].
They bounded the excess risk of their algorithm by using a uniform convergence [58] argument, resulting in a suboptimal dependence on the dimension d .
The work of [36] used essentially the same approach as [62], but obtained an improved rate with a more careful analysis.
However, as discussed, the bound in [36] is O ´ rD
b dn ¯ when ϵ « 1 , which is still suboptimal.
More recently, DP optimization with outliers was studied in special cases of sparse learning [34], multi-arm bandits [56], and ℓ 1 -norm linear regression [61].
DP ERM and DP GLMs without Uniform Lipschitz continuity: The work of [6] provides bounds for constrained DP ERM with arbitrary convex loss functions using a Noisy Clipped SGD algorithm that is similar to our Algorithm 5, except that their algorithm is multi-pass and ours is one pass.
In a concurrent work, [23] considered DP ERM in the unconstrained setting with convex and non-convex loss functions.
Their algorithm, noisy clipped SGD, is also similar to Algorithm 5 and the algorithm of [6].
The results in [23] are not directly comparable to [6] since [23] consider the unconstrained setting while [6] consider the constrained setting, but the rates in [6] are faster.
[23] also analyzes the convergence of noisy clipped SGD with smooth non-convex loss functions.
The works of [5, 54] consider generalized linear models (GLMs) , a particular subclass of convex loss functions and provide empirical and population risk bounds for the unconstrained DP optimization problem.
The unconstrained setting is not comparable to the constrained setting that we consider here: in the unconstrained case, a dimension-independent upper bound is achievable, whereas our lower bounds (which apply to GLMs) imply that a dependence on the dimension d is necessary in the constrained case.
Other works on gradient clipping:
The gradient clipping technique (and adaptive variants of it) has been studied empirically in works such as [1, 4, 20], to name a few.
The work of [20] shows that gradient clipping can prevent SGD from converging, and describes the clipping bias with a disparity measure between the gradient distribution and a geometrically symmetric distribution.
Optimization with biased gradient oracles: The works [3, 6] analyze SGD with biased gradient oracles.
Our work provides a tighter bound for smooth, strongly convex functions and analyzes accelerated SGD and proximal SGD with biased gradient oracles.
DP SO with Uniformly Lipschitz loss functions: In the absence of outlier data, there are a multitude of works studying Lipschitz DP SO, mostly in the convex/strongly convex case.
We do not attempt to provide a comprehensive list of these here, but will name the most notable ones, which provide optimal or state-of-the-art utility guarantees.
The first suboptimal bounds for DP SCO were provided in [9].
The work of [10] established the optimal rate for non-strongly convex DP SCO, by bounding the uniform stability of Noisy DP SGD (without clipping).
The strongly convex case was addressed by [29], who also provided optimal rates in linear times for sufficiently smooth, convex losses.
Since then, other works have provided faster and simpler (optimal) algorithms for the non-smooth DP SCO problem [7, 11, 12, 40] and considered DP SCO with different geometries [7, 13].
State-of-the-art rates for DP SO with the proximal PL condition are due to [45].
Appendix C. Correcting the Errors in the Strongly Convex Upper Bounds Claimed in [36, 62]
While [36, Theorem 5.6] claims an upper bound for smooth strongly convex losses that is tight up to a factor of r O p κ 2 f q , we identify an issue with their proof that invalidates their result.
A similar issue appears in the proof of [62, Theorems 5 and 7], which [36] built upon.
We then show how to salvage a correct upper bound within the framework of [36], albeit at the cost of an additional factor of κ f .
The proof of [36, Theorem 5.6] relies on [36, Theorem 3.2].
The proof of [36, Theorem 3.2], in turn, bounds E } w T ´ w ˚ } ď p λ ` L qp M ` 1 q G λL in the notation of [36], where L is the smoothness parameter, λ is the strong convexity parameter (so L ě λ ), and M is the diameter of W .
Then, it is incorrectly deduced that E r} w T ´ w ˚ } 2 s ď ´ p λ ` L qp M ` 1 q G λL ¯(final line of the proof).
Notice that
E r} w T ´ w ˚ } 2 s can be much larger than p E } w T ´ w ˚ }q 2 in general: for example, if } w T ´ w ˚ } has the Pareto distribution with shape parameter α P p 1 , 2 s and scale parameter 1 , then p E } w T ´ w ˚ }q 2 “ ´ αα ´¯ [Equation]
E p} w [Equation] ´ w ˚ } 2 q “ 8 .
To try to correct this issue, one could use Young’s inequality to instead bound
[Equation]
but the geometric series above diverges to `8 as T Ñ 8 , since
[Equation]
ěðñ p λ ´ L q 2 ě 0 .
Evidently, there is no “easy fix” for the issue in the proofs of [36, Theorem 3.2] and
[62, Theorem 5] (at least without imposing severe restrictions on λ and L and hence dramatically shrinking the function class).
Next, we show how to modify the proof of [36, Theorem 5.6] in order to obtain a correct excess risk upper bound of
[Equation]
(in our notation).
This correction was derived in collaboration with the authors of [36], who have also updated the arXiv version of their paper accordingly.
By waiting until the very of the proof of [36, Theorem 3.2] to take expectation, we can derive
[Equation]
for all t , where we use their L “ β f and λ “ µ notation but our notation F and r ∇ F for the population loss and its biased noisy gradient estimate (instead of their L D notation).
By iterating (4), we can get
[Equation]
Squaring both sides and using Cauchy-Schwartz, we get
[Equation]
Using L -smoothness of F and the assumption made in [36] that ∇ F p w ˚ q “ 0 , and then taking expectation yields
[Equation]
where G 2 ě E ”
[Equation]
p w [Equation] q ´ ∇ F p
[Equation]
q}ı for all t .
It is necessary and sufficient to choose
T “ r Ω p L { λ q to make the first term in (5) less than the second term (up to logarithms).
With this choice of T , we get
[Equation]
where κ f “ L { λ .
Next, we apply the bound on G 2 for the MeanOracle that is used in [36]; this bound is stated in the version of [36, Lemma B.5] that appears in the updated (November 1, 2022) arXiv version of their paper.
The bound (for general γ ) is G 2 “ r O
[Equation]
Plugging this bound on G 2 into (6) yields (3).
Appendix D. More Preliminaries
Remark on Terminology: Following [36, 62], we sometimes use the terminology “heavy-tailed” to refer to distributions satisfying Assumption 1, 2, or 3.
However, as discussed, our results are useful for a broader class of distributions than those typically considered “heavy-tailed”: we are also interested in bounded data that contains outliers, or data that is unbounded but light-tailed (e.g.
Gaussian), since we may have r r !
L f in such cases.
Notation: For functions a and b of input parameters, write a À b if there is an absolute constant A such that a À Ab for all feasible values of input parameters.
More Differential Privacy Preliminaries: Recall the classical definition of differential privacy [27]:
Definition 4 (Differential Privacy) Let ϵ ě 0 , δ P r 0 , 1 q .
A randomized algorithm A : X n Ñ
W is p ϵ, δ q -differentially private (DP) if for all pairs of adjacent data sets X, X 1 P X n (i.e. d hamming p X, X 1 q “ 1 ) and all measurable subsets S Ď W , we have P p
A p X q P S q ď e ϵ P p
A p X 1 q P
S q ` δ .
If δ “ 0 , we say A satisfies pure DP .
If δ ą 0 , then A satisfies approximate DP .
zCDP is weaker than pure DP, but stronger than approximate DP in the following sense:
Proposition 5 [16, Proposition 1.3]
If A is ρ -zCDP, then A is p ρ `a ρ log p 1 { δ q , δ q for any δ ą 0 .
Thus, if ϵ ď a log p 1 { δ q , then any ϵ 2 2 -zCDP algorithm is p 2 ϵ a log p 1 { δ q , δ q -DP.
Our algorithms use the Gaussian mechanism to achieve zCDP:
Proposition 6 [16, Proposition 1.6] Let q : X n Ñ R be a query with ℓ 2 -sensitivity ∆ : “ sup X „ X 1 } q p X q ´ q p X 1 q} .
Then the Gaussian mechanism, defined by M : X n Ñ R , M p X q : “ q p X q ` u for u „ N p 0 , σ 2 q , is ρ -zCDP if σ 2 ě ∆ 2 2 ρ .
The (adaptive) composition of zCDP
algorithms is zCDP, with privacy parameters adding:
Lemma 7 [16, Lemma 2.3] Suppose A : X n Ñ Y satisfies ρ
-zCDP
and A 1 : X n ˆ Y Ñ Z satisfies ρ 1 -zCDP (as a function of its first argument).
Define the composition of A and A 1 , A 2 : X n Ñ Z by A 2 p X q “ A 1 p X, A p X qq .
Then A 2 satisfies p ρ ` ρ 1 q -zCDP.
In particular, the composition of T
ρ -zCDP mechanisms is a T
ρ -zCDP mechanism.
Appendix E. Private Heavy-Tailed Mean Estimation Building Blocks
In each iteration of our SO algorithms, we need a way to privately estimate the mean ∇ F p w t q “
E x „ D r ∇ f p w t , x qs .
If f p¨ , x q is Lipschitz, then one can simply draw a random sample x t from
X and add noise to the stochastic gradient ∇ f p w t , x t q to obtain a DP estimator of ∇ F p w t q : the ℓ 2 -sensitivity of the stochastic gradients is bounded by sup x,x 1 P X } ∇ f p w t , x q ´ ∇ f p w t , x 1 q} ď 2 L f , so the Gaussian mechanism guarantees DP (by Proposition 6).
However, in the setting that we consider, L f (and hence the sensitivity) may be unbounded, so noisy stochastic gradients are not
DP.
Thus, we clip the stochastic gradients (to force the sensitivity to be bounded) before adding noise.
Specifically, we invoke Algorithm 1 on a batch of s stochastic gradients at each iteration of our algorithms.
In Algorithm 1, Π C p z q : “ argmin y P B 2 p 0 ,C q } y ´ z } 2 denotes the projection onto the centered ℓ 2 ball of radius C in R d .
Lemma 8 bounds the bias and variance of Algorithm 1.
Lemma 8 ([8]) Let t x i u si “ 1 „ D s be R d -valued random vectors with E
x i “ ν and E } x i } k ď r p k q for some k ě 2 .
Denote the noiseless average of clipped samples by p ν : “ 1 s ř si “ 1 Π C p x i q .
Then, } E r
ν
´ ν } “ } E p
ν ´ ν } ď E } p ν ´ ν } ď r p k q p k ´ 1 q C k ´ 1 , and E } r
ν ´ E r ν } 2 “ E } r ν ´ E p ν } 2 ď dσ 2 ` r p 2 q s .
Appendix F. Details and Proofs of Results in Section
In this section, we establish the optimal rates (up to logarithms) for the class of DP SCO problems satisfying Assumption 1.
We present our result for convex losses in Section 2, and our result for strongly convex losses in Appendix F.2.
In Appendix F.3, we show how our (tight) upper bounds under Assumption 1 lead to asymptotically tight upper bounds under Assumptions 2 and 3.
In Appendix F.4, we provide lower bounds, which show that our upper bounds are tight.
F.1. Localized Noisy Clipped Subgradient Method for Convex Losses
Our algorithm (Algorithm 3) uses iterative localization [7, 29] with clipping (in Algorithm 2) to handle heavy-tailed data. [Table]
[Table] [Table] η , clip thresh. C , regularization λ ě 0 , constraint set W .
2: Initialize w 0 P W .
3: for t P t 0 , 1 , ¨ ¨ ¨ , T ´ 1 u do 4: r ∇ F t p w t q : “ MeanOracle1 pt ∇ f p w t , x i qu ni “ 1 ; n ; C ; ϵ 2 2 T q for subgradients ∇ f p w t , x i q .
5: w t ` 1 “ Π W ” w t ´ η ´ r ∇ F t p w t q ` λ p w t ´ w 0 q ¯ı 6: end for 7: Output: w T .
The main ideas of Algorithm 3 are:
Clipping only the non-regularized component of the subgradient to control sensitivity and bias: Notice that when we call Algorithm 2 in phase i of Algorithm 3, we only clip the subgradients of f p w t , x j q , not the regularized loss f p w t , x j q ` λ 2 } w t ´ w i ´ 1 } 2 .
Compared to clipping the full gradient of the regularized loss, our selective clipping approach significantly reduces the bias of our subgradient estimator.
This is essential for obtaining our near-optimal excess risk.
Further, this reduction in bias comes at no cost to the variance of our subgradient estimator: the ℓ 2 -sensitivity of our estimator is unaffected by the regularization term. Solve regularized ERM subproblem with a stable DP algorithm : We run a multi-pass zCDP solver on a regularized empirical loss: Multiple passes let us reduce the noise variance in 1: Input: Data X P X n , stepsize η , clip thresh.
t C i u log 2 p n q i “, iteration num.
t T i u log 2 p n q i “, hyperparameter p ě 1 .
2: Initialize w 0 P W .
Let l : “ log 2 p n q .
3: for i P r l s do 4:
Set n i “ 2 ´ i n, η i “ 4 ´ i η , λ i “ 1 η i n pi , T i “ r Θ ´ 1 λ i η i ¯ , and D i “ 2 L f λ i .
Draw new batch B i of n i “ | B i | samples from X without replacement.
Let p F
i
p w q : “ 1 n i ř j P B
i f p w, x j q ` λ i 2 } w ´ w i ´ 1 } 2 .
Use Algorithm 2 with initialization w i ´ 1 to minimize p F i over W i : “ t w P W : } w ´ w i ´ 1 } ď
[-´ 
-]D i u , for T i iterations with clip threshold C i and noise σ 2 i “
4 C
i T i n 2 i ϵ 2 .
Let w i be the output of Algorithm 2.
8: end for 9: Output: w l . phase i by a factor of T i (via strong composition for zCDP) and get a more accurate solution to the ERM subproblem.
Regularization makes the empirical loss strongly convex, which improves on-average model stability and hence generalization of the obtained solution (see Proposition 14 and 15). Localization [6, 29] (i.e. iteratively “zooming in” on a solution):
In early phases (small i ), when we are far away from the optimum w ˚ , we use more samples (larger n i ) and large learning rate η i to make progress quickly.
As i increases, w i is closer to w ˚ , so fewer samples and slower learning rate suffice.
Since step size η i shrinks (geometrically) faster than n i , the effective variance of the privacy noise {+η 2 i σ 
+}
i decreases as i increases.
This prevents w i `from moving too far away from w i (and hence from w ˚ ).
We further enforce this localization behavior by increasing the regularization parameter
λ i and shrinking D i over time. We choose D i as small as possible subject to the constraint that argmin w P W p F i p w q P W i .
This constraint ensures that Algorithm 2 can find w i with small excess risk.
Next, we will provide privacy and excess risk guarantees for Algorithm 3.
In order to precisely state our result, we will need to introduce some notation:
For a batch of data X P X m , we define the k -th empirical moment of f p w, ¨q by [Equation]
where the supremum is also over all subgradients ∇ f p w, x i q P B w f p w, x i q in case f is not differentiable.
For X „ D m , we denote the k -th expected empirical moment by [Equation] and let [Equation]
Note that r r k, 1 “ r r k .
Our excess risk upper bounds will depend on a weighted average of the expected empirical moments for different batch sizes m P t 1 , 2 , 4 , 8 , ¨ ¨ ¨ , n u , with more weight being given to r r m for large m (which are smaller, by Lemma 9 below): for n “ 2 l , define [Equation]
where n i “ 2 ´ i n . Lemma 9 Under Assumptions 1 and 2, we have: r r p k q “ r e p k q [Equation]
In particular, r R k,n ď r r k .
Proof Let l P N , n “ 2 l and consider
[Equation]
Taking expectations over the random draw of X „ D n yields r e p k q n ď r e p k q n { 2 .
Thus, r R k,n ď r r k by the definition of r R n .
With this notation, we can give the precise statement of our upper bound:
Theorem 10 Grant Assumption 1.
Let f p¨ , x q be convex and L f -Lipschitz for all x , with L f À n p { 2 r R 2 k,n [Equation] for some p ě 2 and ϵ ď ? d .
Then, there are algorithmic parameters such that Algorithm 3 is ϵ 2 2 -zCDP, and has excess risk
[Equation]
Moreover, this excess risk is attained in r O p n p ` 1 q subgradient evaluations.
If p P r 1 , 2 q , then the same excess risk bound holds up to logarithmic factors.
The proof of Theorem 10 consists of three main steps: i)
We bound the empirical error of the noisy clipped subgradient subroutine (Lemma 12).
ii)
We prove that if an algorithm is on-average model stable (Definition 13), then it generalizes (Proposition 14).
iii)
We bound the on-average model stability of regularized ERM with non-smooth/non-Lipschitz loss (Proposition 15), leading to an excess population loss bound for Algorithm 2 run on the regularized empirical objective (c.f. lineof Algorithm 3).
By using these results with the proof technique of [29], we can obtain Theorem 10.
We proceed to prove the technical ingredients that will be used in the proof of Theorem 10.
First, we will prove a variant of Lemma 8 that bounds the bias and variance of the subgradient estimator in Algorithm 2. Lemma 11 Let p F λ p w q “ 1 n ř ni “ 1 f p w, x i q` λ 2 } w ´ w 0 } 2 be a regularized empirical loss on a closed convex domain W with ℓ 2 -diameter D . Let r ∇ F λ p w t q “ ∇ p F λ p w t q` b t ` N t “ 1 n ř ni “ 1 Π C p ∇ f p w, x i qq` λ p w ´ w 0 q ` N t be the biased, noisy subgradients of the regularized empirical loss in Algorithm 2, with N t „ N p 0 , σ 2 I d q and b t “ 1 n ř ni “ 1 Π C p ∇ f p w t , x i qq ´ 1 n ř ni “ 1 ∇ f p w t , x i q . Assume p r n p X q p k q ě sup w P W ␣n ř ni “ 1 } ∇ f p w, x i q} k
( for all ∇ f p w, x i q P B w f p w, x i q .
Then, for any
T ě 1 , we have: ˆ B :
[Equation]
sup t Pr T s } b t } ď p r n p X q p k q p k
´ 1 q C k ´and ˆΣ 2 : “ sup t Pr T s
E } N t }
[Equation]
“ dσ 2 .
Proof Fix any t .
We have
[Equation]
by Lemma 8 applied with D as the empirical distribution on X , and x i in Lemma 8 corresponding to ∇ f p w t , x i q in (7).
Taking supremum over t of both sides of (7) and recalling the definition of p r n p X q p k q proves the bias bound.
The noise variance bound is immediate from the distribution of N t .
Using Lemma 11, we can obtain the following convergence guarantee for Algorithm 2:
Lemma 12 Fix X P X
n
and let p F λ p w q “ 1 n ř ni “ 1 f p w, x i q ` λ 2 } w ´ w 0 } 2 for w 0 P W , where W is a closed convex domain with diameter D . Assume f p¨ , x q is convex and p r n p X q p k q ě sup w P W
[Equation]
( for all ∇ f p w, x i q P B w f p w, x i q . Denote p r n p X q “ “p r n p X q p k q ‰ 1 { k and ˆ w “ argmin w P W p F λ p w q .
Let η ď 2 λ .
Then, the output of Algorithm 2 satisfies E
[Equation]
where σ 2 “ 4 C 2 T n 2 ϵ 2 .
Proof We use the notation of Lemma 11 and write r ∇ F λ p w t q “ ∇ p F λ p w t q` b t ` N t “ 1 n ř ni “ 1 Π C p ∇ f p w, x i qq` λ p w ´ w 0 q ` N t as the biased, noisy subgradients of the regularized empirical loss in Algorithm 2, with N t „ N p 0 , σ 2 I d q and b t “ 1 n
[Equation]
t , x i q .
Denote y t ` 1 “ w t ´ r ∇ F λ p w t q , so that w t ` 1 “ Π W p y t ` 1 q .
For now, condition on the randomness of the algorithm (noise).
By strong convexity, we have
[Equation]
where we used non-expansiveness of projection and the definition of r ∇ F λ p w t q in the last line.
Now, re-arranging this inequality and taking expectation, we get
[Equation]
by optimality of ˆ w and the assumption that the noise N t is independent of w t ´ ˆ w and zero mean.
Also,
[Equation]
where ˆ B : “ sup t Pr T s } b t } ď p r n p X q p k q p k ´ 1 q C k ´ 1 and ˆΣ 2 : “ sup t Pr T s E } N t } 2 “ dσ 2 . by Lemma 11. We also used Young’s and Jensen’s inequalities and the fact that E N t “ 0 .
Further,
[Equation]
by Young’s inequality.
Combining these pieces yields
[Equation]
Iterating (8) gives us
[Equation]
since η ď 2 λ .
Plugging in the bounds on ˆ B and ˆΣ from Lemma 11 completes the proof.
Our next goal is to bound the generalization error of regularized ERM with non-Lipschitz/nonsmooth convex loss functions.
We will use a stability argument to obtain such a bound.
Recall the notion of on-average model stability [41]:
Definition 13 Let X “ p x 1 , ¨ ¨ ¨ , x n q and X 1 “ p x 1 1 , ¨ ¨ ¨ , x 1 n q be drawn independently from D .
For i P r n s , let X i : “ p x 1 , ¨ ¨ ¨ , x i ´ 1 , x 1 i , x i ` 1 , ¨ ¨ ¨ , x n q .
We say randomized algorithm A has on-average model stability α (i.e. A is α -on-average model stable) if E “ 1 n ř ni “ 1 } A p X q ´ A p
X i q} 2 ‰ ď α 2 .
The expectation is over the randomness of A and the draws of X and X 1 .
On-average model stability is weaker than the notion of uniform stability [14], which has been used in DP Lipschitz SCO (e.g. by [10]); this is necessary for obtaining our learnability guarantees without
Lipschitz continuity.
The main result in [41] showed that on-average model stable algorithms generalize well if f p¨ , x q is β f - smooth for all x , which leads to a restriction on L f .
We show that neither smoothness nor
Lipschitz continuity of f is needed to ensure generalizability:
Proposition 14 Let f p¨ , x q be convex for all x and grant Assumption 2 for k “ 2 .
Suppose
A : X n Ñ W is α -on-average model stable.
Then for any ζ ą 0 , we have
[Equation]
Proof Let X, X 1 , X i be constructed as in Definition 13.
We may write E r F p
A p X qq´ p F X
p
A p
X qqs “
E r 1 n ř ni “ 1 f p
A p X
i q , x i q ´ f p
A p X q , x i qs , by symmetry and independence of x i and A p X i q (c.f.
Equation B.2]).
Then by convexity, we have
[Equation]
Now, since A p X i q is independent of x i , we have:
[Equation]
Combining the above inequalities and recalling Definition 13 yields the result.
Using Proposition 14, we will bound the generalization error and excess (population) risk of regularized ERM in Proposition 16.
To prove Proposition 16, we first require the following bound on the generalization error of ERM with strongly convex loss:
Proposition 15 Let f p¨ , x q be λ -strongly convex, and grant Assumption 2.
Let A p X q : “ argmin w P W p F X p w q be the ERM algorithm.
Then,
[Equation]
Proof We first bound the stability of ERM and then use Proposition 14 to get a bound on the generalization error.
The beginning of the proof is similar to the proof of [41, Proposition D.6]: Let
X, X 1 , X i be constructed as in Definition 13.
By strong convexity of p F X i and optimality of A p X i q , we have
[Equation]
which implies
[Equation]
Now, for any w P W ,
[Equation]
Hence
[Equation]
by symmetry and independence of A p X q and X 1 .
Re-arranging the above equality and using symmetry yields
[Equation]
Combining (9) with (10) shows that ERM is α -on-average model stable for
[Equation]
The rest of the proof is where we depart from the analysis of [41] (which required smoothness of f p¨ , x q ): Bounding the right-hand side of (11) by Proposition 14 yields
[Equation]
for any ζ ą 0 .
Choosing ζ “ λn 2 , we obtain α
[Equation]
and α 2 ď 4 r 2 λ 2 n 2 . Applying Proposition 14 again yields (for any ζ 1 ą 0 )
[Equation]
by the choice ζ 1 “ λn 2 .
Proposition 16 Let f p¨ , x q be convex, w i ´ 1 , y P W , and ˆ w i : “ argmin w P W p F i p w q , where p F i p w q : “n i ř j P B i f p w, x j q ` λ i 2 } w ´ w i ´ 1 } 2 (c.f. line 6 of Algorithm 3).
Then,
[Equation]
where the expectation is over both the random draws of X from D and B i from X .
Proof Denote the regularized population loss by G i p w q : “ E r p F i p w qs “ F p w q ` λ i 2 } w ´ w i ´ 1 } 2 .
By Proposition 15, we have
[Equation]
Thus,
[Equation]
(12) since E r p F i p ˆ w i qs “ E r min w P W p F i p w qs ď min w P W E r p
F
i p w qs “ min w P W G i p w q ď λ i 2 } y ´ w
i ´ 1 } 2 `
F p y q .
Subtracting F p y q from both sides of (12) completes the proof.
We are ready to prove Theorem 10:
Proof [Proof of Theorem 10] We choose σ 2 i “ 4 C 2 i T i n 2 i ϵ 2 for C
i and T i to be determined exactly later.
Note that for λ
i and η i defined in Algorithm 3, we have η i ď
2 λ i for all i P r l s .
Privacy: Since the batches t B i u li “ 1 are disjoint, it suffices (by parallel composition [48]) to show that w i (produced by T i iterations of Algorithm 2 in line 7 of Algorithm 3) is ϵ 2 2 -zCDP for all i P r l s .
With clip threshold C i and batch size n i , the ℓ 2 sensitivity of the clipped subgradient update is bounded by ∆ “ sup w,X „ X 1 1 n i }
[Equation]
1 n i sup w,x,x
[Equation]
} Π
[Equation]
i p ∇ f p w, x qq ´ Π
C
i p ∇ f p w, x 1 qq} ď 2 C i n i .
(Note that the terms arising from regularization cancel out.)
Thus, by Proposition 6, conditional on the previous updates w 1: t , the p t ` 1 q -st update in line 5 of Algorithm 2 satisfies ϵ 2 2 T i
-zCDP.
Hence, Lemma 7 implies that w i (in lineof Algorithm 3) is ϵ 2 {+
+} -zCDP.
Excess risk: First, our choice of D i ensures that ˆ w i P W i , since
[Equation]
by definition of ˆ w i and L f -Lipschitz continuity of f p¨ , x j q for all j . Then by Lemma 12, we have
[Equation]
conditional on w i ´ 1 and the draws of X „ D n and B i „ X n i .
Taking expectation over the random sampling yields
[Equation]
where dσ 2 i ď {+
+}
[-k ´ 1 q C k ´ 
 i and r Σ 2 i ď dσ 2 i ` r r 2 ni n i ď 
-] dC {+
+}
i T i n 2 i ϵ 2 .
Choosing T i “ 1 λ i η i ln
[Equation]
D [-
-]2 λ i dσ 2 i η i
[Equation]
pi ln p n q and η to be determined later (polynomial in n ), we get
[Equation] [Equation]
Note that under Assumption 1, F is L -Lipshitz, where L “ sup w P W } ∇ F p w q} ď r by Jensen’s inequality.
Now, following the strategy used in the proof of [29, Theorem 4.4], we write
[Equation]
where ˆ w 0 : “ w ˚ .
Using (13), the first term can be bounded as follows:
[Equation]
fl if we choose C l “ r r 2 k ´ ϵ ? d ¯ 1 { k . Therefore,
[Equation]
[Equation]
if we choose
[Equation]
Next, Proposition 16 implies
[Equation]
q
[Equation]
for all i P r l s .
Hence l ÿ i “
[Equation]
Choosing C[-2 k ´ 
-] i “ r r 2 k,n i ˆ ϵn
i ?
d ln p n q ˙ 1 { k approximately equalizes the two terms above involving C i and we get l ÿ
[Equation]
where the last line holds verbatim if p ě 2 and holds up to an additional factor of ln p n q otherwise.
Assume p ě 2 .
Now, choosing
[Equation]
yields l ÿ
[Equation]
by the upper bound that we assumed on L f .
Combining the above pieces completes the excess risk proof.
Subgradient complexity: Our choice of T i “ r Θ
[Equation]
p n q implies that Algorithm 3 uses ř li “ 1 n i T i À ln p n q n p ` 1 subgradient evaluations. Remark 17 (Reduced Computational Cost for Approximate DP)
If one desires p ϵ, δ q -DP or p ϵ, δ qSDP instead of zCDP, then the gradient complexity of Algorithm 3 can be improved to O p n p ` 12 ln p n qq by using Clipped Noisy Stochastic Subgradient Method instead of Algorithm 2 as the subroutine in line 7 of Algorithm 3.
Choosing batch sizes
m i « ?
n i ă n i in this subroutine (and increasing σ 2 i by a factor of O p log p 1 { δ qq ) ensures p ϵ, δ q -DP by [1, Theorem 1] via privacy amplification by subsampling.
The same excess risk bounds hold for any minibatch size m i P r n i s , as the proof of Theorem 10 shows.
F.2. The Strongly Convex Case
Our algorithm is an instantiation of the meta-algorithm described in [29]: Initialize w 0 P W .
For j P r M s : “ r log 2 p log 2 p n qq s , let N j “ 2 j ´ 2 n { log 2 p n q , C j “ !
ř h
ă j N h ` 1 , . . .
, ř h ď j N h ) , and let w j be the output of Algorithm 3 run with input data
X j “ p x s q s P C j initialized at w j ´ 1 .
Output w M .
Assume without loss of generality that N j “ 2 p for some p P N .
Then, with the notation defined in Appendix F, we have the following guarantees:
Theorem 18 Grant Assumption 1.
Let ϵ ď ? d and f p¨ , x q be µ -strongly convex and L f -Lipschitz for all x P X , with L f À n p { 2 r R 2 k,n ˜ 1 ? n ` ˆ ? d ln p n q ϵn ˙ k ´ 1 k ¸ for some p ě 2 .
Then, there is a polynomial-time ϵ 2 2 -zCDP algorithm
A
based on Algorithm 3 with excess risk
[Equation]
Proof Privacy : Since the batches X j used in each phase of the algorithm are disjoint and Algorithmis ϵ 2
-zCDP, privacy of the algorithm follows from parallel composition of DP [48].
Excess risk: Note that N j samples are used in phase j of the algorithm.
For j ě 0 , let D 2 j “
E r} w j ´ w ˚ } 2 s and ∆ j “ E r F p w j q ´ F ˚ s .
By strong convexity, we have D 2 j ď 2∆ j µ .
Also,
[Equation]
for an absolute constant a ě 1 , by Theorem 10.
Denote E j “
[Equation]
Then since N j “ 2 N j ` 1 , we have
[Equation]
[Equation] where the second inequality holds because for any m “ 2 q
[Equation]
we have:
[Equation]
Now
[Equation]
(16) implies that (15) can be re-arranged as
[Equation]
Further
[Equation]
if M ě log log
[Equation]
, then
[Equation]
for an absolute constant A ą 0 , since ∆ 0 ď 2 L 2 µ and E 0 ě 2 L 2 µn implies ∆ 0 { E 0 “ na 2 ď n and 1 log p ∆ 0 { E 0 q “ 1 log p n q´ 2log p a q ď
A log p n q for some A ą 0 , so that
[Equation]
Therefore,
[Equation]
since N M “ n { 4 .
F.3. Asymptotic Upper Bounds Under Assumption
We first recall the notion of subexponential distribution:
Definition 19 (Subexponential Distribution)
A random variable Y is subexponential if there is an absolute constant s ą 0 such that P p| Y | ě t q ď 2 exp ` ´ ts ˘ for all t ě 0 .
For subexponential Y , we define } Y } ψ 1 : “ inf
[Equation]
: P p
[Equation] ě t q [Equation] Essentially all (heavy-tailed) distributions that arise in practice are subexponential [47].
Now, we establish asymptotically optimal upper bounds for a broad subclass of the problem class considered in [36, 62]: namely, subexponential stochastic subgradient distributions satisfying Assumption 3.
These upper bounds are a consequence of Lemma 3 combined with the following theorem (which uses the notation introduced in Section 2):
Theorem 20 Let f p¨ , x q be convex.
Assume r r 2 k ă {+
+} and Y i “ } ∇ f p w, x i q} 2 k is subexponential with E n ě max i Pr n s p} Y i } ψ 1 q @ w P W , ∇ f p w, x i q P B w f p w, x i q .
Assume that for sufficiently large n , we have sup w,x } ∇ f p w, x q} 2 k ď n q r p 2 k q for some q ě 1 and max ´
E n r p 2 k q ,
E 2 n p r p 2 k q q¯ ln ´ 3 nDβ 4 r 2 k ¯ ď ndq , where } ∇ f p w, x q ´ ∇ f p w 1 , x q} ď β } w ´ w 1 } for all w, w 1 P W , x P X , and subgradients ∇ f p w, x q P B w f p w, x q .
Then, lim n Ñ8 r R 2 k,n ď 4 r 2 k . Further, there exists N P N such that for all n ě N , the output of Algorithm 3 satisfies
[Equation]
If f p¨ , x q is µ -strongly convex, then the output of algorithm A (in Appendix F.2) satisfies
[Equation]
While a bound on sup w,x } ∇ f p w, x q} is needed in Theorem 20, it can grow as fast as any polynomial in n and only needs to hold for sufficiently large n .
As n Ñ 8 , this assumption is easily satisfied.
Likewise, Theorem 20 depends only logarithmically on the Lipschitz parameter of the subgradients β , so the result still holds up to constant factors if, say, β ď n p p r { D q as n Ñ 8 for some p ě 1 .
Crucially, our excess risk bounds do not depend on L f or β .
Under Assumption 3, the upper bounds in Theorem 20 hold with r replaced by ?
dγ 1 { k (by
Lemma 3).
These upper bounds, and the ones in Theorem 20, are tight up to logarithms for their respective problem classes, by the lower bounds in Appendix F.4.
Proof [Proof of Theorem 20] Step One: There exists N P N such that r r 22 k,n ď 16 r 22 k for all n ě N .
We will first use a covering argument to show that p r n p X q p 2 k q is upper bounded by 2 2 k ` 1 r p 2 k q with high probability.
For any α ą 0 , we may choose an α -net with N α ď ` 3 D 2 α ˘ d balls centered around points in W α “ t w 1 , w 2 , ¨ ¨ ¨ , w N α u Ă W such that for any w P W there exists i P r N α s with } w ´ w i } ď α
(see e.g. [39] for the existence of such W α ).
For w P W , let r w denote the element of
W α that is closest to w , so that } w ´ r w } ď α .
Now, for any X P X n , we have
[Equation]
where we used Cauchy-Schwartz and Young’s inequality for the first inequality, and the assumption of β -Lipschitz subgradients plus the definition of W α for the second inequality.
Further,
P ˜ 2 2 k
[Equation]
by a union bound and Bernstein’s inequality (see e.g. [59, Corollary 2.8.3]).
Choosing α “ 2 r 2 k β ensures that P p 2 2 k β 2 k α 2 k ą 2 {+
+} k ` 1 r p 2 k q q “ 0 and hence (by union bound)
[Equation]
by the assumption on n .
Next, we use this concentration inequality to derive a bound on r e p 2 k q n : [Equation] e p [Equation]
k q
[Equation]
for sufficiently large n .
Thus, r r 22 k,n ď 16 r 22 k for all sufficiently large n .
This establishes Step One. Step Two: lim n Ñ8 r R 2 k,n ď 4 r 2 k .
For all n “ 2 l , l, i P N , define h n p i q “ 2 ´ i r r [-k 
-]22 k, 2 ´ i n 1 t i Pr log 2 p n qsu .
Note that
ď h n p i q ď g p i q : “ 2 ´ i r r 22 k for all n, i , and that ři “ 1 g p i q “ r r 22 k ă 8 (i.e. g is integrable with respect to the counting measure).
Furthermore, the limit lim n Ñ8 h n p i q “ 2 ´ i lim n Ñ8 r r 22 k, 2 ´ i n exists since Lemmaimplies that the sequence t r r 22 k, 2 ´ i n u 8 n “ 1 is monotonic and bounded for every i P N .
Thus, by Lebesgue’s dominated convergence theorem, we have
[Equation]
where the last inequality follows from Step One .
Therefore, lim n Ñ8 r R 2 k,n ď 4 r 2 k .
By Theoremand Theorem 18, this also implies the last two claims in Theorem 20.
F.4. Lower Bounds
The work of [36] proved lower bounds under Assumption 3 that are tight (by our upper bounds) in most parameter regimes for γ “ D “ µ “ 1 and k “ O p 1 q .
Our (relatively modest) contribution in this subsection is: refining these lower bounds to display the correct dependence on γ, r, r r, D, µ ; tightening the convex lower bound [36, Theorem 6.4] in the regime d ą n ; and extending [36,
Theorems 6.1 and 6.4] to k " 1 .
Our first lower bounds hold even for affine functions:
Theorem 21 Let k ě 2 , D, γ, r p k q , r r p k q ą 0 , β f ě 0 , d ě 40 , n ą 7202 , and ρ ď d .
Then, for any ρ -zCDP algorithm A , there exist W , X Ă R d such that } w ´ w 1 } ď 2 D for all w, w 1 P W , a β f -smooth, linear, convex (in w ) loss f : W ˆ X Ñ R , and distributions D and D 1 on X such that:
Assumption 1 holds and if X 1 „ D 1 n , then
[Equation]
Assumption 2 holds and if X 1 „ D 1 n , then
[Equation]
Assumption 3 holds and if X „ D n , then
[Equation]
Proof We will prove part 3 first.
We begin by proving the result for γ “ D “ 1 .
In this case, it is proved in [36] that
[Equation]
for f p w, x q “ ´x w, x y with W “ B d 2 p 0 , 1 q and X “ t˘ 1 u d , and a distribution satisfying Assumption 3 with γ “ 1 .
Then f p¨ , x q is linear, convex, and β -smooth for all β ě 0 .
We prove the first (non-private) term in the lower bound.
By the Gilbert-Varshamov bound (see e.g. [2, Lemma 6]) and the assumption d ě 40 , there exists a set V Ď t˘ 1 u d with | V | ě 2 d { 20 , d Ham p
ν, ν 1 q ě d 8 for all ν, ν 1 P V , ν ‰ ν 1 .
For ν P V , define the product distribution Q ν “ p Q ν 1 , ¨ ¨ ¨ Q ν d q , where for all j P r d s ,
Q ν j “ # with probability 1 ` δ νj´with probability 1 ´ δ νjfor δ ν j P p 0 , 1 q to be chosen later.
Then E Q ν j : “ µ ν j “ δ ν j and for any w P W , x „ Q ν , we have
[Equation]
23) for δ ν j P p 0 , 1 q .
Now, let p : “ a d { n and δ ν j : “ pν j ? d .
Note that E Q ν : “ µ ν “ pν ? d and w ν : “ µ ν } µ ν } “ ν } ν } .
Also, } µ ν } “ p : “ } µ } for all ν P V .
Now, denoting F Q ν p w q : “ E x „ Q ν f p w, x q , we have for any w P W (possibly depending on X „ Q nν ) that
[Equation]
since } w } , } w ν } ď 1 .
Further, denoting ˆ w : “ argmin ν P V } w ν ´ w } , we have } ˆ w ´ w ν
2 ď 4 } w ν ´ w } 2 for all ν P V (via Young’s inequality).
Hence
[Equation]
Now we apply Fano’s method (see e.g. [64, Lemma 3]) to lower bound max ν P V E } ˆ w ´ w ν } 2 .
For all ν ‰ ν 1 , we have } w ν ´ w ν 1 } 2 ě } ν
´ ν 1 } 2 } ν }
ě 1 since d Ham p
ν, ν 1 q ě d 2 and ν P t˘ 1 u d implies } ν ´ ν 1 } 2 ě d 2 and } ν } 2 “ d .
Also, a straightforward computation shows that for any j P r d s and ν, ν 1 P V ,
[Equation]
for our choice of p , provided p ?
d “ 1 ? n P p 0 , 12 q , which holds if n ą 4 .
Hence by the chain rule for
KL-divergence,
[Equation]
for all ν, ν 1 P V .
Thus, for any w P W , Fano’s method yields
[Equation]
which is Ω p 1 q for d ě 40 ą 20 log p 2 q and n ą 7202 .
Combining this with (28) and plugging in } µ } “ b dn shows that
[Equation]
¸ for some ν P V (for any algorithm A ), where X „ Q nν .
Next, we scale our hard instance for arbitrary γ, D ą 0 .
First, we scale the distribution Q ν Ñ ˜ Q ν “ γ 1 { k Q ν , which is supported on ˜ X “ t˘ γ 1 { k u d .
Denote its mean by E Q ν : “ r µ ν “ γ 1 { k µ ν .
Also we scale W Ñ Ă W “ D W “ B d 2 p 0 , D q .
So our final (linear, convex, smooth) hard instance is f [Equation] [Equation] [Equation]
Denote
[Equation]
p [Equation]
Note that
[Equation]
Further, we have w ˚ : “ argmin w P W F p w q “ µ ν } µ ν } and r w ˚ : “ argmin r w P Ă W r F p r w q “ Dw ˚ .
Therefore, for any w P W , r w “ Dw P Ă W , we have
[Equation]
Thus,
[Equation]
so applying the lower bound for the case D “ γ “ 1 (i.e. for the unscaled F ) yields the desired lower bound via r F .
We will use nearly the same unscaled hard instances used to prove the private and non-private terms of the lower bound in part 3, but the scaling will differ.
Starting with the non-private term, we scale the distribution Q ν Ñ r Q ν “ r r k ?
d Q ν and X Ñ r X “ r r k ?
d X .
Also, scale W Ñ Ă W “ D W “ B d 2 p 0 , D q .
Let f p r w, r x q : “ ´x r w, r x y , which satisfies all the hypotheses of the theorem.
Also,
[Equation]
Now r w ˚ “ Dw ˚ as before and letting r F p¨q : “ E r x „ r Q ν f p¨ , r x q , we have
[Equation]
for any r w “ Dw .
Thus, applying the unscaled non-private lower bound established above yields a lower bound of Ω ´ r rD ?
n ¯ on the non-private excess risk of our scaled instance.
Next, we turn to the scaled private lower bound.
The unscaled hard distribution Q 1 ν given by
[Equation] with probability 1 ´ p p ´ 1 { k ν with probability p (with the same linear f and same W ) provides the unscaled lower bound
[Equation]
by the proof of [36, Theorem 6.4].
We scale Q 1 ν Ñ r Q 1 ν “ r r ?
d Q 1 ν , X Ñ r X “ r r ?
d X , and
W Ñ Ă W “ D W .
Then for any r w P Ă W ,
[Equation]
x „ Ă Q
[Equation]
Moreover, excess risk scales by a factor of r rD ?
d , as we saw above.
Thus, applying the unscaled lower bound completes the proof of part 1.
We use an identical construction to that used above in part 1 except that the scaling factor r r k gets replaced by r k .
It is easy to see that E
[Equation]
‰ for our construction, hence the result follows.
Remark
The main differences in our proof of part 3 of Theorem 21 from the proof of [36,
Theorem 6.4] (for γ “ D “ 1 ) are: 1) we construct a Bernoulli product distribution (built on [24,
Example 7.7]) instead of a Gaussian, which establishes a lower bound that holds for all k ě instead of just k “ O p 1 q ; and 2) we choose a different parameter value (larger p in the notation of the proof) in our application of Fano’s method, which results in a tighter lower bound: the term min t 1 , a d { n u in [36, Theorem 6.4] gets replaced with a d { n .
Also, there exist parameter settings for which our lower bound is indeed strictly greater than the lower bound in [36, Theorem 6.4]: for instance, if d ą n ą d { ρ and k Ñ 8 , then our lower bound simplifies to Ω p b dn q .
On the other hand, the lower bound in [36, Theorem 6.4] breaks as k Ñ 8 (since the k -th moment of their Gaussian goes to infinity); however, even if were extended to k Ñ 8 (e.g. by replacing their Gaussian with our Bernoulli distribution), then the resulting lower bound Ω p 1 ` d ?
ρn q would still be smaller than the one we prove above.
Theorem 23 Let k ě 2 , µ, γ, r r k , r k ą 0 , n P N , d ě 40 , and ρ ď d .
Then, for any ρ -zCDP algorithm A , there exist convex, compact sets W , X Ă R d of diameter D , a µ -smooth, µ -strongly convex (in w ) loss f : W ˆ X Ñ R , and distributions D and D 1 on X such that:
Assumption 1 holds with D « r r k µ , and if X 1 „ D 1 n , then
[Equation]
Assumption 2 holds with D « r k µ , and if X 1 „ D 1 n , then
[Equation]
Assumption 3 holds, D « γ 1 { k ? d µ , and if X „ D n , then
[Equation]
Proof We will prove part 3 first.
We first consider γ “ µ “ 1 and then scale our hard instance.
For f p w, x q : “ 12 } w ´ x } 2 , [36] construct a convex/compact domain W ˆ X Ă R d ˆ R d and distribution
D on X such that
[Equation]
for any k and any ρ -zCDP algorithm A : X n Ñ W if X „ D n .
So, it remains to a) prove the first term ( d { n ) in the lower bound, and then b) show that the scaled instance satisfies the exact hypotheses in the theorem and has excess loss that scales by a factor of γ 2 { k { µ .
We start with task a).
Observe that for f defined above and any distribution D such that E D P W , we have
[Equation]
(see [36, Lemma 6.2]), and
[Equation]
Thus, it suffices to prove that E } A p X q ´ E D } 2 Á dn for some D
such that E |x x
´ E x, e j y| k ď 1 .
This is a known result for products of Bernoulli distributions; nevertheless, we provide a detailed proof below.
First consider the case d “ 1 .
Then the proof follows along the lines of [24, Example 7.7].
Define the following pair of distributions on t˘ 1 u :
[Equation]
with probability
[Equation]
probabilityand
P
[Equation]
probability
[Equation]
with probability
[Equation]
for δ P p 0 , 1 q to be chosen later.
Notice that if X is a random variable with distribution P
ν
( ν P t 0 , 1 u ), then E | X ´ µ | k ď E | X | k ď 1 .
Also, E P ν “ δν for ν P t 0 , 1 u and | E P 1 ´ E P 0 | “ δ (i.e. the two distributions are δ -separated with respect to the metric ρ p a, b q “ | a ´ b | ).
Then by LeCam’s method (see e.g. [24, Eq.
and take Φ p¨q “ p¨q 2 ),
[Equation]
} TV s .
Now, by Pinsker’s inequality and the chain rule for KL-divergence, we have
[Equation]
˙ .
Choosing δ “ 1 ? 2 n ă 1 ? 2 implies } P n 0 ´ P n 1 } 2 TV ď nδ 2 “ 12 .
Hence there exists a distribution ˆ D P t P 0 , P 1 u on R such that
[Equation]
For general d ě 1 , we take the product distribution D : “ ˆ D d on X “ t˘ 1 u d and choose W “
B d 2 p 0 , ?
d q to ensure E D P W (so that (35) holds).
Clearly, E |x D ´ E D , e j y| k ď 1 for all j P r d s .
Further, the mean squared error of any algorithm for estimating the mean of D is
[Equation]
by applying the d “ 1 result to each coordinate.
Next, we move to task b).
For this, we re-scale each of our hard distributions (non-private given above, and private given in the proof of [36, Lemma 6.3] and below in our proof of part 2 of the theorem–see (40)): D Ñ γ 1 { k µ D “ r D , X Ñ γ 1 { k µ X “ r X , W Ñ γ 1 { k µ W “ Ă W and f : W ˆ X Ñ µf “ r f : Ă W ˆ r X .
Then r f p¨ , r x q is µ -strongly convex and µ -smooth for all r x P r X and
[Equation]
for any j P r d s , x „ D , r x „ r D , r w P Ă W .
Thus, the scaled hard instance is in the required class of functions/distributions.
Further, denote r F p w q “ E r f p w, x q , r w ˚ : “ argmin r w P Ă W r F p r w q “ E r D “ γ 1 { k µ E D .
Then, for any w P W , r w : “ γ 1 { k µ w , we have:
[Equation]
In particular, for w : “ A p X q and r w : “ γ 1 { k µ A p X q , we get
E A ,X „ D n « r F ˜ γ 1 { k
[Equation]
for any algorithm A : X n Ñ W .
Writing ˜
A p ˜ X
q : “ γ 1 { k µ A p X q and ˜ X : “ γ 1 { k µ X for X P X n , we conclude
E
[Equation]
A , ˜
[Equation]
for any r A : r X n Ñ Ă W .
Therefore, an application of the unscaled lower bound
[Equation]
which follows by combining part 3a) above with [36, Lemma 6.3], completes the proof of part 3.
We begin by proving the first (non-private) term in the lower bound:
For our unscaled hard instance, we will take the same distribution D “ P dν (for some ν P t 0 , 1 u ) on X “ t˘ 1 u d and quadratic f described above in part 1a with W : “ B d 2 p 0 , ?
d q .
The choice of W ensures E D P W so that (35) holds.
Further,
[Equation]
Thus, if we scale f Ñ r f “ µf , W Ñ Ă W : “ r r k µ ? 9 d W , X Ñ r X : “ r r k µ ?
9 d X and D Ñ r D “ r r k µ ?
9 d D , then r f p¨ , r x q is µ -strongly convex and µ -smooth, and
E « sup r w P Ă W
[Equation]
Moreover, if ´ [Equation]
¯
[Equation] “ r A : r X n Ñ Ă W is any algorithm and r X „ r D n , then by (36) and (35), we have
[Equation]
Next, we prove the second (private) term in the lower bound.
Let f be as defined above.
For our unscaled hard distribution, we follow [8, 36] and define a family of distributions t Q ν u ν P V on
R d , where V Ă t˘ 1 u d will be defined later.
For any given ν P V , we define the distribution Q ν as follows: X ν „ Q ν iff
[Equation]
with probability
[Equation]
p p [Equation] with probability p (40) where p : “ min ´ 1 , ?
d n ?
ρ ¯ .
Now, we select a set V Ă t˘ 1 u d such that | V | ě 2 d { 20 and d Ham p ν, ν 1 q ě d 8 for all ν, ν 1 P V , ν ‰ ν 1 : such V exists by standard Gilbert-Varshamov bound (see e.g. [2, Lemma 6]).
For any ν P V , if x „ Q ν and w P W : “ B d 2 p 0 , ?
dp ´ 1 { k q , then
E [Equation] sup
[Equation]
Note also that our choice of W and p ď 1 ensures that E r Q
ν s P W .
Moreover, as in the proof of [36,
Lemma 6.3], zCDP
Fano’s inequality (see [36, Theorem 1.4]) implies that for any ρ -zCDP algorithm
A , sup
[Equation]
Thus,
[Equation]
for some ν P V , by (35).
Now we scale our hard instance:
f Ñ r f “ µf , W Ñ Ă W : “ r r k 2 µ ?
d W ,
X Ñ r X : “ r r k 2 µ ?
d X and D Ñ r D “ r r k 2 µ ?
d D .
Then r f p¨ , r x q is µ -strongly convex and µ -smooth, and
E « sup
[Equation]
Moreover, if
[Equation]
is any ρ -zCDP algorithm and r X „ r D n , then
[Equation]
by (41).
We use an identical construction to that used above in part 1 except that the scaling factor r r k gets replaced by r k .
It is easy to see that E
[Equation]
for our construction, and the lower bound in part 2 follows just as it did in part 1.
This completes the proof.
Remark 24 Note that the lower bound proofs construct bounded (hence subexponential) distributions and uniformly L f -Lipschitz, β f -smooth losses that easily satisfy the conditions in Theorem 20.
Appendix G. Details and Proofs of Results in SectionG.1. Noisy Clipped Accelerated SGD for Smooth Convex Losses
We present our generic framework for accelerated SO with biased/noisy stochastic graadients in Algorithm 4.
It is built on the AC-SA of [31], and is the first accelerated DP algorithm for heavy-tailed SO.
1: Input: Data samples X P X n , number of iterations T ď n , convex loss f p¨ , x q , step size parameters t η t u t Pr T s , t α t u t Pr T s such that α 1 “ 1 , α t P p 0 , 1 q for all t ě 2 , DP mean estimation oracle MeanOracle (and its hyperparameters), privacy parameter ρ “ ϵ 2 { 2 .
2: Initialize w ag 0 “ w 0 P W and t “ 1 .
3: for t P r T s do 4: w mdt : “ p [Equation] ´ α t q [Equation]
Draw new batch B t (without replacement) of n { T samples from X .
6: r ∇ F t p
[Equation]
q : “ MeanOracle
[Equation]
∇
[Equation]
q 7: w t : “ argmin
[Equation]
[Equation]
[Equation]
p
[Equation]
´
[Equation]
9: end for 10: Output: w agT .
As a first step towards bounding the excess risk of our algorithm, we provide excess risk guarantees for Algorithm 4 in terms of the bias and variance of the MeanOracle :
Proposition 25 (Informal) Consider Algorithm 4 run with a MeanOracle satisfying r ∇ F t p w mdt q “ ∇ F p w mdt q` b t ` N t , where } b t } ď B (with probability 1 ), E N t “ 0 , E } N t } 2 ď Σ 2 for all t P r T ´ 1 s , and t N t u
Tt “ 1 are independent.
Assume that F : W Ñ R is convex and β -smooth.
Then there are choices of algorithmic parameters in Algorithm 4 such that:
[Equation]
The proof of Proposition 25, given towards the end of this subsection, involves a careful analysis of the bias and noise that propogates throughout the algorithm, and uses tools from [31].
15 We shall apply Proposition 25 with b t : “ E r ∇ F t p w t q ´ ∇ F p w t q and N t : “ r ∇ F t p w t q ´ E r ∇ F t p w t q for r ∇ F t p w t q given by Algorithm 1.
Then, combining Proposition 25 with Lemma 8 yields the main result of this subsection:
Theorem 26 (Smooth, Convex - Informal) Let ϵ ą 0 and assume F is convex and β -smooth.
Then, there are parameters such that Algorithm 4 instantiated with MeanOracle Algorithm 1 is ϵ 2 2 -zCDP
and
[Equation]
The full statement of Theorem 26 and its proof is given at the end of this subsection.
Remark 27 (Optimal rate for “sufficiently smooth” convex functions) Notice that the upper bound in Theorem 26 scales with the smoothness parameter β .
Thus, for sufficiently small β , the optimal rates (see Theorem 21) are attained.
For example, when k “ 2 , the upper bound in (43) matches the respective lower bound in Theorem 21 when β À r k D
´ d 5 ϵn ¯ 1 { 18 ; this would be the case for example if β and D are constants and d ě p ϵn q 1 { 5 .
In particular, for affine functions–which were not addressed in prior works [36, 62] since these works assumed ∇ F p w ˚ q “ 0 –we have β “ 0 , so that Algorithmis optimal (up to constant factors) for all k ě 2 .
Having discussed the dependence on β , let us focus on understanding how the bound in Theorem 26 scales with n, d and ϵ .
Thus, let us fix β “ D “ γ “ 1 and r “ ?
d for simplicity.
If k “ 2 , then the bound in (43) simplifies to O
[Equation]
, whereas the lower bound in Theorem(part 2) is Ω
[Equation]
¯ .
Therefore, the bound in (43) is tight if d 3 { 2 Á ϵn .
For general n, d, ϵ , (43) is nearly tight up to a multiplicative factor of ` ϵnd [Equation] ˘ 1 { 18 .
By comparison, the previous state-of-the-art ( not linear time ) bound for ϵ « 1 was O ´
[Equation] [36, Theorem 5.4].
Our bound (43) improves over [36,
Theorem 5.4] if d Á n 1 { 6 , which is typical in practical ML applications.
As k Ñ 8 , (43) becomes p w, x q does not depend on w , hence ∇ 2 ww f p w, x q “ 0 . The assumption made in [36, 62] that ∇ F p w ˚ q “ 0 is needed for the mean oracle of [33], which is used in [36, 62]. Also, note that the lower bound construction in Theorem 21 uses an affine function.
O ´b dn ` ` dn ˘ 4 { 5 ¯ for ϵ « 1 , which is strictly better than the bound in [36, Theorem 5.4].
Additionally, the gradient complexity of our algorithm is n , which is superior to the O p n 2 { d q complexity of the algorithm in [36].
Next, we provide the formal versions and proofs of results presented above.
Proposition 28 (Complete Version of Proposition 25) Consider Algorithm 4 run with a MeanOracle satisfying r ∇ F t p w mdt q “ ∇ F p w mdt q ` b t ` N t , where } b t } ď B (with probability 1 ), E N t “ 0 ,
E } N t } 2 ď Σ 2 for all t P r T ´ 1 s , and t
N t u Tt “ 1 are independent.
Assume that F : W Ñ R is convex and β -smooth, F p w 0 q ´ F ˚ ď ∆ , and } w 0 ´ w ˚ } ď D .
Suppose parameters are chosen in
Algorithm 4 so that for all t P r T s , η t ą βα 2 t and η t { Γ t “ η 1 { Γ 1 , where Γ t : “ # 1 , [Equation]
Then,
[Equation]
In particular, choosing α t “ 2 t ` 1 and η t “ 4 η t p t ` 1 q , @ t ě 1 , where η ě 2 β implies
[Equation]
BD. Further, setting η “ max ! 2 β, T
[Equation]
[Equation]
[Equation]
B 2 D ) implies
[Equation]
Proof We begin by extending [31, Proposition 4] to biased/noisy stochastic gradients.
Fix any w t ´ 1 , w agt ´ 1 P W .
By [31, Lemma 3], we have
[Equation]
for any z P W .
Denote
[Equation]
and d t : “ w agt ´ w mdt “ α t p w t ´ w t ´ 1 q .
Then using (45) with z “ w mdt , we have
[Equation]
by the expression for d t .
Now we apply [31, Lemma 2] with p p u q “ α t rx r ∇ F t p w mdt , u ys , µ 1 “ 0 , µ 2 “ η t , ˜ x “ w mdt , and
˜ y “ w t ´ 1 to obtain (conditional on all randomness) for any w P W :
[Equation]
Next, we combine the above inequality with (46) to get
F
[Equation]
for all w P W .
By Cauchy-Schwartz, we can bound
[Equation]
48) where the last inequality follows from maximizing the concave quadratic function q p} d t }q : “ ´ ” [Equation] ı } d t } 2 ` } N [Equation]
` b t }} d t } with respect to } d t } .
Plugging the bound (48) back into (47) shows that
[Equation]
[Equation]
Then it can be shown (see [31, Proposition 5]) that the assumptions on η t and α t imply that
[Equation]
for any w P W and any T ě 1 .
Now,
[Equation]
by definition.
Hence by convexity of F ,
[Equation]
Also, since Γ t { η t “ Γ 1 { η 1 for all t ě 1 , we have Γ T
T ÿ
[Equation]
since Γ 1 “ 1 .
Substituting the above bounds into (50), we get
[Equation]
Now, setting w “ w ˚ and taking expectation yields
[Equation]
where we used conditional independence of N t and w ˚ ´ w t ´ 1 given w t ´ 1 , Young’s inequality,
Cauchy-Schwartz, and the definitions of B 2 and Σ 2 .
This establishes the first claim of the theorem.
The second and third claims are simple corollaries, which can be verified as in [31, Proposition 7] and the ensuing discussion.
Theorem 29 (Complete Version of Theorem 26) Let ϵ ą 0 and assume F is convex and β -smooth.
Then, there are parameters such that Algorithm 4 instantiated with MeanOracle Algorithm 1 is ϵ 2 2 -zCDP.
Further, if n ě T : “
R min "´ βDr
[Equation]
¯
[Equation]
k {p 5 k ´ 1 q ´
[Equation]
d
[Equation]
βDr
[Equation]
[Equation]
*V , then,
[Equation]
Proof
Privacy: Choose σ 2 “ 4 C 2 T 2 ϵ 2 n 2 .
First, the collection of all r ∇ F t p w mdt q , t P r T s is ϵ 2 2 -zCDP: since the batches of data drawn in each iteration are disjoint, it suffices (by parallel composition [48]) to show that r ∇ F t p w mdt q is ϵ 2 2 -zCDP for all t .
Now, the ℓ 2 sensitivity of each clipped gradient update is bounded by ∆ “ sup w,X „ X 1 } Tn
[Equation]
sup w,x,x 1 } Tn Π C p ∇ f p w, x qq ´
Π C p ∇ f p w, x 1 qq} ď 2 CTn .
Thus, r ∇ F t p w mdt q is ϵ 2 2 -zCDP by Proposition 6.
Second, the iterates w agt are deterministic functions of r ∇ F t p w mdt q , so the post-processing property of differential privacy [16, 26] ensures that Algorithm 4 is ϵ 2 2 -zCDP.
Excess risk: Consider round t P r T s of Algorithm 4, where Algorithm 1 is run on input data t ∇ f p w t , x ti qu n { T i “ 1 .
Denote the bias of Algorithm 1 by b t : “ E r ∇ F t p w t q´ ∇
F p w t q , where r ∇ F t p w t q “ r ν in the notation of Algorithm 1.
Also let p ∇ F t p w t q : “ ˆ µ (in the notation of Lemma 8) and denote the noise by N t “ r ∇ F t p w t q ´ ∇ F p w t q ´ b t “ r ∇ F t p w t q ´ E r ∇ F t p w t q .
Then we have
B : “ sup t Pr T s } b t } ď r k p k ´ 1 q C k ´ 1 and Σ 2 : “ sup t Pr T s E r} N t } 2 s ď dσ 2 ` r 2 Tn À dC 2 T 2 ϵ 2 n 2 ` r 2 Tn , by Lemma 8.
Plugging these estimates for B and Σ 2 into Proposition 25 and setting C “ r p ϵn ?
dT q 1 { k , we get
[Equation]
Now, our choice of T implies that βD 2 T 2 ď rD
[Equation]
and we get the result upon plugging in T .
G.2. Noisy Clipped SGD for Strongly Convex Losses
Our framework for strongly convex losses, given in Algorithm 5, is to run SGD with biased/noisy stochastic gradients given by some MeanOracle .
This general framework is similar to the frameworks of [36, 62], but with some key differences.
The main differences in our approach lie in the choice of algorithmic parameters (including MeanOracle , step size, and iterate averaging weights), as well as in our analysis of the algorithm.
ď n , MeanOracle (and truncation/minibatch parameters), privacy parameter ρ “ ϵ 2 { 2 , stepsizes t η t u Tt “ 0 , averaging weights t ζ t u
Tt “ 0 .
2: Initialize w 0 P W .
3: for t P t 0 , 1 , ¨ ¨ ¨ , T u do 4:
Draw new batch B t (without replacement) of n { T samples from X . 5: r
[Equation]
F
[Equation]
q
[Equation]
“ MeanOracle
[Equation]
6: w
[Equation]
7: end for 8: Output: p w T : “ 1 Z T ř
Tt “ 0 ζ t w t ` 1 , where Z T “ ř
Tt “ 0 ζ t .
We have the following privacy and utility guarantees for Algorithm 5:
Theorem 30 (Smooth, Strongly Convex) Let ϵ ą 0 , and assume F is µ -strongly convex and βsmooth with κ “ βµ ď n { ln p n q .
Then, there are parameters such that Algorithm 5 instantiated with
MeanOracle Algorithm 1 is ϵ 2 2 -zCDP, and
[Equation]
To obtain Theorem 30, we will apply Proposition 31 to the biased noisy stochastic gradients provided by the private mean oracle Algorithm 1.
Combining Proposition 31 with Lemma 8 and a suitable choice of algorithmic parameters yields the excess risk bounds in Theorem 30.
The full proof is deferred to the end of this subsection.
As we saw in Theorem 23, the bound (57) is optimal up to a factor of r O p κ p k ´ 1 q{ k q .
To prove the excess risk bounds in Theorem 30, we first derive a novel convergence guarantee for projected SGD with generic noisy biased stochastic gradients:
Proposition 31 Let F : W Ñ R be µ -strongly convex and β -smooth with condition number κ : “ βµ .
Let w t ` 1 : “ Π W r w t ´ η t r ∇ F t p w t qs , where r ∇ F t p w t q “ ∇ F p w t q ` b t ` N t , such that the bias and noise (which can depend on w t and the samples drawn) satisfy } b t } ď B (with probability 1 ), E N t “ 0 , E } N t } 2 ď Σ 2 for all t P r T ´ 1 s , and that t N t u
Tt “ 1 are independent.
Then, there exist stepsizes t η t u
Tt “ 1 and weights t ζ t u Tt “ 0 such that the average iterate p w T : “
[Equation]
satisfies
[Equation]
Compared to the results in [6] for (non-strongly) convex DP ERM and [3] for non-private unconstrained PL losses, Proposition 31 is tighter, since we leverage smoothness and strong convexity.
Our analysis also corrects the issue in [36, 62].
Proof [Proof of Proposition 31] Define g p w t q “ ´ 1 η t p w t ` ´
w t q .
Then
[Equation] Now
[Equation]
conditional on all randomness
[Equation]
we use smoothness and strong convexity to write:
[Equation]
where we used the fact that x Π W p
y q ´ x, Π W p
y q ´ y y ď
for all x P W , y P R d (c.f.
[15, Lemma 3.1]) to obtain the last inequality.
Thus,
[Equation]
Combining the above inequality with (58)
[Equation]
we get
[Equation]
Next, consider
[Equation]
by independence of N t (which has zero mean) and w t ´ w ˚ , and Young’s inequality.
Next, note that v : “ w t ´ η t p ∇ F p w t q ` b t q is independent of N t , so E x N t , Π W p v qy “ 0 .
Thus,
[Equation]
by Cauchy-Schwartz and non-expansiveness of projection.
Further,
[Equation]
by Young’s inequality.
Therefore,
[Equation]
Plugging this bound back into (59) and choosing η t ď 1 β ď 1 µ yields:
[Equation]
Next, we apply Lemma 32 (see below) with r t : “ E } w t ´ w ˚ } 2 , s t : “ E F p w t ` 1 q ´ F ˚ ´ 2 B 2 µ , a : “ µ 2 , b : “ 2 , c “ 2Σ 2 , and g “ β .
We may assume s t ě 0 for all t : if this inequality breaks for some t , then simply return w t ` 1 instead of p w T to obtain E F p w t q ´ F ˚ ă 2 B 2 µ .
Thus,
[Equation]
Finally, Jensen’s inequality yields the theorem.
Lemma 32 [55, Lemma 3] Let b ą 0 , let a, c ě 0 , and t η t u t ě 0 be non-negative step-sizes such that η t ď 1 g for all t ě 0 for some parameter g ě a .
Let t r t u t ě
and t s t u t ě 0 be two non-negative sequences of real numbers which satisfy
[Equation]
for all t ě 0 .
Then there exist particular choices of step-sizes η t ď 1 g and averaging weights ζ t ěsuch that
[Equation]
where Γ T : “ ř
Tt “ 0 γ t .
Theorem 33 (Re-statement of Theorem 30) Let ϵ ą 0 , and assume F is µ -strongly convex and β -smooth with κ “ βµ ď n { ln p n q .
Then, there are parameters such that Algorithm 5 instantiated with MeanOracle Algorithm 1 is ϵ 2 2 -zCDP, and
[Equation]
Proof Privacy: Choose σ 2 “ 4 C 2 T 2 ϵ 2 n 2 .
Since the batches of data drawn in each iteration are disjoint, it suffices (by parallel composition [48]) to show that r ∇ F t p w t q is ϵ 2 2 -zCDP for all t .
Now, the ℓ 2 sensitivity of each clipped gradient update is bounded by ∆ “ sup w,X „ X 1 } T n ř x P B t Π C p ∇ f p w, x qq ´ ř x 1 P
[Equation]
[Equation]
Π C p ∇
[Equation]
p w, x 1 qq
[Equation]
sup
[Equation]
x
[Equation]
p ∇ f p w, x qq ´ Π
C
p ∇ f p w, x 1 qq} ď 2 CTn .
Hence Proposition 6 implies that the algorithm is ϵ 2 2 -zCDP.
Excess risk: For any iteration t P r T s , denote the bias of Algorithm 1 by b t : “ E r ∇ F t p w t q´ ∇
F p w t q , where r ∇ F t p w t q “ r ν in the notation of Algorithm 1.
Also let p ∇ F t p w t q : “ ˆ ν (in the notation of
Lemma 8) and denote the noise by N t “ r ∇ F t p w t q´ ∇ F p w t q´ b t “ r ∇ F t p w t q´ E r ∇ F t p w t q .
Then we have B : “ sup t Pr T s } b t } ď r p k q p k ´ 1 q C k [Equation] 1 and Σ 2 : “ sup t Pr T s E r} N t } 2 s ď dσ [Equation]
r
[Equation]
Tn À dC
[Equation]
T 2 ϵ 2 n 2 ` r 2 k Tn , by Lemma 8.
Plugging these bias and variance estimates into Proposition 31, we get
[Equation]
Choosing C “ r k ´ ϵ 2 n 2 dT ¯ 1 { 2 k implies
[Equation]
Finally, choosing T “ R
[Equation]
À κ ln p n q yields the result.
Appendix H. Details and Proofs of Results in Section
Assume:
f p w, x q “ f 0 p w, x q ` f 1 p w q ; f 0 p¨ , x q is differentiable (maybe non-convex), f 1 is proper, closed, and convex (maybe non-differentiable) for all x P X ; and F p w q “ F 0 p w q ` f 1 p w q “
E x „ D r f 0 p w, x qs ` f 1 p w q satisfies the Proximal-PL condition [37]:
Definition 34 ( µ -PPL) Let F p w q “ F 0 p w q ` f 1 p w q be bounded below; F 0 is β -smooth and f 1 is convex.
F satisfies Proximal Polyak-Łojasiewicz inequality with parameter µ ą
if
[Equation]
Theorem 34 is an extension of the classical PL inequality [50], allowing for constrained optimization and/or non-smooth regularizer, depending on the choice of f 1 .
For PPL losses, we propose Algorithm 6, which is a linear time algorithm.
Recall that the proximal operator of a convex function g is defined as
[Equation]
Proximal operators generalize projections: if g “ ι W , then prox g p
z q “ Π W p z q : “ argmin y P W } y ´ z } 2 .
The privacy and excess risk guarantees of Algorithm 6 are provided in Theorem 35:
Theorem 35 (Proximal-PL) Let ϵ ą 0 and assume F p w q “ F 0 p w q ` f 1 p w q is µ -PPL for βsmooth F 0 , with κ “ βµ ď n { ln p n q .
Then, there are parameters such that Algorithm 6 is ϵ 2 2 -zCDP, and
[Equation]
1: Input: Data X P X n , T ď n , MeanOracle (and truncation/minibatch parameters), privacy parameter ρ “ ϵ 2 { 2 , stepsizes t η t u T ´ 1 t “ 0 .
2: Initialize w 0 P W .
3: for t P t 0 , 1 , ¨ ¨ ¨ , T ´ 1 u do 4:
Draw new batch B t (without replacement) of n { T samples from X .
5: r ∇ F 0 t p w t q : “ MeanOracle pt ∇ f 0 p w t , x qu x P B t ; nT ; ϵ 2 2 q 6: w t ` 1 “ prox η t f´ w t ´ η t r ∇ F 0 t p w t q ¯ 7: end for 8: Output: w T .
The bound in Theorem 35 nearly matches the smooth, strongly convex lower bound in Theoremup to the r O p κ p 2 k ´ 2 q{ 2 q factor, and is attained without convexity.
In particular, Algorithm 6 is nearly optimal.
17 To prove Theorem 35, we first derive Proposition 36, a generic convergence guarantee for
Algorithm 6 with biased, noisy stochastic gradients in terms of the bias and variance of the oracle:
Proposition
Consider Algorithm 6 with biased, noisy stochastic gradients: r ∇ F 0 t p w t q “ ∇ F 0 p w t q ` b t ` N t , and stepsize η “ 12 β .
Assume that the bias and noise satisfy } b t } ď B (with probability 1 ), E N t “ 0 , E } N t } 2 ď Σ 2 for all t P r T ´ 1 s , and that t N t u
Tt “ 1 are independent.
Assume further that F is µ -PPL, F 0 is β -smooth, and F p w
q
´
F ˚ ď ∆ .
Then,
[Equation]
Proposition 36 generalizes [3, Theorem 6]–which provides a similar bound for the unconstrained, classical PL problem–to the proximal setting.
However, the proof of Proposition 36 is very different from the proof of [3, Theorem 6], since the proximal operator makes it difficult to bound the excess loss without convexity when the stochastic gradients are biased/noisy.
Our proof draws inspiration from the proof of [45, Theorem 3.1] (for the case of unbiased stochastic gradients and
Lipschitz loss function).
Specifically, we view each biased/noisy proximal evaluation as an objective perturbation [19] problem.
Then, using techniques from the analysis of objective perturbation, we bound the difference between the errors of the biased, noisy stochastic proximal gradient steps and the unbiased noiseless proximal gradient steps, the latter of which can be bounded via the PPL inequality.
Compared to the proof of [45, Theorem 3.1], here we need to carefully handle the bias term and bound the error without appealing to Lipschitz continuity of f .
Proof [Proof of Proposition 36]
Our proof extends the ideas in [45] to generic biased and noisy gradients without using Lipschitzness of the loss.
By β -smoothness, for any r P r T ´ 1 s , we have
[Equation]
where we used Young’s inequality to bound
[Equation]
[Equation]
Next
[Equation]
we will bound E
[Equation]
r } 2 ` f 1 p w r ` 1 q ´ f 1 p w r q ` x b r ` N r , w r ` 1 ´ w r y ‰ .
Denote H priv r p y q : “
[Equation]
∇
[Equation]
p
[Equation]
r q , y
[Equation]
r y `
[Equation]
} y
[Equation]
r
[Equation]
y
[Equation]
r
[Equation]
`
[Equation]
r [Equation] N r , y ´ w r y and H r p y q : “ x ∇ F 0 p w r q , y ´ w r y ` β } y ´ w r } 2 ` f 1 p y q ´ f 1 p w r q .
Note that H r and H priv r are 2 β -strongly convex.
Denote the minimizers of these two functions by y ˚ and y priv ˚ respectively.
Now, conditional on w r and N r ` b r , we claim that
[Equation]
To prove (66), we will need the following lemma:
Lemma 37 [43, Lemma B.2] Let
H p
y q , h p
y q be convex functions on some convex closed set
Y Ď R d and suppose that H is 2 β -strongly convex.
Assume further that h is L h -Lipschitz.
Define y 1 “ arg min
y P Y
H p
y q and y 2 “ arg min y P Y
r H p
y q ` h p y
qs .
Then } y 1 ´ y 2 } 2 ď L h 2 β .
We apply Lemma 37 with H p
y q : “ H r p y q , h p y q : “ x N r ` b r , y y , L h
“ } N r ` b r } , y 1 “ y ˚ , and y 2 “ y priv ˚ to get } y ˚
´
y priv ˚ } ď } N r ` b r } 2 β .
On the other hand,
H priv r
[Equation]
, y ˚ y .
Combining these two inequalities yields
[Equation]
as claimed.
Also, note that w r ` 1 “ y priv ˚ .
Hence
[Equation]
satisfies
[Equation]
where we used the assumptions that F is µ -PPL and F 0 is 2 β -smooth in the last inequality.
Plugging the above bounds back into (64)
[Equation]
we obtain
[Equation]
whence
[Equation] [Equation]
Using (72) recursively and summing the geometric series
[Equation]
we get
[Equation]
With Proposition 36 in hand, Theorem 35 then follows from substituting the bias and variance bounds of our MeanOracle (given in Lemma 8) into Proposition 36, and optimizing for clip threshold and batch size n { T .
We provide the detailed proof below:
Theorem 38 (Re-statement of Theorem 35) Let ϵ ą 0 and assume F p w q “ F 0 p w q ` f 1 p w q is µ -PPL for β -smooth F 0 , with κ “ βµ ď n { ln p n q .
Then, there are parameters
such that Algorithmis ϵ 2
-zCDP, and
[Equation]
Proof We choose σ
[Equation]
Privacy: By parallel composition (since each sample is used only once) and the post-processing property of DP (since the iterates are deterministic functions of the output of MeanOracle ), it suffices to show that r ∇ F t p w t q is ϵ 2 2 -zCDP for all t ě 0 .
By our choice of σ 2 and Proposition 6, r ∇ F t p w t q is ϵ 2 2zCDP, since it’s sensitivity is bounded by sup X „ X 1 ,w Tn ››ř [Equation] t
[Equation]
Excess risk: For any iteration t P r T s , denote the bias of Algorithm 1 by b t : “ E r ∇ F t p w t q´ ∇
F p w t q , where r ∇ F t p w t q “ r ν in the notation of Algorithm 1.
Also let p ∇ F t p w t q : “ ˆ ν (in the notation of
Lemma 8) and denote the noise by N t “ r ∇ F t p w t q ´ ∇ F p w t q ´ b t “ r ∇ F t p w t q ´ E r ∇ F t p w t q .
Then we have B : “ sup t Pr T s } b t } ď r k p k ´ 1 q C k ´ 1 and Σ 2 : “ sup t Pr T s E r} N t } 2 s ď dσ 2 ` r 2 k Tn ď 4 dC 2 T 2 ϵ 2 n 2 ` r 2 k Tn , by Lemma 8.
Plugging these bounds on B 2 and Σ 2 into Proposition 36, and choosing
T “
Q
[Equation]
p n q where ∆ ě F p w 0 q ´ F ˚ , we have:
[Equation]
for any C ą 0 .
Choosing C
“ r ´ [Equation] 2 n 2 dT¯ 1 { 2 k makes the last two terms in the above display equal, and we get
[Equation]
as claimed.