Momentum Aggregation for Private Non-convex ERM
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We introduce new algorithms and convergence guarantees for privacy-preservingnon-convex Empirical Risk Minimization (ERM) on smooth d -dimensional objectives.
We develop an improved sensitivity analysis of stochastic gradient descent onsmooth objectives that exploits the recurrence of examples in different epochs.
Bycombining this new approach with recent analysis of momentum with private aggregation techniques, we provide an ( (cid:15), δ ) -differential
private algorithm that ﬁndsa gradient of norm O (cid:16)
d 1 / 3 ( (cid:15)N ) 2 / 3 (cid:17) in O (cid:16) N 7 / 3 (cid:15) 4 / 3 d 2 /(cid:17) gradient evaluations, improvingthe previous best gradient bound of ˜ O (cid:16) d 1 / 4 √ (cid:15)N (cid:17) .
Introduction
In recent years, statistical machine learning models have been deployed in many domains such ashealth care, education, criminal justice or social studies [Chen et al., 2021,
He et al., 2019, Jianget al., 2017].
However, the release of statistical estimates based on these sensitive data comes withthe risk of leaking personal information of individuals in the original dataset.
One naive solutionfor this problem is to remove all the identifying information such as names, races or social securitynumbers.
Unfortunately, this is usually not enough to preserve privacy.
It has been shown in variousworks that an adversary can take advantages of structural properties of the rest of the dataset toreconstruct information about certain individuals [Backstrom et al., 2007, Dinur and Nissim, 2003].
Thus, we would need a stronger privacy-preserving mechanism.
Over the past couple of decades,differential privacy [Dwork et al., 2006] has emerged as the dominant privacy notion for machinelearning problems.
Deﬁnition 1 (Differential Privacy [Dwork and Roth, 2014]) .
A randomized algorithm M : X N (cid:55)→ R dsatisﬁes ( (cid:15), δ ) − differential privacy (( (cid:15), δ )-DP) if for any two data sets D, D (cid:48)
∈
X N differing by atmost one element and any event E ⊆ R d , it holds that:
[Equation]
Roughly speaking, differential privacy guarantees that the outputs of two neighboring datasets(datasets that differ in at most one datapoint) are almost the same with high probability, thuspreventing the adversary from identifying any individual’s data.
In this paper, we are interested in designing ( (cid:15), δ ) − private algorithms for non-convex empirical riskminimization (ERM) problems.
In ERM problems, given N i.i.d samples x 1 , ..., x N ∈ X from someunknown distribution P , the goal is to ﬁnd w ∈ R d such that w minimizes the empirical loss deﬁnedas follows:
[Equation]
where f : R d × X (cid:55)→ R is the loss function associated with the learning problem.
This is a settingthat commonly arises in modern machine learning problem.
For example, in image classiﬁcationproblems, the data point x would be a tuple of (image, label), w denotes the parameters of ourmodel, and f ( w, x ) represents composing the model predictions with some loss function such ascross-entropy.
We are interested in ﬁnding a critical point, or a point such that the norm of theempirical gradient (cid:107)∇ F ( w ) (cid:107) is as small as possible.
Further, we want all the outputs w 1 , w 2 , ..., w Tto be differentially private with respect to the N training samples.
Private ERM has been well studied in the convex settings.
The approaches in this line of workcan be classiﬁed into three main categories: output perturbation [Dwork et al., 2006, Chaudhuriet al., 2011, Zhang et al., 2017, Wu et al., 2017], objective perturbation [Chaudhuri et al., 2011,
Kifer et al., 2012, Iyengar et al., 2019, Talwar et al., 2014], and gradient perturbation [Bassily et al.
,2014, Wang et al., 2017, Jayaraman et al., 2018, Wang et al., 2018].
All of these approaches havebeen shown to achieve the asymptotically optimal bound ˜ O (cid:16) √ d (cid:15)N (cid:17) for smooth convex loss (withoutput perturbation requiring strong convexity to get the optimal bound) in (near) linear time.
Onthe other hand, the literature on private non-convex ERM is nowhere as comprehensive.
The ﬁrsttheoretical bound in private non-convex ERM is from [Zhang et al., 2017].
They propose an algorithmcalled Random Round Private Stochastic Gradient Descent (RRSGD) which is inspired by the resultsfrom [Bassily et al., 2014, Ghadimi and Lan, 2013].
RRSGD is able to guarantee the utility boundof O
[Equation]
(cid:17) .
However, RRSGD takes O (cid:0) N 2 d (cid:1) gradient computations to achievethis, which can be troublesome for high-dimensional problem.
[Wang et al., 2018] then improvesupon this utility bound by a factor of O (cid:0) (log( n/δ )) 1 / 4 (cid:1) .
They achieve this rate by using full-batchgradient descent which is not a common practice in non-private machine learning in which very largebatch sizes actually require careful work to make training efﬁcient.
Recently
[Wang et al., 2019b]tackle both runtime and utility issues by introducing a private version of the Stochastic Recursive
Momentum (DP-SRM)
[Cutkosky and Orabona, 2019].
By appealing to variance reduction as wellas privacy ampliﬁcation by subsampling [Balle et al., 2018, Abadi et al., 2016], DP-SRM achievethe bound O
[Table]
[Equation]
gradient complexity.
However, DR-SRM stillrequires the batch size to be O (cid:16) √
[Equation] [Equation]
for the analysis to work.
Finally, although our focus in thispaper is the ERM problem, there are also various works on private stochastic non-convex/convexoptimization [Bassily et al., 2019, 2021, Feldman et al., 2018, 2020, Zhou et al., 2020, Asi et al.
,2021, Wang et al., 2019a, Kulkarni et al., 2021].
Contributions.
We ﬁrst provide the analysis for the private version of Normalized SGD (DP-NSGD)[Cutkosky and Mehta, 2020] for unconstrained non-convex ERM.
By using the tree-aggregationtechnique [Chan et al., 2011, Dwork et al., 2010] to compute the momentum privately, we canensure the privacy guarantee while adding noise of only ˜ O (cid:16) √ T (cid:15) √ N (cid:17) (where T is the total number ofiterations).
This allows us to achieve the same asymptotic bound ˜ O (cid:16) d 1 / 4 √ (cid:15)N (cid:17) on the expectation of thegradient as [Zhang et al., 2017, Wang et al., 2018, 2019b] without appealing to privacy ampliﬁcationtechniques which is usually required for private SGD to have a good utility guarantee [Abadi et al.,2016].
DP-NSGD also does not require a large batch size as in [Wang et al., 2018, 2019b]; it hasutility guarantee for any batch size.
This tree-aggregation technique is morally similar to the approachin [Kairouz et al., 2021, Guha Thakurta and Smith, 2013] for online learning.
However, unlike in[Kairouz et al., 2021, Guha Thakurta and Smith, 2013], we do not restrict our loss function to beconvex and we will also extend tree-aggregation technique to SGD with momentum.
Further, weprovide a new variant of Normalized SGD that takes advantages of the fact that the gradients of thenearby iterates are close to each other due to smoothness.
This new algorithm is able to guarantee anerror of ˜ O (cid:16) d 1 / 3 ( (cid:15)N ) 2 / 3 (cid:17) in O (cid:16) N 7 / 3 (cid:15) 4 / 3 d 2 /(cid:17) gradient computations which, to our knowledge, is the bestknown rate for private non-convex ERM.
Organization.
The rest of the paper is organized as follows.
In section 2, we deﬁne our problem ofinterest and the assumptions that we make on the problem settings.
We also provide some backgroundon Differential Privacy as well as some high-level intuition on tree-aggregation technique.
We thenformally describe our ﬁrst private variant of Normalized SGD in section 3 and discuss its privacyguarantee and theoretical utility bound.
In section 4, we introduce a novel sensitivity-reduced analysis for Normalized SGD that allows us to improve upon the utility bound in section 3.
Finally, weconclude with a discussion in section 5.
Preliminaries
Assumptions.
We deﬁne our loss function as f ( w, x ) : R d × X (cid:55)→ R where x ∈ X is some samplefrom the dataset.
Throughout the paper, we make the following assumptions on f ( w, x ) .
Deﬁne adifferentiable function f ( . ) :
R d (cid:55)→ R to be G -Lipschitz iff (cid:107)∇ f ( y ) (cid:107) ≤ G for all y ∈ R d , and to be
L -smooth iff (cid:107)∇ f ( y 1 ) − ∇ f ( y 2 ) (cid:107) ≤ L (cid:107) y 1 − y 2 (cid:107) where (cid:107) .
(cid:107) is the standard 2-norm unless speciﬁedotherwise.
We assume that f ( w, x ) is differentiable, G -Lipschitz, and L -smooth with probability 1.
We also assume that provided the initial point
w 1 ∈ R
d there exists some upper bound R of F ( w 1 )or formally: sup w ∈ R d F ( w 1 ) − F ( w ) ≤ R .
Differential Privacy.
Let D be a dataset containing N datapoints.
Then two datasets D, D (cid:48) are saidto be neighbors
if (cid:107) D − D (cid:48) (cid:107) 1 = 1 or in other words, the two datasets differ at exactly one entry.
Nowwe have the deﬁnition for sensitivity:
Deﬁnition 2.
( L 2 -sensitivity)
The L 2 -sensitivity of a function f ( . ) :
X N (cid:55)→ R d is deﬁned as follows:
[Equation]
In this work, we also make use of Renyi Differential Privacy (RDP)
[Mironov, 2017].
RDP is arelaxation of ( (cid:15), δ ) − DP and can be used to improve the utility bound and do composition efﬁcientlywhile still guaranteeing privacy.
Deﬁnition 3.
(( α, (cid:15) )
-RDP
[Mironov, 2017])
A randomized algorithm f : X N (cid:55)→ R d is said to have(cid:15)
− RDP of order
α if for any neighboring datasets D, D (cid:48) it holds that:
[Equation]
where D
[Equation]
αg [1:8] g [1:4] g [1:2] g [1:1] g [2:2] g [3:4] g [3:3] g [4:4] g [5:8] g [5:6] g [5:5] g [6:6] g [7:8] g [7:7] g [8:8]
Tree-aggregation.
Consider the problem of releasing private partial sum s [ x,y ]
= (cid:80) yi
= x g i givena stream of vectors g 1 , . . . , g t .
Assuming (cid:107) g
i (cid:107) ≤ G ∀ i ∈ [ t ] , then the simplest solution for thisproblem is to add Gaussian noise with standard deviation ˜ O (cid:0) √ t/(cid:15) (cid:1) to each g
i so that each g iis ( (cid:15)/ √ t, δ/t ) − DP.
Then, by advanced composition, any partial sum s [ x,y ] would be ( (cid:15), δ ) − DP.
However, this problem can be solved much more efﬁciently using tree-aggregation [Chan et al. ,2011, Dwork et al., 2010].
In tree-aggregation, we consider a complete binary tree where each ofthe leaf nodes is labeled with g i and each internal node in the tree is the sum of its children.
Thetree is illustrated in Fig.1.
It is clear from Fig.1 that each g i only affects its ancestors, or at most(cid:100) log t (cid:101) + 1 nodes.
Thus, by adding Gaussian noise with standard deviation O (log t/(cid:15) ) to every node, the complete tree will be ( (cid:15), δ ) − DP.
Furthermore, since we only need at most log t nodes to computeany partial sum s [ x,y ] , the noise added to every s [ x,y ] would be bounded by O (log 1 . 5 t/(cid:15) ) , which is alot less than the noise that we would add using advanced composition.
To get a general idea on how we can use this tree-aggregation technique to design private algorithms,let us analyse the update of SGD with initial point w 1 = 0 :
[Equation]
where ∇ f ( w t , x t ) is the gradient of the loss function evaluated on the current iterate w t and somesample x t .
Notice that every iterate w t +1 would be a function of the sum of the gradients s t = 118 (cid:80) ti =1 ∇ f ( w t , x t ) up to iteration t .
Thus, if we can use tree-aggregation to privately compute everypartial sum s [ x,y ] = (cid:80) yi
= x ∇ f ( w i , z i ) , we can the use these partial sums to compute every iterate w tand w t is also private by post-processing [Dwork and Roth, 2014].
Similarly, we can also use thisaggregation technique for SGD with momentum where we replace ∇ f ( w t , x t ) with the momentumm t .
This method of computing momentum would be the main ingredient for our private algorithmsand a more detailed discussions are provided in section 3 and 4.
Tree-aggregated momentum hasbeen brieﬂy mentioned in the momentum variant of Differentially Private Follow-The-RegularizedLeader (DP-FTRL) in [Kairouz et al., 2021].
However, their algorithm does not directly composethe momentum using the tree-aggregation and they also do not provide any formal guarantee for thealgorithm.
Private Normalized SGD
Before we go into the details of our private algorithm, ﬁrst let us set up some notations that arefrequently used in the rest of the paper.
We deﬁne the set of subsets I as follows:
[Equation]
Each interval in I corresponds to a node in the tree described above (In Fig.1, the set
I is{ [1 , 1] , [2 , 2] , [1 , 2] , . . . } ).
Thus, the set of intervals I is essentially a different way to store thebinary tree for tree-aggregation mechanism.
Then, for every interval [ a, b ] , there exists O (log T )disjoint subsets [ x 1 , y 1 ] , ..., [ x n , y n ] ∈ I such that [ a, b ] = (cid:83) ni
=
[ x i , y i ] Daniely et al.
Wedenote the set of such subsets as COMPOSE ([ a, b ]) , which can be computed using Algorithm 3.
Now we can proceed to the analysis of the private Normalized SGD algorithm described in Algorithm1.
The main idea of Algorithm 1 is based on two observations.
The ﬁrst observation is w t is a functionof the momentum.
Thus, similar to SGD, if we can compute the momentum privately, every iteratew 1 , . .
. , w T would also be private by post-processing.
The second observation is that m t is anexponentially weighted sum of the gradients and can be computed efﬁciently using tree-aggregation.
To see this, notice that m t in can be written as follows:
[Equation]
where we denote x π qiri as the sample drawn at iteration i
=
q i N + r i .
Now, if we set f [ x,y ] =α (cid:80) yt (cid:48) =
x +1 (1 − α )
y − t (cid:48) ∇ f ( w (cid:48) t , x π qt (cid:48) rt (cid:48) ) , then:m t = (cid:88) [ x [Equation] COMPOSE ([1 ,t ])
[Equation]
Thus, if we consider f [ x,y ] as the value of the node [ x, y ] in the tree, we can compute m t using atmost O (log t ) nodes from the tree.
We will also accumulate the noise in the same way.
Assuming ζ [ x,y ] as the noise we need to add to node [ x, y ] to make f [ x,y ] private, then the noisy ˆ m t is:ˆ m t = (cid:88)
[ x,y ] ∈ COMPOSE ([1 ,t ]) (1 − α ) t − y f
[ x,y ] + (cid:88)
[ x,y ] ∈ COMPOSE ([1 ,t ]) (1 − α ) t − y ζ
[ x,y ] = m t + (cid:88) [ x,y ] ∈ COMPOSE ([1 ,t ]) (1 − α ) t − y ζ
[ x,y ] which is essentially the update of Algorithm 1.
Then, as long as f [ x,y ] has low sensitivity, the noisewe need to add to make m t private would be small since there are only at most O (log t ) intervals in
COMPOSE ([1 , t ]) .
Input: noise parameter σ , momentum parameter α , dataset {+noise m 
+}
← ˆ m 0 ← 0 .
NOISE ←
V ← (min( R, (cid:98) log 2 ( N ) (cid:99) ) + 1) TN + (cid:80)
Rj = (cid:98) log 2 ( N )+1 (cid:99) (cid:98) T 2 j (cid:99) .
for b = 0 , . . .
, R − 1 do for a = 0 , . . .
, (cid:98) T/ 2 b (cid:99) − 1 do
Sample ζ [ a +1: b ] ∼ N (0 , 16 α 2 G 2 σ 2 V I ) .
Z[’ a + 1 , b ’] ← ζ [ a +1: b ] .
end for end for for q = 0 , . . .
, (cid:98) T/N (cid:99) do
Sample a permutation π q 1 , . . . , π qN of { 1 , . . . , N } uniformly at random.
for i = 1 , . . . , min { N, T − qN } do t ← qN + i . Send interval [1 , t ] to Algorithm 3 to get the set S = COMPOSE ([1 , t ]) . for every interval S i ∈ S do
NOISE ← NOISE + (1 − α ) b − ( S i [1])
Z [ S i ] .
end for m t ← (1 − α ) m t − 1 + α ∇ f ( w t , x qπ i ) .
ˆ m t ← m t + NOISE w t
+1 ← w t − ˆ m t (cid:107) ˆ m t (cid:107)
NOISE ←end for end for return w 1 , . .
w T
Theorem 4.
(Privacy guarantee)
Suppose that f ( w, x ) is G − Lipschitz for all w ∈ R d , x ∈ X ,
Algorithm 1 is (cid:0) z, z 2 σ 2 (cid:1) Renyi-differentially private for all z .
Consequentially, if δ ≥ exp( − (cid:15) ) , thenwith σ ≥√ log(1 /δ )
(cid:15) , Algorithm 1 is ( (cid:15), δ ) -differentially private.
Proof sketch of Theorem 4.
Denote t =
q t N + r t for any iteration t for r t ∈ [1 , N ] and q t ≥ 0 .
Wedeﬁne:
[Equation]
To compute the sensitivity of f [ x,y ) , recall that the loss function f is G -Lipschitz.
Further, any givendatapoint x i can contribute at most (cid:100)| x − y | /N (cid:101) gradient terms in the summation deﬁning f
[ x,y ] .
Thusf [ x,y ] has sensitivity ∆ = 2 αG (cid:80) (cid:100)| x − y | /N (cid:101)− 1 j = (1 − α ) jN ≤ 4 αG for α ≥ 1 N (Proposition 13).
Next,we compute the maximum number of interval that contains any datapoint x i for i ∈ { 1 , . . .
, N } .
Given two neighboring datasets D and D’, we use q to indicate the index of the datapoint that is different between the two datasets.
We then denote S [ x,y ] ⊂ { 1 , ..., N } as the set that contains theindices of the datapoints that contribute to node [ x, y ] (essentially which elements of the datasetare relevant to the value of node [ x, y ] ).
For any i ∈ { 1 , . . . , N } , for any b ≤ (cid:98) log 2 ( N ) (cid:99) , there areat most TN different a ∈ { 0 , . . .
, (cid:98) T/ 2 b (cid:99) − 1 } such that q ∈ S [ a 2 b +1 , ( a +1)2 b ] for b ≤ (cid:98) log 2 ( N ) (cid:99) .
Further, for any b > (cid:98) log 2 ( N ) (cid:99) , i ∈ S [ a 2 b +1 , ( a +1)2 b ] = { 1 , . .
. , N } for all a .
Therefore, for any ithere are at most V = min
( R +1 , (cid:98) log 2 ( N )+1 (cid:99) )
TN + (cid:80)
Rj = (cid:98) log 2 ( N )+1 (cid:99) (cid:98) T 2 j (cid:99) sets S i
such that i ∈ S i .
The privacy guarantee then follows via the ordinary composition properties of Renyi differentialprivacy.
The full proof is included in the appendix for completeness.
Before proving our utility bound, we need the following lemma.
Lemma 5.
[Essentially Lemma
[Cutkosky and Mehta [2020]] Deﬁne:
[Equation]
Suppose w 1 , ..., w T is a sequence of iterates deﬁned by w t
+1 =
w t − η ˆ m t (cid:107) ˆ m t (cid:107) for some arbitrarysequence ˆ m 1 , ..., ˆ m T .
Pick ˆ w uniformly at random from w 1 , ..., w T .
Then:
[Equation]
Now we are ready to prove the utility guarantee of Algorithm
[Equation]
Theorem 6.
(Utility guarantee)
Assuming f ( w, x ) is G -Lipschitz, L -smooth for all w ∈ R d , x ∈ X ,and F ( w 1 ) is bounded by R .
Then Algorithm 1 with η =
[Equation]
[Equation]
, and ˆ w that is pick uniformly random from w 1 , . .
, w Tguarantees:
[Equation]
The analysis of Algorithm 1 pretty much follows the analysis of non-private Normalized SGDin Cutkosky and Mehta [2020].
We ﬁrst bound the error between the noisy momentum and thetrue empirical risk gradient E [ (cid:107) ˆ m t [Equation] ∇ F ( [Equation]
(cid:107) ]
≤
[Equation] [Equation] t − m t (cid:107) ] + E [ (cid:107) m t − ∇ F ( w t ) (cid:107) ] .
Theﬁrst term is bounded by the standard deviation of the added noise.
The second term requires amore delicate analysis due to the use of random reshufﬂing (RR) in our algorithm (which is amore common practice than sampling in real-world problems).
This introduces bias to the gradientestimates.
Or formally, if we deﬁne q i = (cid:98) iN (cid:99) and r = i − q i N for every iteration i ∈
[ T ] , wehave E [ ∇ f ( w i , x q i π ri )]
(cid:54) =
∇ F ( w i ) .
Most of the known analysis of RR is done with regular SGD
Mishchenko et al.
[2020], Nguyen et al.
[2019] since the bias becomes an even bigger problem in
SGD with momentum due to the accumulative nature of momentum.
To circumvent this problem werewrite m t as α (cid:80) ti =1 (1 − α ) t − i ∇ f ( w q i N , x q i π ri )
+ O ( ηN )
and use the fact that E [ ∇ f ( w q i , x π qiNri )]
=∇ F ( w q i N ) where w q i is the iterate at the beginning of the epoch and x q i π ri is a datapoint that issampled in that particular epoch (this is true because w q i N is independent of any data sampled inthe epoch).
Then, by applying Jensen’s inequality and the deﬁnition of empirical loss, we have thebound E [ (cid:107) ˆ m t − ∇ F ( w t ) (cid:107) ] ≤ ˜ O (cid:16) √
α + ηN + ηα + α √ dT (cid:15) √ N (cid:17) .
Finally, we can apply Lemma 5 withthe appropriate hyperparameters to get the utility bound in Theorem 6.
Sensitivity Reduced Normalized SGD
Input: Initial Point w 1 , training set x 1 , . . .
, x N , learning rates η , momentum parameter α = 1 − β , cross-batch parameter γ , Lipschitz constant G , smoothness constant L , noise variance σ 2 .
Set V ≤ 2 N = 3 log 2 N, V > 2 N = 4 TN log 2 ( N ) .
Set δ G = 4 G, δ ∆ =
δ r = 2 ηN .
Let q i = (cid:98) iN (cid:99) , r i
= i − q
i N for any iteration i .
for q = 0 , . . .
, (cid:98) T/N (cid:99) do
Sample a permutation π q 1 , . . .
, π qN of { 1 , . . .
, N } uniformly at random.
for r = 0 , . . .
, min { N − 1 , T − qN } do t ← qN + r for k = 0 , . .
, (cid:98) log 2 ( t ) (cid:99) do if t mod 2 k =
then if t ≤ 2 N then
Sample ζ Gt ∼ N (0 , δ 2 G σ 2 V ≤ 2 N I ) else
Sample ζ Gt ∼ N (0 , δ 2 G σ 2 V > 2 N I ) end if
Sample ζ rt ∼ N (0 , δ 2 r σ 2 V > 2 N I ) .
ˆ F G [ t − 2 k +1 ,t ] = (cid:80) ti =
t − 2 k +1 (1 − α ) t − i ∇ f ( w q i N , x
π q i ri )
+ ζ G t
ˆ F r [ t − 2 k
+1 ,t ] = (cid:80) ti =
t − 2 k +1 (1 − α ) t − i ( ∇ f ( w i , x π qiri )
− ∇ f ( w q i N , x π qiri ))
+ ζ rt if t − 2 k + 1 > N then
Sample ζ ∆ t ∼ N (0 , δ 2∆ σ 2 V > 2 N I ) .
ˆ F ∆[ t − 2 k +1 ,t ] = (cid:80) ti =
t − 2 k +1 ( [Equation]
Compute ˆ m t using the reconstruction algorithm (Algorithm 4) [Equation]
In this section, we will devise an algorithm that has an asymptotically tighter bound than ˜ O (cid:16) d 1 / 4 √ (cid:15)T (cid:17) ,which is the best known bound for ( (cid:15), δ ) − DP non-convex ERM.
One natural direction is to use abetter algorithm for non-convex optimization than SGD.
It is known that in non-convex optimizationoperating on smooth losses, the optimal rate for SGD is O (1 /T 1 / 4 ) Arjevani et al.
[2019] while moreadvanced methods that use variance reduction Fang et al.
[2018], Zhou et al.
[2018], Tran-Dinh et al.[2019] or Hessian-vectors product Arjevani et al.
[2020], Tran and Cutkosky [2021] can achievethe rate
O
(1 /T 1 / 3 ) .
Thus, one could hope that these methods can improve the bound of privatenon-convex ERM.
Unfortunately, this does not seem to be the case in reality.
For example, Wanget al.
[2019b] proposes a variant of the variance-reduction based stochastic recursive momentum(STORM) Cutkosky and Orabona [2019] called DP-SRM but their algorithm shows little asymptoticimprovement compared to previous known bound.
DP-SRM improves the error bound by a factor oflog( n/δ ) but this seems to be the result of sampling Abadi et al.
[2016] rather than variance reduction.
We would like to explore an alternative direction to improve the error bound.
Instead of using a betternon-convex optimization methods, we will reuse the Normalized SGD in section 3 but with a tighterprivacy analysis that requires us to add less noise than ˜ O (cid:16) √ T (cid:15) √ N (cid:17) .
The motivation for our methods comes from full-batch SGD.
Consider the following full-batchnormalized gradient descent update: w t +1 =
w t − η g t (cid:107) g t (cid:107) where g t = ∇ F ( w t ) .
Notice that (fort > 1 ) g t =
g t − 1 + ∇ F ( w t ) − ∇ F ( w t − 1 ) .
Now, given g t − 1 , the sensitivity of g t can be bounded bythe sensitivity of ∇ F ( w t ) − ∇ F ( w t − 1 ) , which in turn is only O ( η/N ) due to smoothness and thefact that (cid:107) w t − w t − 1 (cid:107) =
η .
This is a lot smaller than the naive bound of O (1 /N )
since η (cid:28) 1 .
Wecan iterate the above idea to write g t = ∇ F ( w 1 ) + (cid:80) ti =2 ∇ F ( w i ) − ∇ F ( w i − 1 ) .
In this way, each g t is a partial sum of terms with sensitivity O
( η/N ) , which can be be privately estimated with anerror of ˜ O
( η/N(cid:15) )
using tree-aggregation.
The main intuition here is that except for the ﬁrst iterate,each iterate is close to the previous iterate due to smoothness - that is, ∇ F ( w t ) − ∇ F ( w t − 1 ) haslow sensitivity.
In fact, we can further control the sensitivity of g t by expressing g t
=
(1 − γ )( g t − 1 +∇ F ( w t ) − ∇ F ( w t − 1 ))
+ γ ∇ F ( w t ) for some parameter γ < 1 .
Now, with appropriate value of γ ,g t can potentially have even lower sensitivity compared to the case γ = 0 that we discuss above.
Moving beyond the full-batch case, we would like to replace the full-batch gradients ∇ F ( w t ) in thisargument with momentum estimates m t .
Unfortunately, it is not the case that m t − m t − 1 has lowsensitivity.
Instead, notice that the distance between two iterates evaluated on the same sample of twoconsecutive epochs will be at most ηN (cid:28) 1 for appropriate η , and we can show that the sensitivityof m t − m t − N scales by this same distance factor.
Our method described in Algorithm 2 follows thisobservation to compute low sensitivity momentum queries.
To see this, let us rewrite the momentumm t in Algorithm 1:
[Equation]
In order to exploit this decomposition, we deﬁne G [ a,b ] = (cid:80) bt = a (1 − α ) b − t ∇ f ( w q t N , x π qtrt ) ,r [ a,b ] = (cid:80) bt = a (1 − α ) b − t ( ∇ f ( w t , x π qtrt )
− ∇ f ( w q t N , x π qtrt )) and ∆ [ a,b ] = (cid:80) bt = a (1 −α ) b − t ( ∇ f ( w q t N , x π qtrt ) − ∇ f ( w ( q t − 1) N , x π qtrt )) .
Then, (1) states:
[Equation]
We can further reﬁne this expansion by observing (Lemma 17):
[Equation]
Thus, we will estimate G [ a,b ] , r [ a,b ] and ∆ [ a,b ] for all intervals [ a, b ] using tree aggregation, and then 234 recombine these estimates using (2) and (3) to estimate m t .
The critical observation is that eachterm of the sums for r [ a,b ] and ∆ [ a,b ] have sensitivity at most O ( ηN ) , because the normalized updateimplies (cid:107) w t − w t (cid:48) (cid:107) ≤ ηN for all | t − t (cid:48) | ≤ N , while the O (1) sensitivity terms in G [ a,b ] are scaleddown by γ .
Morally speaking, we can think of the ﬁrst two terms in Eq.
3 as the ﬁrst iterate in the full-batchexample with some exponentially weighted factors and the third term as the (1 − γ )( ∇ F ( w t ) −∇ F ( w t − 1 ))
+ γ ∇ F ( w t ) part.
Overall, if we set γ = ηN , the noise we need to add to make m tprivate is ˜ O (cid:16) α(cid:15) + α √ ηT(cid:15)
+ αηN √ T (cid:15) √ N (cid:17) , which is smaller than the ˜ O ( α √ T (cid:15) √ N ) noise that we add in Section3.
Formally, we have the following error bound for the momentum.
Lemma 7.
Suppose α ≥ 1 N and γ ≤ 12 , using the update of Algorithm 2, we have:
[Equation]
To prove Lemma 7, we sum up the variance of the Gaussian noise needed to add to make every termin Eq.3 and ˆ r t private.
Then E [ (cid:107) ˆ m t − m t (cid:107) ] ≤ √ d V AR where V AR is the total noise variance and dis the dimension of the noise vector.
Theorem 8.
(Privacy guarantee)
Suppose α ≥ 1 /N and f is G -Lipschitz and L -smooth.
Then
Algorithm 2 is ( z, 3 z/ 2 σ 2 ) Renyi differentially private for all z .
With σ ≥√ log(1 /δ )
(cid:15) , Algorithm 2 is( (cid:15), δ ) -DP
The proof of Theorem 8 follows the proof of tree-aggregated momentum with Renyi differentialprivacy as in section 3.
We defer the proof to the appendix.
Now we have the utility bound for Algorithm 2.
Theorem 9.
(Utility guarantee)
Assuming f ( w, x ) is G -Lipschitz, L -smooth for all w ∈ R d , x ∈ X ,and F ( w 1 ) is bounded by R .
Then Algorithm 2 with γ = ηN, η = 1 √ NT , α = N 3 / 4 (cid:15) T 3 /
[Table]
√ d , T = N 7 / 3 (cid:15) 4 /
[Table]
d 2 /,(cid:15)
≤ T 3 / 4 √ d N 3 / 4 , and ˆ w that is picked uniformly at random from w 1 , ..., w T guarantees:
[Equation]
where K = 16 log 2 T log(1 /δ ) .
The analysis of Algorithm 2 is similar to the analysis of Algorithm 1.
We still bound the error betweenthe noisy momentum and the true empirical risk gradient as E [ (cid:107) ˆ m t − ∇ F ( w t ) (cid:107) ] ≤ E [ (cid:107) ˆ m t − m t (cid:107) ]+ 259 E
[ (cid:107) m t − ∇
F
w t ) (cid:107) ] .
E [ (cid:107) m t − ∇ F ( w t ) (cid:107) ] is the same as in Theorem 6 but now, the noise bound
E [ (cid:107) ˆ m t − m t (cid:107) ] is only ˜ O (cid:16) α √ d (cid:15) + α √ ηdT(cid:15) + αηN √ dT (cid:15) √ N (cid:17) instead of ˜ O (cid:16) α √ dT (cid:15) √ N (cid:17) due to our sensitivityreduced analysis.
Then we can apply Lemma 5 to get our ﬁnal utility guarantee.
The full proof isprovided in the appendix.
Conclusions
In this paper, we present two new private algorithms for non-convex ERM.
Both algorithms arevariants of Normalized SGD with momentum and both utilizes tree-aggregation method to privatelycompute the momentum.
Our ﬁrst algorithm overcomes the large batch size and privacy ampliﬁcationtechniques requirements in previous works while still achieving the state-of-the-art asymptotic bound˜ O (cid:16) d 1 / 4 √ (cid:15)N (cid:17) .
Our second algorithm uses the insight from full-batch SGD operating on smooth lossesto reduce the sensitivity of the momentum.
This allows the algorithm to achieve the utility bound˜ O (cid:16) d 1 / 3 ( (cid:15)N ) 2 / 3 (cid:17) , which, to our knowledge, is the best known bound for private non-covex ERM.
Limitations: There are several limitations that one could further explore to improve the results ofthis paper.
The most natural direction is to incorporate privacy ampliﬁcation by shufﬂing into bothalgorithms.
Based on the current analysis, it is not obvious that we can apply privacy ampliﬁcationimmediately to tree-aggregated momentum.
However, it is possible that there are some cleverways that one can come up to make the analysis work.
It has been shown in previous work thatprivacy ampliﬁcation by shufﬂing does help us achieve the ˜ O (cid:16) d 1 / 4 √ (cid:15)N (cid:17) utility bound.
Therefore, itis our hope that by combining privacy ampliﬁcation with our sensitivity reduced algorithm, wecan achieve an even better utility bound than ˜ O (cid:16) d 1 / 3 ( (cid:15)N ) 2 / 3 (cid:17) .
Furthermore, even though our ﬁrstalgorithm has comparable run time to previous works, the second algorithm has slightly less ideal runtime.
The current algorithm requires ˜ O (cid:16) N 7 / 3 (cid:15) 4 / 3 d 2 /(cid:17) gradient evaluations but has the total run timeof ˜ O
( N 11 / 3 /d 2 / 3 ) due to the extra run time from the compose and reconstruction algorithms.
Wewould like to reduce this to quadratic run time.
We will leave these for future research.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] S • Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]Please do not modify the questions and only use the provided macros for your answers.
Note that theChecklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.1.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No] this is atheoretical paper.
It has no potential negative societal impacts
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A](b)
Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[N/A](c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A](d)
Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
your work uses existing assets, did you cite the creators
[N A
Did you mention the license of the assets?
[N A]
Did you include any new assets either in the supplemental material or as a URL?
[N/A]
Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N A
you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N
If you used crowdsourcing or conducted research with human subjects...
you include the full text of instructions given to participants and screenshots, if
applicable
[N
any potential participant risks with links to Institutional Review
approvals, applicable
N you
the estimated paid participants the total amount
spent on participant compensation