Momentum Aggregation for Private Non-convex ERM
Abstract
We introduce new algorithms and convergence guarantees for privacy-preserving non-convex Empirical Risk Minimization (ERM) on smooth d -dimensional objectives.
We develop an improved sensitivity analysis of stochastic gradient descent on smooth objectives that exploits the recurrence of examples in different epochs.
By combining this new approach with recent analysis of momentum with private aggregation techniques, we provide an ( (cid:15), δ ) -differential
private algorithm that ﬁnds a gradient of norm ˜ O (cid:16) d 1 / 3 ( (cid:15)N ) 2 / 3 (cid:17) in O (cid:16) N 7 / 3 (cid:15) 4 / 3 d 2 /(cid:17) gradient evaluations, improving the previous best gradient bound of ˜ O (cid:16) d 1 / 4 √ (cid:15)N (cid:17) .
Introduction
In recent years, statistical machine learning models have been deployed in many domains such as health care, education, criminal justice or social studies [Chen et al., 2021,
He et al., 2019, Jiang et al., 2017].
However, the release of statistical estimates based on these sensitive data comes with the risk of leaking personal information of individuals in the original dataset.
One naive solution for this problem is to remove all the identifying information such as names, races or social security numbers.
Unfortunately, this is usually not enough to preserve privacy.
It has been shown in various works that an adversary can take advantages of structural properties of the rest of the dataset to reconstruct information about certain individuals [Backstrom et al., 2007, Dinur and Nissim, 2003].
Thus we would need a stronger privacy-preserving mechanism.
Over the past couple of decades, differential privacy [Dwork et al., 2006] has emerged as the dominant privacy notion for machine learning problems.
Deﬁnition 1 (Differential Privacy [Dwork and Roth, 2014]) .
A randomized algorithm M : X N (cid:55)→ R d satisﬁes ( (cid:15), δ ) − differential privacy (( (cid:15), δ )-DP) if for any two data sets D, D (cid:48) ∈
X N differing by at most one element and any event E ⊆ R d , it holds that:
[Equation]
Roughly speaking, differential privacy guarantees that the outputs of two neighboring datasets (datasets that differ in at most one datapoint) are almost the same with high probability, thus preventing the adversary from identifying any individual’s data.
In this paper, we are interested in designing ( (cid:15), δ ) − private algorithms for non-convex empirical risk minimization (ERM) problems.
In ERM problems, given N i.i.d samples x 1 , ..., x N ∈ X from some unknown distribution P , the goal is to ﬁnd w ∈ R d such that w minimizes the empirical loss deﬁned as follows: [Equation]
[Equation]
where f : R d × X (cid:55)→ R is the loss function associated with the learning problem.
This is a setting that commonly arises in modern machine learning problem.
For example, in image classiﬁcation problems, the data point x would be a tuple of (image, label), w denotes the parameters of our model, and f ( w, x ) represents composing the model predictions with some loss function such as cross-entropy.
We are interested in ﬁnding a critical point, or a point such that the norm of the empirical gradient (cid:107)∇ F ( w ) (cid:107) is as small as possible.
Further, we want all the outputs w 1 , w 2 , ..., w T to be differentially private with respect to the N training samples.
Private ERM has been well studied in the convex settings.
The approaches in this line of work can be classiﬁed into three main categories: output perturbation [Dwork et al., 2006, Chaudhuri et al., 2011, Zhang et al., 2017, Wu et al., 2017], objective perturbation [Chaudhuri et al., 2011,
Kifer et al., 2012, Iyengar et al., 2019, Talwar et al., 2014], and gradient perturbation [Bassily et al., 2014, Wang et al., 2017, Jayaraman et al., 2018, Wang et al., 2018].
All of these approaches have been shown to achieve the asymptotically optimal bound ˜ O (cid:16) √ d (cid:15)N (cid:17) for smooth convex loss (with output perturbation requiring strong convexity to get the optimal bound) in (near) linear time.
On the other hand, the literature on private non-convex ERM is nowhere as comprehensive.
The ﬁrst theoretical bound in private non-convex ERM is from [Zhang et al., 2017].
They propose an algorithm called Random Round Private Stochastic Gradient Descent (RRSGD) which is inspired by the results from [Bassily et al., 2014, Ghadimi and Lan, 2013].
RRSGD is able to guarantee the utility bound of O (cid:16) ( d log( n/δ )log(
[Table]
(cid:17) .
However, RRSGD takes O (cid:0) N 2 d (cid:1) gradient computations to achieve this, which can be troublesome for high-dimensional problem.
[Wang et al., 2018] then improves upon this utility bound by a factor of O (cid:0) (log( n/δ )) 1 / 4 (cid:1) .
They achieve this rate by using full-batch gradient descent which is not a common practice in non-private machine learning in which very large batch sizes actually require careful work to make training efﬁcient.
Recently, [Wang et al., 2019b] tackles both runtime and utility issues by introducing a private version of the Stochastic Recursive
Momentum (DP-SRM)
[Cutkosky and Orabona, 2019].
By appealing to variance reduction as well as privacy ampliﬁcation by subsampling [Balle et al., 2018, Abadi et al., 2016], DP-SRM achieve the bound O (cid:16) ( d log(1 /δ )) 1 / 4 √ (cid:15)N
[Equation]
in O (cid:16)
[Equation]
gradient complexity.
However, DR-SRM still requires the batch size to be O (cid:16) √ (cid:15)Nd 1 / 4 (cid:17) for the analysis to work.
Finally, although our focus in this paper is the ERM problem, there are also various works on private stochastic non-convex/convex optimization [Bassily et al., 2019, 2021a,b, Feldman et al., 2018, 2020, Zhou et al., 2020, Asi et al., 2021, Wang et al., 2019a, Kulkarni et al., 2021].
Contributions.
We ﬁrst provide the analysis for the private version of Normalized SGD (DP-NSGD) [Cutkosky and Mehta, 2020] for unconstrained non-convex ERM.
By using the tree-aggregation technique [Chan et al., 2011, Dwork et al., 2010] to compute the momentum privately, we can ensure the privacy guarantee while adding noise of only ˜ O (cid:16) √ T (cid:15) √ N (cid:17) (where T is the total number of iterations).
This allows us to achieve the same asymptotic bound ˜ O (cid:16) d 1 / 4 √ (cid:15)N (cid:17) on the expectation of the gradient as [Zhang et al., 2017, Wang et al., 2018, 2019b] without appealing to privacy ampliﬁcation techniques which is usually required for private SGD to have a good utility guarantee [Abadi et al., 2016].
DP-NSGD also does not require a large batch size as in [Wang et al., 2018, 2019b]; it has utility guarantee for any batch size.
This tree-aggregation technique is morally similar to the approach in [Kairouz et al., 2021, Guha Thakurta and Smith, 2013] for online learning.
However, unlike in [Kairouz et al., 2021, Guha Thakurta and Smith, 2013], we do not restrict our loss function to be convex and we will also extend tree-aggregation technique to SGD with momentum.
Further, we provide a new variant of Normalized SGD that takes advantages of the fact that the gradients of the nearby iterates are close to each other due to smoothness.
This new algorithm is able to guarantee an error of ˜ O (cid:16) d 1 / 3 ( (cid:15)N ) 2 / 3 (cid:17) in O (cid:16) N 7 / 3 (cid:15) 4 / 3 d 2 /(cid:17) gradient computations which, to our knowledge, is the best known rate for private non-convex ERM.
Organization.
The rest of the paper is organized as follows.
In section 2, we deﬁne our problem of interest and the assumptions that we make on the problem settings.
We also provide some background on Differential Privacy as well as some high-level intuition on tree-aggregation technique.
We then formally describe our ﬁrst private variant of Normalized SGD in section 3 and discuss its privacy guarantee and theoretical utility bound.
In section 4, we introduce a for Normalized SGD that allows us to improve upon the utility bound in section 3.
Finally, we conclude with a discussion in section 5.
Preliminaries
Assumptions.
We deﬁne our loss function as f ( w, x ) : R d × X (cid:55)→ R where x ∈ X is some sample from the dataset.
Throughout the paper, we make the following assumptions on f ( w, x ) .
Deﬁne a differentiable function f ( . ) :
R d (cid:55)→ R to be G -Lipschitz iff (cid:107)∇ f ( y ) (cid:107) ≤ G for all y ∈ R d , and to be
L -smooth iff (cid:107)∇ f ( y 1 ) − ∇ f ( y 2 ) (cid:107) ≤ L (cid:107) y 1 − y 2 (cid:107) where (cid:107) .
(cid:107) is the standard 2-norm unless speciﬁed otherwise.
We assume that f ( w, x ) is differentiable, G -Lipschitz, and L -smooth with probability 1.
We also assume that provided the initial point
w 1 ∈ R
d there exists some upper bound R of F ( w 1 ) or formally: sup w ∈ R d F ( w 1 ) − F ( w ) ≤ R .
Differential Privacy.
Let D be a dataset containing N datapoints.
Then two datasets D, D (cid:48) are said to be neighbors if (cid:107) D − D (cid:48) (cid:107) 1 = 1 or in other words, the two datasets differ at exactly one entry.
Now we have the deﬁnition for sensitivity:
Deﬁnition 2.
( L 2 -sensitivity)
The L 2 -sensitivity of a function f ( . ) :
X N (cid:55)→ R d is deﬁned as follows:
[Equation]
In this work, we also make use of Renyi Differential Privacy (RDP)
[Mironov, 2017].
RDP is a relaxation of ( (cid:15), δ ) − DP and can be used to improve the utility bound and do composition efﬁciently while still guaranteeing privacy.
Deﬁnition 3.
(( α, (cid:15) )
-RDP
[Mironov, 2017])
A randomized algorithm f : X N (cid:55)→ R d is said to have (cid:15) − RDP of order
α if for any neighboring datasets D, D (cid:48) it holds that:
[Equation]
where D {+α ( P || Q ) (cid:44) 
+}
α − 1 log E x ∼ Q (cid:16) P ( x ) Q ( x ) (cid:17) α g [1:8] g [1:4] g [1:2] g [1:1] g [2:2] g [3:4] g [3:3] g [4:4] g [5:8] g [5:6] g [5:5] g [6:6] g [7:8] g [7:7] g [8:8]
Tree-aggregation.
Consider the problem of releasing private partial sum s
[ y,z ] = (cid:80) zi
= y g
i given a stream of vectors g 1 , . . . , g t .
Assuming (cid:107) g
i (cid:107) ≤ G ∀ i ∈ [ t ] , then the simplest solution for this problem is to add Gaussian noise with standard deviation ˜ O (1 /(cid:15) ) to each g
i so that each g i is ( (cid:15), δ ) − DP.
Then error from the noise added to any partial sum will be bounded by O ( √ T/(cid:15) ) .
However, this problem can be solved much more efﬁciently using tree-aggregation [Chan et al., 2011, Dwork et al., 2010].
In tree-aggregation, we consider a complete binary tree where each of the leaf nodes is labeled with g i and each internal node in the tree is the sum of its children.
The tree is illustrated in Fig.1.
It is clear from Fig.1 that each g i only affects its ancestors, or at most (cid:100) log t (cid:101) + 1 nodes.
Thus, by adding
Gaussian noise with standard deviation O (log t/(cid:15) ) to every node, the complete tree will be ( (cid:15), δ ) − DP.
Furthermore, since we only need at most log t nodes to compute any partial sum s [ y,z ] , the noise added to every s [ y,z ] would be bounded by O (log 1 . 5 t/(cid:15) ) , which is a lot less than the noise that we would add using advanced composition.
To get a general idea on how we can use this tree-aggregation technique to design private algorithms, let us analyse the update of SGD with initial point w 1 = 0 :
[Equation]
where ∇ f ( w t , x t ) is the gradient of the loss function evaluated on the current iterate w t and some sample x t .
Notice that every iterate w t +1 would be a function of the sum of the gradients s t = (cid:80) ti =1 ∇ f ( w t , x t ) up to iteration t .
Thus, if we can use tree-aggregation to privately compute every partial sum s [ y,z ] = (cid:80) zi
= y ∇ f ( w i , x i ) , we can the use these partial sums to compute every iterate w t , and w t is also private by post-processing [Dwork and Roth, 2014].
Similarly, we can also use this aggregation technique for SGD with momentum where we replace ∇ f ( w t , x t ) with the momentum m t .
This method of computing momentum would be the main ingredient for our private algorithms.
A more detailed discussions are provided in section 3 and 4.
Tree-aggregated momentum has been brieﬂy mentioned in the momentum variant of Differentially Private Follow-The-Regularized-Leader (DP-FTRL) in [Kairouz et al., 2021].
However, their algorithm does not directly compose the momentum using the tree-aggregation and they also do not provide any formal guarantee for the algorithm.
Private Normalized SGD
Before we go into the details of our private algorithm, ﬁrst let us set up some notations that are frequently used in the rest of the paper.
We deﬁne the set of subsets I as follows:
[Equation]
a 2 b + 1 , ( a + 1)2 b ] | a, b ∈ Z , [ a 2 b + 1 , ( a + 1)2 b ] ⊂ { 1 , . . .
, T }} Each interval in I corresponds to a node in the tree described above (In Fig.1, the set
I is { [1 , 1] , [2 , 2] , [1 , 2] , . . . } ).
Thus, the set of intervals I is essentially a different way to store the binary tree for tree-aggregation mechanism.
Then, for every interval [ a, b ] , there exists O (log T ) disjoint subsets
[ y 1 , z 1 ] , ..., [ y n , z n ] ∈ I such that [ a, b ] = (cid:83) ni
=
[ y i , z i ] Daniely et al.
We denote the set of such subsets as COMPOSE ( a, b ) , which can be computed using Algorithm 3.
Now we can proceed to the analysis of the private Normalized SGD algorithm described in Algorithm 1.
The main idea of Algorithm 1 is based on two observations.
The ﬁrst observation is w t is a function of the momentum.
Thus, similar to SGD, if we can compute the momentum privately, every iterate w 1 , . .
. , w T would also be private by post-processing.
The second observation is that m t is an exponentially weighted sum of the gradients and can be computed efﬁciently using tree-aggregation.
To see this, notice that the momentum m t can be written as follows:
[Equation]
where we denote x π qir i as the sample drawn at iteration i
=
q i N + r i .
Now, if we set f [ y,z ] = α (cid:80) zt (cid:48) =
y (1 − α )
z − t (cid:48) ∇ f ( w t (cid:48) , x π qt (cid:48) rt (cid:48) ) , then: m t = (cid:88)
[ y,z ] ∈ COMPOSE (1 ,t ) (1 − α ) t − z f
[ y,z ]
Thus, if we consider f [ y,z ] as the value of the node [ y, z ] in the tree, we can compute m t using at most
O (log t ) nodes from the tree.
We will also accumulate the noise in the same way.
Assuming ζ [ y,z ] as the noise we need to add to node [ y, z ] to make f [ y,z ] private, then the noisy ˆ m t is:
[Equation]
which is essentially the update of Algorithm 1.
Then, as long as f [ y,z ] has low sensitivity, the noise we need to add to make m t private would be small since there are only at most O (log t ) intervals in
COMPOSE (1 , t ) .
Algorithm 1 Differentially private Normalized SGD with momentum (DP-NSGD)
Input: noise parameter σ , momentum parameter α = 1 − β , dataset ( x 1 , . . .
, x N ) , tree depth R ,
Lipschitz constant G , loss function f ( w, x ) , noise map Z. m 0 ← ˆ m 0 ← 0 .
NOISE ←
V ← (min( R, (cid:98) log 2 ( N ) (cid:99) ) + 1) TN + (cid:80) R j = (cid:98) log 2 ( N )+1 (cid:99) (cid:98) T 2 j (cid:99) .
for b = 0 , . . .
, R − 1 do for a = 0 , . . .
, (cid:98) T/ 2 b (cid:99) − 1 do
Sample ζ [ a 2 b +1 , ( a +1)2 b ] ∼ N (0 , 16 α 2 G 2 σ 2 V I ) .
Z[ a 2 b + 1 , ( a + 1)2 b ] ← ζ [ a 2 b +1 , ( a +1)2 b ] .
end for end for for q = 0 , . . .
, (cid:98) T/N (cid:99) do
Sample a permutation π q 1 , . . . , π qN of { 1 , . . . , N } uniformly at random.
for i = 1 , . . .
, min { N, T − qN } do t ← qN + i . Send interval [1 , t ] to Algorithm 3 to get the set S = COMPOSE (1 , t ) . for every interval S i ∈ S do
NOISE ← NOISE + (1 − α ) t − ( S i [1])
Z [ S i ] ( S i [1] indicates the end index of each interval
S i ) end for m t ← (1 − α ) m t − 1 + α ∇ f ( w t , x qπ i ) .
ˆ m t ← m t + NOISE w t
+1 ← w t − η ˆ m t (cid:107) ˆ m
[Equation]
(cid:107)
NOISE ←end for end for return w 1 , . .
w T
Theorem 4.
(Privacy guarantee)
Suppose that f ( w, x ) is G − Lipschitz for all w ∈ R d , x ∈ X ,
Algorithm 1 is (cid:0) z, z 2 σ 2 (cid:1) Renyi-differentially private for all z .
Consequentially, if δ ≥ exp( − (cid:15) ) , then with σ ≥√ log(1 /δ )
(cid:15) , Algorithm 1 is ( (cid:15), δ ) -differentially private.
Proof sketch of Theorem 4.
Denote t =
q t N + r t for any iteration t for r t ∈ [1 , N ] and q t ≥ 0 .
We deﬁne:
[Equation]
To compute the sensitivity of f [ y,z ] , recall that the loss function f is G -Lipschitz.
Further, any given datapoint x i can contribute at most (cid:100)| y − z | /N (cid:101) gradient terms in the summation deﬁning f
[ y,z ] .
Thus f [ y,z ] has sensitivity ∆ = 2 αG (cid:80) (cid:100)| y − z | /N (cid:101)− 1 j = (1 − α ) jN ≤ 4 αG for α ≥ 1 N (Proposition 13).
Next, we compute the maximum number of interval that contains any datapoint x i for i ∈ { 1 , . . .
, N } .
Given two neighboring datasets D and D’, we use s to indicate the index of the datapoint that is different between the two datasets.
We then denote S
[ y,z ] ⊂ { 1 , ..., N } as the set that contains the indices of the datapoints that contribute to node [ y, z ] (essentially which elements of the dataset are relevant to the value of node [ y, z ] ).
For any i ∈ { 1 , . . . , N } , for any b ≤ (cid:98) log 2 ( N ) (cid:99) , there are at most TN different a ∈ { 0 , . . .
, (cid:98) T/ 2 b (cid:99) − 1 } such that s ∈ S
[ a 2 b +1 , ( a +1)2 b ] for b ≤ (cid:98) log 2 ( N ) (cid:99) .
Further, for any b > (cid:98) log 2 ( N ) (cid:99) , i ∈ S [ a 2 b +1 , ( a +1)2 b ] = { 1 , . .
. , N } for all a .
Therefore, for any i there are at most V = min
( R +1 , (cid:98) log 2 ( N )+1 (cid:99) )
TN + (cid:80)
Rj = (cid:98) log 2 ( N )+1 (cid:99) (cid:98) T 2 j (cid:99) sets S i
such that i ∈ S i .
The privacy guarantee then follows via the ordinary composition properties of Renyi differential privacy.
The full proof is included in the appendix.
Before proving our utility bound, we need the following lemma.
Lemma 5.
[Essentially Lemma
[Cutkosky and Mehta [2020]] Deﬁne:
[Equation]
Suppose w 1 , ..., w T is a sequence of iterates deﬁned by w t
+1 =
w t − η ˆ m t (cid:107) ˆ m t (cid:107) for some arbitrary sequence ˆ m 1 , ..., ˆ m T .
Pick ˆ w uniformly at random from w 1 , ..., w T .
Then:
[Equation]
Now we are ready to prove the utility guarantee of Algorithm
[Equation]
Theorem 6.
(Utility guarantee)
Assuming f ( w, x ) is G -Lipschitz, L -smooth for all w ∈ R d , x ∈ X , and F ( w 1 ) is bounded by R .
Then Algorithm 1 with η =
[Equation]
[Equation]
N , and ˆ w that is pick uniformly random from w 1 , . .
. , w T guarantees:
[Equation]
The analysis of Algorithm 1 pretty much follows the analysis of non-private Normalized SGD in Cutkosky and Mehta [2020].
We ﬁrst bound the error between the noisy momentum and the true empirical risk gradient E [ (cid:107) ˆ m t − ∇ F ( w t ) (cid:107) ] ≤ E [ (cid:107) ˆ m t − m t (cid:107) ]
+ E [ (cid:107) m t − ∇ F ( w t ) (cid:107) ] .
The ﬁrst term is bounded by the standard deviation of the added noise.
The second term requires a more delicate analysis due to the use of random reshufﬂing (RR) in our algorithm (which is a more common practice than sampling in real-world problems).
This introduces bias to the gradient estimates.
Or formally, if we deﬁne q i = (cid:98) iN (cid:99) and r = i − q i N for every iteration i ∈
[ T ] , we have E [ ∇ f ( w i , x q i π ri )] (cid:54) =
∇ F ( w i ) .
Most of the known analysis of RR is done with regular SGD
Mishchenko et al.
[2020], Nguyen et al.
[2019] since the bias becomes an even bigger problem in
SGD with momentum due to the accumulative nature of momentum.
To circumvent this problem we rewrite m t as α (cid:80) ti =1 (1 − α ) t − i ∇ f ( w q i N , x q i π ri )
+ O ( ηN )
and use the fact that E [ ∇ f ( w q i , x π qiNri )]
= ∇ F ( w q i N ) where w q i is the iterate at the beginning of the epoch and x q i π ri is a datapoint that is sampled in that particular epoch (this is true because w q i N is independent of any data sampled in the epoch).
Then, by applying Jensen’s inequality and the deﬁnition of empirical loss, we have the bound E [ (cid:107) ˆ m t − ∇ F ( w t ) (cid:107) ] ≤ ˜ O (cid:16) √ α + [Equation]
+ [Equation]
Finally, we can apply Lemma 5 with the appropriate hyperparameters to get the utility bound in Theorem 6.
Input: Initial Point w 1 , training set x 1 , . . .
, x N , learning rates η , momentum parameter α = 1 − β , cross-batch parameter γ , Lipschitz constant G , smoothness constant L , noise variance σ 2 .
Set V ≤ 2 N = 3 log 2 N, V > 2 N = 4 TN log 2 ( N ) .
Set δ G = 4 G, δ ∆ =
δ r = 2 ηN .
for q = 0 , . . .
, (cid:98) T/N (cid:99) do
Sample a permutation π q 1 , . . .
, π qN of { 1 , . . .
, N } uniformly at random.
for r = 0 , . . .
, min { N − 1 , T − qN } do t ← qN + r t for k = 0 , . .
, (cid:98) log 2 ( t ) (cid:99) do if t mod 2 k =
then if t ≤ 2 N then
Sample ζ Gt ∼ N (0 , δ 2 G σ 2 V ≤ 2 N I ) else
Sample ζ Gt ∼ N (0 , δ 2 G σ 2 V > 2 N I ) end if
Sample ζ rt ∼ N (0 , δ 2 r σ 2 V > 2 N I ) .
ˆ F
[Equation]
ti
[Equation] [Equation]
[Equation]
w
[Equation]
, x
[Equation] [Equation]
ˆ F
[Equation]
ti =
[Equation] [Equation]
+1 (1 − α )
[Equation]
− i (
[Equation]
w
[Equation]
, x
[Equation] [Equation]
w
[Equation]
, x
[Equation]
rt if t − 2 k + 1 > N then
Sample ζ ∆ t ∼ N (0 , δ 2∆ σ 2 V > 2 N I ) .
ˆ F ∆[ t − 2 k +1 ,t ] = (cid:80) ti =
t − 2 k +1 (1 − α ) t − i ( ∇ f ( w qN , x π qri )
− ∇ f ( w ( q − 1) N , x
Sensitivity Reduced Normalized SGD
In this section, we will devise an algorithm that has an asymptotically tighter bound than ˜ O (cid:16) d 1 / 4 √ (cid:15)N (cid:17) , which is the best known bound for ( (cid:15), δ ) − DP non-convex ERM.
One natural direction is to use a better algorithm for non-convex optimization than SGD.
It is known that in non-convex optimization operating on smooth losses, the optimal rate for SGD is O (1 /N 1 / 4 ) Arjevani et al.
[2019] while more advanced methods that use variance reduction Fang et al.
[2018], Zhou et al.
[2018], Tran-Dinh et al.
[2019] or Hessian-vectors product Arjevani et al.
[2020], Tran and Cutkosky [2021] can achieve the rate O (1 /N 1 / 3 ) .
Thus, one could hope that these methods can improve the bound of private non-convex ERM.
Unfortunately, this does not seem to be the case in reality.
For example, Wang et al.
[2019b] proposes a variant of the variance-reduction based stochastic recursive momentum (STORM) Cutkosky and Orabona [2019] called DP-SRM but their algorithm shows little asymptotic improvement compared to previous known bound.
DP-SRM improves the error bound by a factor of log( n/δ ) but this seems to be the result of sampling Abadi et al.
[2016] rather than variance reduction.
We would like to explore an alternative direction to improve the error bound.
Instead of using a better non-convex optimization methods, we will reuse the Normalized SGD in section 3 but with a tighter privacy analysis that requires us to add less noise than ˜ O (cid:16) √ T (cid:15) √ N (cid:17) .
The motivation for our methods comes from full-batch SGD.
Consider the following full-batch normalized gradient descent update: w t +1 =
w t − η g t (cid:107) g t (cid:107) where g t = ∇ F ( w t ) .
Notice that (for t > 1 ) g t =
g t − 1 + ∇ F ( w t ) − ∇ F ( w t − 1 ) .
Now, given g t − 1 , the sensitivity of g t can be bounded by the sensitivity of ∇ F ( w t ) − ∇ F ( w t − 1 ) , which in turn is only O ( η/N ) due to smoothness and the fact that (cid:107) w t − w t − 1 (cid:107) =
η .
This is a lot smaller than the naive bound of O (1 /N )
since η (cid:28) 1 .
We can iterate the above idea to write g t = ∇ F ( w 1 ) + (cid:80) ti =2 ∇ F ( w i ) − ∇ F ( w i − 1 ) .
In this way, each g t is a partial sum of terms with sensitivity O
( η/N ) , which can be be privately estimated with an error of ˜ O
( η/N(cid:15) )
using tree-aggregation.
The main intuition here is that except for the ﬁrst iterate, each iterate is close to the previous iterate due to smoothness - that is, ∇ F ( w t ) − ∇ F ( w t − 1 ) has low sensitivity.
In fact, we can further control the sensitivity of g t by expressing g t
=
(1 − γ )( g t − 1 + ∇ F ( w t ) − ∇ F ( w t − 1 ))
+ γ ∇ F ( w t ) for some parameter γ < 1 .
Now, with appropriate value of γ , g t can potentially have even lower sensitivity compared to the case γ = 0 that we discuss above.
Moving beyond the full-batch case, we would like to replace the full-batch gradients ∇ F ( w t ) in this argument with momentum estimates m t .
Unfortunately, it is not the case that m t − m t − 1 has low sensitivity.
Instead, notice that the distance between two iterates evaluated on the same sample at two consecutive epochs will be at most ηN (cid:28) 1 for appropriate η , and we can show that the sensitivity of m t − m t − N scales by this same distance factor.
Our method described in Algorithm 2 follows this observation to compute low sensitivity momentum queries.
To see this, let us rewrite the momentum m t in Algorithm 1:
[Equation]
In order to exploit this decomposition, we deﬁne G [ a,b ] = (cid:80) bt = a (1 − α ) b − t ∇ f ( w q t N , x π qtrt ) , r [ a,b ] = (cid:80) bt = a (1 − α ) b − t ( ∇ f [Equation]
w t , x π qtrt ) − ∇ f ( w q t N , x π qtrt )) and ∆ [ a,b ] = (cid:80) bt = a (1 − α ) b − t ( ∇ f ( w q t N , x π qtrt ) − ∇ f ( w ( q t − 1) N , x π qtrt )) .
Then, (1) states:
[Equation]
Considering t = qN + r , we can further reﬁne this expansion by observing (Lemma 17): G
[Equation]
,t ] [Equation] Thus, in Algorithm 2, we will estimate G [ a,b ] , r [ a,b ] and ∆ [ a,b ] for all intervals [ a, b ] using tree aggregation, and then recombine these estimates using (2) and (3) to estimate m t .
The critical observation is that each term of the sums for r [ a,b ] and ∆ [ a,b ] have sensitivity at most O ( ηN ) , because the normalized update implies (cid:107) w t − w t (cid:48) (cid:107) ≤ ηN for all | t − t (cid:48) | ≤ N , while the O (1) sensitivity terms in G [ a,b ] are scaled down by γ . However, notice that in the ﬁrst two epochs, we don’t have the extra parameter γ that helps us control the noise added to the momentum. Thus, if we just naively add noise based on the maximum sensitivity (which is V > 2 N ), the noise will blow up since it scales with T/N .
To remedy this, we add two different types of noise to G [ t − 2 k +1 ,t ] ( k ∈ { 0 , . . . , (cid:98) log 2 ( t ) (cid:99) ) depends on whether t ≥ 2 N . By dividing the sensitivity into 2 cases, we can make the momentum of the ﬁrst 2 epochs ( (cid:15), δ ) − DP by adding noise of ˜ O
(1 /(cid:15) ) and every other epochs ( (cid:15), δ ) − DP by adding noise of ˜ O
( γ √ T/ √ N(cid:15) ) . Then, the whole procedure would be ( O ( (cid:15) ) , O ( δ )) − DP overall. Morally speaking, we can think of the ﬁrst two terms of the update of Algorithm 2 (Eq. 3) as the ﬁrst iterate in the full-batch example with some exponentially weighted factors and the third term as the (1 − γ )( ∇ F ( w t ) − ∇ F ( w t − 1 ))
+ γ ∇ F ( w t ) part.
Overall, if we set γ = ηN , the noise we need to add to make m t private is ˜ O (cid:16) α(cid:15) + α √ ηT(cid:15)
+ αηN √ T (cid:15) √ N (cid:17) , which is smaller than the ˜ O ( α √ T (cid:15) √ N ) noise that we add in Section 3.
Formally, we have the following error bound for the momentum.
Lemma 7.
Suppose α ≥ 1 N and γ ≤ 12 , using the update of Algorithm 2, we have:
[Equation]
To prove Lemma 7, we sum up the variance of the Gaussian noise needed to add to make every term in Eq.3 and ˆ r t private.
Then E [ (cid:107) ˆ m t − m t (cid:107) ] ≤ √ d V AR where V AR is the total noise variance and d is the dimension of the noise vector.
Theorem 8.
(Privacy guarantee)
Suppose α ≥ 1 /N and f is G -Lipschitz and L -smooth.
Then
Algorithm 2 is ( z, 3 z/ 2 σ 2 ) Renyi differentially private for all z .
With σ ≥√
[Equation] , Algorithm 2 is ( (cid:15), δ ) -DP
The proof of Theorem 8 follows the proof of tree-aggregated momentum with Renyi differential privacy as in section 3.
We defer the proof to the appendix.
Now we have the utility bound for Algorithm 2.
Theorem 9.
(Utility guarantee)
Assuming f ( w, x ) is G -Lipschitz, L -smooth for all w ∈ R d , x ∈ X , and F ( w 1 ) is bounded by R .
Then Algorithm 2 with γ = ηN, η =
[Equation]
(cid:15) ≤ T 3 / 4 √ d N 3 / 4 , and ˆ w that is picked uniformly at random from w 1 , ..., w T guarantees: [Equation] √
N
[Equation]
where K = 16 log 2 T log(1 /δ ) .
The analysis of Algorithm 2 is similar to the analysis of Algorithm 1.
We still bound the error between the noisy momentum and the true empirical risk gradient as E [ (cid:107) ˆ m t − ∇ F ( w t ) (cid:107) ] ≤ E [ (cid:107) ˆ m t − m t (cid:107) ]
+ E [ (cid:107) m t − ∇ F ( w t ) (cid:107) ] .
E [ (cid:107) m t − ∇ F ( w t ) (cid:107) ] is the same as in Theorem 6 but now, the noise bound
E [ (cid:107) ˆ m t − m t (cid:107) ] is only ˜ O (cid:16) α √ d (cid:15) + α √ ηdT(cid:15) + αηN √ dT (cid:15) √ N (cid:17) instead of ˜ O (cid:16) α √ dT (cid:15) √ N (cid:17) due to our sensitivityreduced analysis.
Then we can apply Lemma 5 to get our ﬁnal utility guarantee.
The full proof is provided in the appendix.
Conclusions
In this paper, we present two new private algorithms for non-convex ERM.
Both algorithms are variants of Normalized SGD with momentum and both utilizes tree-aggregation method to privately compute the momentum.
Our ﬁrst algorithm overcomes the large batch size and privacy ampliﬁcation techniques requirements in previous works while still achieving the state-of-the-art asymptotic bound ˜ O (cid:16) d 1 / 4 √ (cid:15)N (cid:17) .
Our second algorithm uses the insight from full-batch SGD operating on smooth losses to reduce the sensitivity of the momentum.
This allows the algorithm to achieve the utility bound ˜ O (cid:16) d 1 / 3 ( (cid:15)N ) 2 / 3 (cid:17) , which, to our knowledge, is the best known bound for private non-covex ERM.
Limitations: There are several limitations that one could further explore to improve the results of this paper.
The most natural direction is to incorporate privacy ampliﬁcation by shufﬂing into both algorithms.
Based on the current analysis, it is not obvious that we can apply privacy ampliﬁcation immediately to tree-aggregated momentum.
However, it is possible that there are some clever ways that one can come up to make the analysis work.
It has been shown in previous work that privacy ampliﬁcation by shufﬂing does help us achieve the ˜ O (cid:16) d 1 / 4 √ (cid:15)N (cid:17) utility bound.
Therefore, it is our hope that by combining privacy ampliﬁcation with our sensitivity reduced algorithm, we can achieve an even better utility bound than ˜ O (cid:16) d 1 / 3 ( (cid:15)N ) 2 / 3 (cid:17) .
Furthermore, even though our ﬁrst algorithm has comparable run time to previous works, the second algorithm has slightly less ideal run time.
The current algorithm requires ˜ O (cid:16) N 7 / 3 (cid:15) 4 / 3 d 2 /(cid:17) gradient evaluations but has the total run time of ˜ O
( N 11 / 3 /d 2 / 3 ) due to the extra run time from the compose and reconstruction algorithms.
We would like to reduce this to quadratic run time.
We will leave these for future research
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] S • Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[No] this is a theoretical paper.
It has no potential negative societal impacts (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]