Original: Gradient Perturbation is Underrated for Differentially Private Convex Optimization
Revised: Gradient Perturbation is Underrated for Differentially Private Convex Optimization

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: Abstract
Revised: Abstract

Original: Gradient perturbation, widely used for diﬀerentially private optimization, injects noise at every iterative update to guarantee diﬀerential privacy.
Revised: Gradient perturbation, widely used for diﬀerentially private optimization, injects noise at every iterative update to guarantee diﬀerential privacy.

Original: Previous work ﬁrst determines the noise level that can satisfy the privacy requirement and then analyzes the utility of noisy gradient updates as in non-private case.
Revised: Previous work ﬁrst determines the noise level that can satisfy the privacy requirement and then analyzes the utility of noisy gradient updates as in non-private case.

Original: In this paper, we explore how the privacy noise aﬀects the optimization property.
Revised: In this paper, we explore how the privacy noise aﬀects the optimization property.

Original: We show that for diﬀerentially private convex optimization, the utility guarantee of both DP-GD and DP-SGD is determined by an expected curvature rather than the minimum curvature.
Revised: We show that for diﬀerentially private convex optimization, the utility guarantee of both DP-GD and DP-SGD is determined by an expected curvature rather than the minimum curvature.

Original: The expected curvature represents the average curvature over the optimization path, which is usually much larger than the minimum curvature and hence can help us achieve a signiﬁcantly improved utility guarantee.
Revised: The expected curvature represents the average curvature over the optimization path, which is usually much larger than the minimum curvature and hence can help us achieve a signiﬁcantly improved utility guarantee.

Original: By using the expected curvature , our theory justiﬁes the advantage of gradient perturbation over other perturbation methods and closes the gap between theory and practice.
Revised: By using the expected curvature , our theory justiﬁes the advantage of gradient perturbation over other perturbation methods and closes the gap between theory and practice.

Original: Extensive experiments on real world datasets corroborate our theoretical ﬁndings.
Revised: Extensive experiments on real world datasets corroborate our theoretical ﬁndings.

Original:  Introduction
Revised:  Introduction

Original: Machine learning has become a powerful tool for many practical applications.
Revised: Machine learning has become a powerful tool for many practical applications.

Original: The training process often needs access to some private dataset, e g., applications in ﬁnancial and medical ﬁelds.
Revised: The training process often needs access to some private dataset, e g., applications in ﬁnancial and medical ﬁelds.

Original: Recent work has shown that the model learned from training data may leak unintended information of individual records (Fredrikson et al., 2015; Wu et al., 2016;
Revised: Recent work has shown that the model learned from training data may leak unintended information of individual records (Fredrikson et al., 2015; Wu et al., 2016;

Original: Shokri et al., 2017;
Revised: Shokri et al., 2017;

Original: Hitaj et al., 2017).
Revised: Hitaj et al., 2017).

Original: It is known that Diﬀerential privacy (DP) (Dwork et al., 2006a;b) is a golden standard for privacy preserving data analysis.
Revised: It is known that Diﬀerential privacy (DP) (Dwork et al., 2006a;b) is a golden standard for privacy preserving data analysis.

Original: It provides provable privacy guarantee by ensuring the inﬂuence of any individual record is negligible.
Revised: It provides provable privacy guarantee by ensuring the inﬂuence of any individual record is negligible.

Original: It has been deployed into real world applications by large-scale corporations and U
Revised: It has been deployed into real world applications by large-scale corporations and U

Original: S. Census Bureau (Erlingsson et al., 2014; McMillan, 2016; Abowd, 2016; Ding et al., 2017).
Revised: S. Census Bureau (Erlingsson et al., 2014; McMillan, 2016; Abowd, 2016; Ding et al., 2017).

Original: We study the fundamental problem when diﬀerential privacy meets machine learning: the diﬀerentially private empirical risk minimization (DP-ERM) problem (Chaudhuri & Monteleoni, 2009; Chaudhuri et al., 2011; Kifer et al., 2012; Bassily et al., 2014; Talwar et al., 2015;Wu et al., 2017; Zhang et al., 2017;
Revised: We study the fundamental problem when diﬀerential privacy meets machine learning: the diﬀerentially private empirical risk minimization (DP-ERM) problem (Chaudhuri & Monteleoni, 2009; Chaudhuri et al., 2011; Kifer et al., 2012; Bassily et al., 2014; Talwar et al., 2015;

Original:  Wang et al., 2017; Smith et al., 2017; Jayaraman et al., 2018; Feldman et al., 2018; Iyengar et al.,  2019).
Revised: Wu et al., 2017; Zhang et al., 2017; Wang et al., 2017; Smith et al., 2017; Jayaraman et al., 2018; Feldman et al., 2018; Iyengar et al., 2019; Wang & Gu, 2019).

Original: DP-ERM minimizes the empirical risk while guaranteeing that the output of learning algorithm is diﬀerentially private with respect to the training data.
Revised: DP-ERM minimizes the empirical risk while guaranteeing that the output of learning algorithm is diﬀerentially private with respect to the training data.

Original: Such privacy guarantee provides strong protection against potential adversaries (Hitaj et al., 2017; Rahman et al., 2018).
Revised: Such privacy guarantee provides strong protection against potential adversaries (Hitaj et al., 2017; Rahman et al., 2018).

Original: In order to guarantee privacy, it is necessary to introduce randomness to the algorithm.
Revised: In order to guarantee privacy, it is necessary to introduce randomness to the algorithm.

Original: There are usually three ways to introduce randomness according to the time of adding noise: output perturbation, objective perturbation and gradient perturbation .
Revised: There are usually three ways to introduce randomness according to the time of adding noise: output perturbation, objective perturbation and gradient perturbation .

Original: Output perturbation (Wu et al., 2017; Zhang et al., 2017)
Revised: Output perturbation (Wu et al., 2017; Zhang et al., 2017)

Original: ﬁrst runs the learning algorithm the same as in the non-private case then adds noise to the output parameter.
Revised: ﬁrst runs the learning algorithm the same as in the non-private case then adds noise to the output parameter.

Original: Objective perturbation (Chaudhuri et al., 2011; Kifer et al., 2012; Iyengar et al., 2019) perturbs the objective (i.e., the empirical loss) then release the minimizer of the perturbed objective.
Revised: Objective perturbation (Chaudhuri et al., 2011; Kifer et al., 2012; Iyengar et al., 2019) perturbs the objective (i.e., the empirical loss) then release the minimizer of the perturbed objective.

Original: Gradient perturbation (Song et al., 2013; Bassily et al., 2014; Abadi et al., 2016; Wang et al., 2017; Lee & Kifer, 2018; Jayaraman et al., 2018) perturbs each intermediate update.
Revised: Gradient perturbation (Song et al., 2013; Bassily et al., 2014; Abadi et al., 2016; Wang et al., 2017; Lee & Kifer, 2018; Jayaraman et al., 2018) perturbs each intermediate update.

Original: If each update is diﬀerentially private, the composition theorem of diﬀerential privacy ensures the whole learning procedure is diﬀerentially private.
Revised: If each update is diﬀerentially private, the composition theorem of diﬀerential privacy ensures the whole learning procedure is diﬀerentially private.

Original: Gradient perturbation comes with several advantages over output/objective perturbations.
Revised: Gradient perturbation comes with several advantages over output/objective perturbations.

Original: Firstly, gradient perturbation does not require strong assumption on the objective because it only needs to bound the sensitivity of gradient update rather than the whole learning process.
Revised: Firstly, gradient perturbation does not require strong assumption on the objective because it only needs to bound the sensitivity of gradient update rather than the whole learning process.

Original: Secondly, gradient perturbation can release the noisy gradient at each iteration without damaging the privacy guarantee as diﬀerential privacy is immune to post processing (Dwork et al., 2014).
Revised: Secondly, gradient perturbation can release the noisy gradient at each iteration without damaging the privacy guarantee as diﬀerential privacy is immune to post processing (Dwork et al., 2014).

Original: Thus, it is a more favorable choice for certain applications such as distributed optimization (Rajkumar & Agarwal, 2012; Agarwal et al., 2018; Jayaraman et al., 2018).
Revised: Thus, it is a more favorable choice for certain applications such as distributed optimization (Rajkumar & Agarwal, 2012; Agarwal et al., 2018; Jayaraman et al., 2018).

Original: At last, gradient perturbation often achieves better empirical utility than output/objective perturbations for DP-ERM.
Revised: At last, gradient perturbation often achieves better empirical utility than output/objective perturbations for DP-ERM.

Original: However, the existing theoretical utility guarantee for gradient perturbation is the same as or strictly inferior to that of other perturbation methods as shown in Table 1.
Revised: However, the existing theoretical utility guarantee for gradient perturbation is the same as or strictly inferior to that of other perturbation methods as shown in Table 1.

Original: This motivates us to ask “What is wrong with the theory for gradient perturbation?
Revised: This motivates us to ask “What is wrong with the theory for gradient perturbation?

Original: Can we justify the empirical advantage of gradient perturbation theoretically?”
Revised: Can we justify the empirical advantage of gradient perturbation theoretically?”

Original: We revisit the analysis for gradient perturbation approach.
Revised: We revisit the analysis for gradient perturbation approach.

Original: Previous work (Bassily et al., 2014; Wang et al., 2017; Jayaraman et al., 2018) derive the utility guarantee of gradient perturbation via two steps.
Revised: Previous work (Bassily et al., 2014; Wang et al., 2017; Jayaraman et al., 2018) derive the utility guarantee of gradient perturbation via two steps.

Original: They ﬁrst determine the noise variance at each step that meets the privacy requirement and then derive the utility guarantee by using the convergence analysis the same as in non-private case.
Revised: They ﬁrst determine the noise variance at each step that meets the privacy requirement and then derive the utility guarantee by using the convergence analysis the same as in non-private case.

Original: However, the noise to guarantee privacy naturally aﬀects the optimization procedure, but previous approach does not exploit the interaction between privacy noise and optimization of gradient perturbation.
Revised: However, the noise to guarantee privacy naturally aﬀects the optimization procedure, but previous approach does not exploit the interaction between privacy noise and optimization of gradient perturbation.

Original: In this paper, we utilize the fact the privacy noise aﬀects the optimization procedure and establish new and much tighter utility guarantees for gradient perturbation approaches.
Revised: In this paper, we utilize the fact the privacy noise aﬀects the optimization procedure and establish new and much tighter utility guarantees for gradient perturbation approaches.

Original: Our contribution can be summarized as follows.
Revised: Our contribution can be summarized as follows.

Original: • We introduce an expected curvature that can characterize the optimization property accurately when there is perturbation noise at each gradient update.
Revised: • We introduce an expected curvature that can characterize the optimization property accurately when there is perturbation noise at each gradient update.

Original: • We establish the utility guarantees for DP-GD for both convex and strongly convex objectives based on the expected curvature rather than the usual minimum curvature.
Revised: • We establish the utility guarantees for DP-GD for both convex and strongly convex objectives based on the expected curvature rather than the usual minimum curvature.

Original: • We also establish the the utility guarantees for
Revised: • We also establish the the utility guarantees for

Original: DP-SGD for both convex and strongly convex objectives based on the expected curvature .
Revised: DP-SGD for both convex and strongly convex objectives based on the expected curvature .

Original: To the best of our knowledge, this is the ﬁrst work to remove the dependency on minimum curvature for DP-ERM algorithms.
Revised: To the best of our knowledge, this is the ﬁrst work to remove the dependency on minimum curvature for DP-ERM algorithms.

Original: In DP-ERM literature, there is a gap between the utility guarantee of non-strongly convex objectives and that of strongly convex objectives.
Revised: In DP-ERM literature, there is a gap between the utility guarantee of non-strongly convex objectives and that of strongly convex objectives.

Original: However, by using the expected curvature , we show that some of the non-strongly convex objectives can achieve the same order of utility guarantee as the strongly convex objectives, matching the empirical observation.
Revised: However, by using the expected curvature , we show that some of the non-strongly convex objectives can achieve the same order of utility guarantee as the strongly convex objectives, matching the empirical observation.

Original: This is because the expected curvature could be relatively large even for non-strongly convex objectives.
Revised: This is because the expected curvature could be relatively large even for non-strongly convex objectives.

Original: As we mentioned earlier, prior to our work, there is a mismatch between theoretical guarantee and empirical observation of gradient perturbation approach compared with other two perturbation approaches.
Revised: As we mentioned earlier, prior to our work, there is a mismatch between theoretical guarantee and empirical observation of gradient perturbation approach compared with other two perturbation approaches.

Original: Our result theoretically justiﬁes the advantage of gradient perturbation and close the mismatch.
Revised: Our result theoretically justiﬁes the advantage of gradient perturbation and close the mismatch.

Original: 1. Paper Organization
Revised: 1. Paper Organization

Original: The rest of this paper is organized as follows.
Revised: The rest of this paper is organized as follows.

Original: Section 2 introduces notations and the
Revised: Section 2 introduces notations and the

Original: DP-ERM task.
Revised: DP-ERM task.

Original: In Sections 3, we ﬁrst introduce the expected curvature and establish the utility guarantee of both DP-GD and DP-SGD based on such expected curvature.
Revised: In Sections 3, we ﬁrst introduce the expected curvature and establish the utility guarantee of both DP-GD and DP-SGD based on such expected curvature.

Original: Then we give some discussion on three perturbation approaches.
Revised: Then we give some discussion on three perturbation approaches.

Original: We conduct extensive experiments in Section 4.
Revised: We conduct extensive experiments in Section 4.

Original: Finally, we conclude in Section 5.
Revised: Finally, we conclude in Section 5.

Original: [Table]
Revised: [Table]

Original:  Preliminary
Revised:  Preliminary

Original: We introduce notations and deﬁnitions in this section.
Revised: We introduce notations and deﬁnitions in this section.

Original: Given dataset D = { d 1 , . . .
Revised: Given dataset D = { d 1 , . . .

Original: , d n } , the objective function F ( x ; D ) is deﬁned as F ( x ; D ) =
Revised: , d n } , the objective function F ( x ; D ) is deﬁned as F ( x ; D ) =

Original: ∆ 1 n (cid:80) ni
Revised: ∆ 1 n (cid:80) ni

Original: =1 f ( x ; d i ) , where f ( x ; d i ) :
Revised: =1 f ( x ; d i ) , where f ( x ; d i ) :

Original: R p
Revised: R p

Original: → R is the loss of model x ∈ R p for the record d i .
Revised: → R is the loss of model x ∈ R p for the record d i .

Original: For simplicity, we use F ( x ) to denote F ( x ; D ) .
Revised: For simplicity, we use F ( x ) to denote F ( x ; D ) .

Original: We use (cid:107) v (cid:107) to denote the l 2 norm of a vector v .
Revised: We use (cid:107) v (cid:107) to denote the l 2 norm of a vector v .

Original: We use X ∗ f
Revised: We use X ∗ f

Original: = arg min x ∈ R p
Revised: = arg min x ∈ R p

Original: f
Revised: f

Original: ( x ) to denote the set of optimal solutions of f ( x ) .
Revised: ( x ) to denote the set of optimal solutions of f ( x ) .

Original: Throughout this paper, we assume X ∗ f non-empty.
Revised: Throughout this paper, we assume X ∗ f non-empty.

Original: Deﬁnition 1 (Objective properties) .
Revised: Deﬁnition 1 (Objective properties) .

Original: For any x , y ∈ R p , a function f : R p
Revised: For any x , y ∈ R p , a function f : R p

Original: → R
Revised: → R

Original: • is L -lipschitz if | f ( x ) − f ( y ) | ≤ L (cid:107) x − y (cid:107) .
Revised: • is L -Lipschitz if | f ( x ) − f ( y ) | ≤ L (cid:107) x − y (cid:107) .

Original: • is β -smooth if f ( y ) ≤ f ( x ) + (cid:104)∇ f ( x ) , y − x (cid:105) + β 2 (cid:107)
Revised: • is β -smooth if f ( y ) ≤ f ( x ) + (cid:104)∇ f ( x ) , y − x (cid:105) + β 2 (cid:107)

Original: y − x (cid:107) 2 .
Revised: y − x (cid:107) 2 .

Original: • is convex if (cid:104)∇ f ( x ) − ∇ f ( y ) , x − y (cid:105) ≥ 0 .
Revised: • is convex if (cid:104)∇ f ( x ) − ∇ f ( y ) , x − y (cid:105) ≥ 0 .

Original: • is µ -strongly convex (or µ -SC) if (cid:104)∇ f ( x ) − ∇ f ( y ) , x − y (cid:105) ≥ µ (cid:107) x − y (cid:107) 2 .
Revised: • is µ -strongly convex (or µ -SC) if (cid:104)∇ f ( x ) − ∇ f ( y ) , x − y (cid:105) ≥ µ (cid:107) x − y (cid:107) 2 .

Original: The strong convexity coeﬃcient µ is the lower bound of the minimum curvature of function f over the domain.
Revised: The strong convexity coeﬃcient µ is the lower bound of the minimum curvature of function f over the domain.

Original: We say that two datasets D, D (cid:48) are neighboring datasets (denoted as D ∼ D (cid:48) ) if D can be obtained by arbitrarily modifying one record in D (cid:48) (or vice versa).
Revised: We say that two datasets D, D (cid:48) are neighboring datasets (denoted as D ∼ D (cid:48) ) if D can be obtained by arbitrarily modifying one record in D (cid:48) (or vice versa).

Original: In this paper we consider ( (cid:15), δ ) -diﬀerential privacy as follows.
Revised: In this paper we consider ( (cid:15), δ ) -diﬀerential privacy as follows.

Original: Deﬁnition 2 ( ( (cid:15), δ ) -DP (Dwork et al., 2006a;b)) .
Revised: Deﬁnition 2 ( ( (cid:15), δ ) -DP (Dwork et al., 2006a;b)) .

Original: A randomized mechanism M : D → R guarantees ( (cid:15), δ ) -diﬀerential privacy if for any two neighboring input datasets D, D (cid:48) and for any subset of outputs S ⊆ R it holds that Pr [ M ( D ) ∈ S ] ≤ e (cid:15)
Revised: A randomized mechanism M : D → R guarantees ( (cid:15), δ ) -diﬀerential privacy if for any two neighboring input datasets D, D (cid:48) and for any subset of outputs S ⊆ R it holds that Pr [ M ( D ) ∈ S ] ≤ e (cid:15)

Original: Pr [ M ( D (cid:48) )
Revised: Pr [ M ( D (cid:48) )

Original: ∈ S ] + δ .
Revised: ∈ S ] + δ .

Original: We note that δ can be viewed as the probability that original (cid:15) -DP fails and a meaningful setting requires δ (cid:28) 1 n .
Revised: We note that δ can be viewed as the probability that original (cid:15) -DP fails and a meaningful setting requires δ (cid:28) 1 n .

Original: By its deﬁnition, diﬀerential privacy controls the maximum inﬂuence that any individual record can produce.
Revised: By its deﬁnition, diﬀerential privacy controls the maximum inﬂuence that any individual record can produce.

Original: Smaller (cid:15), δ implies less information leak but usually leads to worse utility.
Revised: Smaller (cid:15), δ implies less information leak but usually leads to worse utility.

Original: One can adjust (cid:15), δ to trade oﬀ between privacy and utility.
Revised: One can adjust (cid:15), δ to trade oﬀ between privacy and utility.

Original: DP-ERM requires the output x out ∈ R p is diﬀerentially private with respect to the input dataset D .
Revised: DP-ERM requires the output x out ∈ R p is diﬀerentially private with respect to the input dataset D .

Original: Let x ∗ ∈
Revised: Let x ∗ ∈

Original: X ∗ F be one of the optimal solutions of F ( x ) , the utility of DP-ERM algorithm is measured by expected excess empirical risk : E
Revised: X ∗ F be one of the optimal solutions of F ( x ) , the utility of DP-ERM algorithm is measured by expected excess empirical risk : E

Original: [ F ( x out ) − F ( x ∗ )] , where the expectation is taken over the algorithm randomness.
Revised: [ F ( x out ) − F ( x ∗ )] , where the expectation is taken over the algorithm randomness.

Original:  Main Results
Revised:  Main Results

Original: In this section, we ﬁrst deﬁne the expected curvature ν and explain why it depends only on the average curvature.
Revised: In this section, we ﬁrst deﬁne the expected curvature ν and explain why it depends only on the average curvature.

Original: We then use such expected curvature to improve the analysis of both
Revised: We then use such expected curvature to improve the analysis of both

Original: DP-SGD and DP-GD.
Revised: DP-SGD and DP-GD.

Original: 3. Expected Curvature
Revised: 3. Expected Curvature

Original: In non-private setting, the analysis of convex optimization relies on the strongly convex coeﬃcient µ , which is the minimum curvature over the domain and can be extremely small for some common objectives.
Revised: In non-private setting, the analysis of convex optimization relies on the strongly convex coeﬃcient µ , which is the minimum curvature over the domain and can be extremely small for some common objectives.

Original: Previous work on DP-ERM uses the same analysis as in non-private case and therefore the resulting utility bounds rely on the minimum curvature.
Revised: Previous work on DP-ERM uses the same analysis as in non-private case and therefore the resulting utility bounds rely on the minimum curvature.

Original: In our analysis, however, we avoid the dependency on the minimum curvature by exploiting how the privacy noise aﬀects the optimization.
Revised: In our analysis, however, we avoid the dependency on the minimum curvature by exploiting how the privacy noise aﬀects the optimization.

Original: With the perturbation noise, the expected curvature that the optimization path encounters is related to the average curvature instead of the minimum curvature.
Revised: With the perturbation noise, the expected curvature that the optimization path encounters is related to the average curvature instead of the minimum curvature.

Original: Deﬁnition 3 uses ν to capture such average curvature with Gaussian noise.
Revised: Deﬁnition 3 uses ν to capture such average curvature with Gaussian noise.

Original: We use x ∗ =
Revised: We use x ∗ =

Original: arg min x ∈X ∗ (cid:107) x − x 1 (cid:107) to denote the closest solution to the initial point.
Revised: arg min x ∈X ∗ (cid:107) x − x 1 (cid:107) to denote the closest solution to the initial point.

Original: Deﬁnition 3 (Expected curvature) .
Revised: Deﬁnition 3 (Expected curvature) .

Original: A convex function F : R p
Revised: A convex function F : R p

Original: → R , has expected curvature ν with respect to noise N (0 , σ 2 I p ) if for any x ∈ R p and ˜ x
Revised: → R , has expected curvature ν with respect to noise N (0 , σ 2 I p ) if for any x ∈ R p and ˜ x

Original: = x − z where z ∼ N (0 , σ 2 I p ) , it holds that
Revised: = x − z where z ∼ N (0 , σ 2 I p ) , it holds that

Original: [Equation]
Revised: [Equation]

Original: where the expectation is taken with respect to z .
Revised: where the expectation is taken with respect to z .

Original: Claim 1.
Revised: Claim 1.

Original: If F is µ -strongly convex, we have ν ≥ µ .
Revised: If F is µ -strongly convex, we have ν ≥ µ .

Original: Proof.
Revised: Proof.

Original: It can be veriﬁed that ν
Revised: It can be veriﬁed that ν

Original: = µ always holds because of the strongly convex deﬁnition.
Revised: = µ always holds because of the strongly convex deﬁnition.

Original: In fact, ν represents the average curvature and is much larger than µ .
Revised: In fact, ν represents the average curvature and is much larger than µ .

Original: We use x (cid:48) to denote the transpose of x .
Revised: We use x (cid:48) to denote the transpose of x .

Original: Let H x = ∇ 2 F ( x ) be the Hessian matrix evaluated at x .
Revised: Let H x = ∇ 2 F ( x ) be the Hessian matrix evaluated at x .

Original: Using Taylor expansion to approximate the l.h.s. of Eq (1), we have
Revised: We use Taylor expansion to approximate the left hand side of Eq (1) as follows

Original: [Equation]
Revised: [Equation]

Original: For  positive semi-deﬁnite matrix H x , we have tr( H x ) is the sum of the eigenvalues of H x . Rearrange the resulting inequality shows Deﬁnition 3 holds for any ν (cid:46) tr
Revised: For convex objective, the Hessian matrix is positive semi-deﬁnite and tr( H x ) is the sum of the eigenvalues of H x . We can further express out the right hand side of Eq (1) as follows

Original: [Equation]
Revised: [Equation]

Original: H [Equation] σ
Revised: Based on the above approximation, we can estimate the value of ν in Deﬁnition 3: ν (cid:46)

Original: [Equation]
Revised: [Equation]

Original: For relatively large σ 2 , this implies ν ≈ tr( H x ) p that is the average curvature at x .
Revised: For relatively large σ 2 , this implies ν ≈ tr( H x ) p that is the average curvature at x .

Original: Large variance is a reasonable setting because meaningful diﬀerential privacy guarantee requires non-trivial amount of noise.
Revised: Large variance is a reasonable setting because meaningful diﬀerential privacy guarantee requires non-trivial amount of noise.

Original: The above analysis suggests that ν can be independent of and much larger than µ .
Revised: The above analysis suggests that ν can be independent of and much larger than µ .

Original: This is indeed true for many convex objectives.
Revised: This is indeed true for many convex objectives.

Original: Let us take the l 2 regularized logistic regression as an example.
Revised: Let us take the l 2 regularized logistic regression as an example.

Original: The objective is strongly convex only due to the l 2 regularizer.
Revised: The objective is strongly convex only due to the l 2 regularizer.

Original: Thus, the minimum curvature (strongly convex coeﬃcient) is the regularization coeﬃcient λ .
Revised: Thus, the minimum curvature (strongly convex coeﬃcient) is the regularization coeﬃcient λ .

Original: Sharmir et al.
Revised: Sharmir et al.

Original: [1] shows the optimal choice of λ is Θ( n − 1 / 2 )
Revised: [1] shows the optimal choice of λ is Θ( n − 1 / 2 )

Original: (Section 4.3 in [1]).
Revised: (Section 4.3 in [1]).

Original: In practice, typical choice of λ is even smaller and could be on the order of n − 1 .
Revised: In practice, typical choice of λ is even smaller and could be on the order of n − 1 .

Original: Figure 1 compares the minimum and average curvatures of regularized logistic regression during the training process.
Revised: Figure 1 compares the minimum and average curvatures of regularized logistic regression during the training process.

Original: The average curvature is basically unaﬀected by the regularization term λ .
Revised: The average curvature is basically unaﬀected by the regularization term λ .

Original: In contrast, the minimum curvature reaches λ in ﬁrst few steps.
Revised: In contrast, the minimum curvature reaches λ in ﬁrst few steps.

Original: Therefore removing the dependence on minimum curvature is a signiﬁcant improvement.
Revised: Therefore removing the dependence on minimum curvature is a signiﬁcant improvement.

Revised: We also plot the curvatures for another dataset KDDCup99 in the Appendix C.

Revised: The resulting curvatures are similar to Figure 1.

Original: Perturbation noise is necessary to attain ν > µ .
Revised: Perturbation noise is necessary to attain ν > µ .

Original: We note that ν
Revised: We note that ν

Original: = µ when the training process does not involve perturbation noise (corresponding to σ = 0 in Deﬁnition 3). 
Revised: = µ when the training process does not involve perturbation noise (corresponding to σ = 0 in Deﬁnition 3). For

Original: For example, objective/output perturbation cannot utilize this expected curvature condition as no noise is injected in their training process.
Revised: [Figure] > example, objective/output perturbation cannot utilize this expected curvature condition as no noise is injected in their training process.

Original: Therefore, among three existing perturbation methods, gradient perturbation is the only method can leverage such eﬀect of noise.[Figure]
Revised: Therefore, among three existing perturbation methods, gradient perturbation is the only method can leverage such eﬀect of noise.

Original: We note that µ = 0 does not necessarily lead to ν = 0 .
Revised: We note that µ = 0 does not necessarily lead to ν = 0 .

Original: A concrete example is given in Figure 2 (from Negahban et al.
Revised: A concrete example is given in Figure 2 (from Negahban et al.

Original: It provides an illustration of the loss function in the high-dimensional ( p > n ) setting, i.e., the resticted strongly convex scenario: the loss is curved in certain directions but completely ﬂat in others.
Revised: It provides an illustration of the loss function in the high-dimensional ( p > n ) setting, i.e., the resticted strongly convex scenario: the loss is curved in certain directions but completely ﬂat in others.

Original: The average curvature of such objective is always positive but the worst curvature is 0 .
Revised: The average curvature of such objective is always positive but the worst curvature is 0 .

Revised: Though some recent work shows the utility guarantee of high dimensional DP-ERM task may not depend on the worst curvature (Wang & Gu, 2019), Figure 2 still provides a good illustration for the case of ν > µ = 0 .

Original: Moreover, as shown in Figure 1, the average curvature of logistic regression  is positive during the training procedure even the regularization term is 0 .
Revised: Moreover, as shown in Figure 1, the average curvature of logistic regression on Adult dataset is above 0 during the training procedure even the regularization term is 0 .

Original: As we will show later, ν being positive  over the optimization path is suﬃcient for our optimization analysis.
Revised: As we will show later, a positive ν over the optimization path is suﬃcient for our optimization analysis.

Original: 3. Utility Guarantee of DP-GD Based on Expected Curvature
Revised: 3. Utility Guarantee of DP-GD Based on Expected Curvature

Original: In this section we show that the expected curvature can be used to improve the utility bound of DP-GD (Algorithm 1).
Revised: In this section we show that the expected curvature can be used to improve the utility bound of DP-GD (Algorithm 1).

Original: Algorithm 1: Diﬀerentially Private Gradient Descent (DP-GD)
Revised: Algorithm 1: Diﬀerentially Private Gradient Descent (DP-GD)

Original: Input: Privacy parameters (cid:15), δ ; running steps T ; learning rate η .
Revised: Input : Privacy parameters (cid:15), δ ; running steps T ; learning rate η .

Original: Loss function F ( x ) with Lipschitz constant L .
Revised: Loss function F ( x ) with Lipschitz constant L .

Original: for t = 1 to T do
Revised: for t = 1 to T do

Original: Compute g t = ∇ F ( x t ) .
Revised: Compute g t = ∇ F ( x t ) .

Original: Update parameter x t +

Revised: Update parameter x t +

Original: = x
Revised: = x

Original: t − η t ( g t + z t ) , where z t ∼ N (cid:0) 0 , σ 2 t I p (cid:1) .
Revised: t − η t ( g t + z t ) , where z t ∼ N (cid:0) 0 , σ 2 t I p (cid:1) .

Original: end for
Revised: end for

Original: Algorithm 1 is ( (cid:15), δ ) -DP if we set σ t =
Revised: Algorithm 1 is ( (cid:15), δ ) -DP if we set σ t =

Original: Θ (cid:18)
Revised: Θ (cid:18)

Original: L √ T log(1 /δ )
Revised: L √ T log(1 /δ )

Original: n(cid:15) (cid:19) (Jayaraman et al., 2018).
Revised: n(cid:15) (cid:19) (Jayaraman et al., 2018).

Original: Let x 1 , . . .
Revised: Let x 1 , . . .

Original: , x T be the training path and ν = min { ν 1 , . . .
Revised: , x T be the training path and ν = min { ν 1 , . . .

Original: , ν T } be the minimum expected curvature over the path.
Revised: , ν T } be the minimum expected curvature over the path.

Original: Now we present the utility guarantee of DP-GD for the case of ν > 0 .
Revised: Now we present the utility guarantee of DP-GD for the case of ν > 0 .

Original: Theorem 1 (Utility guarantee, ν > 0 .) .
Revised: Theorem 1 (Utility guarantee, ν > 0 .) .

Original: Suppose F is  β -smooth.
Revised: Suppose F is L -Lipschitz and β -smooth with ν expected curvature.

Original: Set η ≤ 1 β , T = 2log( n ) ην and σ t =
Revised: Set η ≤ 1 β , T = 2log( n ) ην and σ t =

Original: Θ (cid:16) L (cid:112) T log(1 /δ )
Revised: Θ (cid:16) L (cid:112) T log(1 /δ )

Original: /n(cid:15) (cid:17) , we have
Revised: /n(cid:15) (cid:17) , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: All proofs in this paper are relegated to Appendix A.
Revised: All proofs in this paper are relegated to Appendix A.

Original: Remark 1.
Revised: Remark 1.

Original: Theorem 3 only depends on the expected curvature over the training path ν .
Revised: Theorem 3 only depends on the expected curvature over the training path ν .

Original: The expectation is taken over the algorithm randomness if without speciﬁcation.
Revised: The expectation is taken over the algorithm randomness if without speciﬁcation.

Original: Theoremsigniﬁcantly improves the original analysis of DP-GD because of our arguments in Section 3.1.
Revised: Theoremsigniﬁcantly improves the original analysis of DP-GD because of our arguments in Section 3.1.

Original: If ν = 0 , then the curvatures are ﬂatten in all directions.
Revised: If ν = 0 , then the curvatures are ﬂatten in all directions.

Original: One example is the linear function, which is used by Bassily et al.
Revised: One example is the linear function, which is used by Bassily et al.

Original: (2014) to derive their utility lower bound.
Revised: (2014) to derive their utility lower bound.

Original: Such simple function may not be commonly used as loss function in practice.
Revised: Such simple function may not be commonly used as loss function in practice.

Original: Nonetheless, we give the utility guarantee for the case of ν = 0 in Theorem 2.
Revised: Nonetheless, we give the utility guarantee for the case of ν = 0 in Theorem 2.

Original: Theorem 2 (Utility guarantee, ν = 0 .) .
Revised: Theorem 2 (Utility guarantee, ν = 0 .) .

Original: Suppose F is  β -smooth.
Revised: Suppose F is L -Lipschitz and β -smooth.

Original: Set η = 1 β , T = nβ(cid:15) √ p and σ t = Θ (cid:16) L (cid:112) T log(1 /δ )
Revised: Set η = 1 β , T = nβ(cid:15) √ p and σ t = Θ (cid:16) L (cid:112) T log(1 /δ )

Original: /n(cid:15)
Revised: /n(cid:15)

Original: (cid:17) .
Revised: (cid:17) .

Original: Let ¯ x = 1 T (cid:80)
Revised: Let ¯ x = 1 T (cid:80)

Original: Ti
Revised: Ti

Original: =1 x i +1 , we have
Revised: =1 x i +1 , we have

Original: E [ F (¯ x ) − F ( x ∗ )] = O (cid:18) √ pL 2 log (1 /δ ) n(cid:15) (cid:19) .
Revised: [Equation]

Original: We use parameter averaging to reduce the inﬂuence of perturbation noise because gradient update does not have strong contraction eﬀect when ν = 0 .
Revised: We use parameter averaging to reduce the inﬂuence of perturbation noise because gradient update does not have strong contraction eﬀect when ν = 0 .

Original: 3. Utiltiy Guarantee of DP-SGD Based on Expected Curvature
Revised: 3. Utiltiy Guarantee of DP-SGD Based on Expected Curvature

Original: Stochastic gradient descent has become one of the most popular optimization methods because of the cheap one-iteration cost.
Revised: Stochastic gradient descent has become one of the most popular optimization methods because of the cheap one-iteration cost.

Original: In this section we show that expected curvature can also improve the utility analysis for DP-SGD (Algorithm 2).
Revised: In this section we show that expected curvature can also improve the utility analysis for DP-SGD (Algorithm 2).

Original: We note that ∇ f ( x ) represents an element from the subgradient set evaluated at x when the objective is not smooth.
Revised: We note that ∇ f ( x ) represents an element from the subgradient set evaluated at x when the objective is not smooth.

Original: Before stating our theorem, we introduce the moments accountant technique (Lemma 1) that is essential to establish privacy guarantee.
Revised: Before stating our theorem, we introduce the moments accountant technique (Lemma 1) that is essential to establish privacy guarantee.

Original: Lemma 1 (Abadi et al.
Revised: Lemma 1 (Abadi et al.

Original: There exist constants c 1 and c 2 so that given running steps
Revised: There exist constants c 1 and c 2 so that given running steps

Original: T , for any (cid:15) < c 1 T/n 2 , Algorithm 2 is ( (cid:15), δ ) -diﬀerentially private for any δ > 0 if we choose σ ≥ c√ Tlog (1 /δ ) n(cid:15) .
Revised: T , for any (cid:15) < c 1 T/n 2 , Algorithm 2 is ( (cid:15), δ ) -diﬀerentially private for any δ > 0 if we choose σ ≥ c√ Tlog (1 /δ ) n(cid:15) .

Original: Input :Dataset D = { d 1 , . . . , d n } . f i x ) =
Revised: Input :

Original: ; di
Revised: { d

Original: Lipschitz

Original: [Table] for [Table] to
Revised: d n }

Original: [Table] do Sample i t from [ n ] uniformly  Compute g
Revised: Individual loss f

Original: t =∇ f
Revised: ( x ) = f

Original: i t ( x t ) .
Revised: ( x ; d i ) with

Original: [-Update parameter x t +
-]
Revised: [-Update parameter x t +
-]Lipschitz constant L g

Original: = x t − η t( g t+ z t ) , where z t ∼ N (cid:0) 0 , L 2 σ 2 I p (cid:1) .
Revised: Update parameter t − η t g t z t ) , where z t ∼ N (cid:0) 0 , L 2 σ 2 I p (cid:1) .

Original: 5 end
Revised: 5 end

Original: For the case of ν > 0 , Theorem 3 presents the utility guarantee of DP-SGD.
Revised: For the case of ν > 0 , Theorem 3 presents the utility guarantee of DP-SGD.

Original: Theorem 3 (Utility guarantee, ν > 0 .) .
Revised: Theorem 3 (Utility guarantee, ν > 0 .) .

Revised: Suppose F is L -Lipschitz with ν expected curvature.

Original: Choose σ based on Lemma 1 to guarantee ( (cid:15), δ ) -DP.
Revised: Choose σ based on Lemma 1 to guarantee ( (cid:15), δ ) -DP.

Original: Set η t = 1 νt and T = n 2 (cid:15) 2 , we have
Revised: Set η t = 1 νt and T = n 2 (cid:15) 2 , we have

Original: [Equation]
Revised: E [ F ( x T ) − F ( x ∗ )] = O (cid:18) pL 2 log ( n ) log (1 /δ ) n 2 (cid:15) 2 ν (cid:19) .

Original: Remark 2.
Revised: Remark 2.

Original: Theorem 3 does not require smooth assumption.
Revised: Theorem 3 does not require smooth assumption.

Original: Theorem 3 shows the utility guarantee of DP-SGD also depends on ν rather than µ .
Revised: Theorem 3 shows the utility guarantee of DP-SGD also depends on ν rather than µ .

Original: We set
Revised: We set

Original: T =
Revised: T =

Original: Θ( n 2 ) following Bassily et al.
Revised: Θ( n 2 ) following Bassily et al.

Original: We note that T =
Revised: We note that T =

Original: Θ( n 2 ) is necessary even for non-private SGD to reach 1 /n 2 precision.
Revised: Θ( n 2 ) is necessary even for non-private SGD to reach 1 /n 2 precision.

Original: We next show for a relatively coarse precision, the running time can be reduced signiﬁcantly.
Revised: We next show for a relatively coarse precision, the running time can be reduced signiﬁcantly.

Original: Theorem 4.
Revised: Theorem 4.

Revised: Suppose F is L -Lipschitz with ν expected curvature.

Original: Choose σ based on Lemma 1 to guarantee ( (cid:15), δ ) -DP.
Revised: Choose σ based on Lemma 1 to guarantee ( (cid:15), δ ) -DP.

Original: Set η t = 1 νt and T = n(cid:15) √ p .
Revised: Set η t = 1 νt and T = n(cid:15) √ p .

Original: Suppose p < n 2 , we have
Revised: Suppose p < n 2 , we have

Original: [Equation]
Revised: [Equation]

Revised: ( x ∗ )]

Revised: = O (cid:18) √ pL 2 log( n ) n(cid:15)ν (cid:19) .

Original: We note that the analysis of Bassily et al.
Revised: We note that the analysis of Bassily et al.

Original: (2014) yields E [ F ( x T ) − F ( x ∗ )]
Revised: (2014) yields E [ F ( x T ) [Equation]

Original: = O (cid:16) √ pL 2 log 2 ( n ) n(cid:15)µ (cid:17) if setting T = n(cid:15) √ p , which still depends on the minimum curvature.
Revised: if setting T = n(cid:15) √ p , which still depends on the minimum curvature.

Original: Theorem 5 shows the utility for the case of ν = 0 .
Revised: Theorem 5 shows the utility for the case of ν = 0 .

Original: Theorem 5 (Utility guarantee, ν = 0 .) .
Revised: Theorem 5 (Utility guarantee, ν = 0 .) .

Revised: Suppose F is L -Lipschitz.

Original: Assume (cid:107) x t (cid:107) ≤ D for t ∈ [ T ] .
Revised: Assume (cid:107) x t (cid:107) ≤ D for t ∈ [ T ] .

Original: Choose σ based on Lemma 1 to guarantee ( (cid:15), δ ) -DP.
Revised: Choose σ based on Lemma 1 to guarantee ( (cid:15), δ ) -DP.

Original: Let G =
Revised: Let G =

Original: L (cid:112) 

Revised: L (cid:112)

Original: + pσ 2 , set η t = DG √ t and T = n 2 (cid:15) 2 , we have
Revised: + pσ 2 , set η t = DG √ t and T = n 2 (cid:15) 2 , we have

Original: E [ F ( x T ) − F ( x ∗ )] = O (cid:32)(cid:112) p log (1 /δ ) L log ( n ) n(cid:15) (cid:33) .
Revised: [Equation]

Original: This utility guarantee can be derived from Theorem 2 in (Shamir & Zhang, 2013).
Revised: This utility guarantee can be derived from Theorem 2 in (Shamir & Zhang, 2013).

Original: 3. Discussion on three perturbation approaches.
Revised: 3. Discussion on three perturbation approaches.

Original: In this section, we brieﬂy discuss two other perturbation approaches and compare them to the gradient perturbation approach.
Revised: In this section, we brieﬂy discuss two other perturbation approaches and compare them to the gradient perturbation approach.

Original: Output perturbation (Wu et al., 2017; Zhang et al., 2017) perturbs the learning algorithm after training.
Revised: Output perturbation (Wu et al., 2017; Zhang et al., 2017) perturbs the learning algorithm after training.

Original: It adds noise to the resulting model of non-private learning process.
Revised: It adds noise to the resulting model of non-private learning process.

Original: The magnitude of perturbation noise is propositional to the maximum inﬂuence one record can cause on the learned model.
Revised: The magnitude of perturbation noise is propositional to the maximum inﬂuence one record can cause on the learned model.

Original: Take the gradient descent algorithm as an example.
Revised: Take the gradient descent algorithm as an example.

Original: At each step, the gradient of diﬀerent records would diverge the two sets of parameters generated by neighboring datasets, the maximum distance expansion is related to the Lipschitz coeﬃcient.
Revised: At each step, the gradient of diﬀerent records would diverge the two sets of parameters generated by neighboring datasets, the maximum distance expansion is related to the Lipschitz coeﬃcient.

Original: At the same time, the gradient of the same records in two datasets would shrink the parameter distance because of the contraction eﬀect of the gradient update.
Revised: At the same time, the gradient of the same records in two datasets would shrink the parameter distance because of the contraction eﬀect of the gradient update.

Original: The contraction eﬀect depends on the smooth and strongly convex coeﬃcient.
Revised: The contraction eﬀect depends on the smooth and strongly convex coeﬃcient.

Original: Smaller strongly convex coeﬃcient leads to weaker contraction.
Revised: Smaller strongly convex coeﬃcient leads to weaker contraction.

Original: The sensitivity of output perturbation algorithm is the upper bound on the largest possible ﬁnal distance between two sets of parameters.
Revised: The sensitivity of output perturbation algorithm is the upper bound on the largest possible ﬁnal distance between two sets of parameters.

Original: Objective perturbation (Chaudhuri et al., 2011; Kifer et al., 2012; Iyengar et al., 2019) perturbs the objective function before training.
Revised: Objective perturbation (Chaudhuri et al., 2011; Kifer et al., 2012; Iyengar et al., 2019) perturbs the objective function before training.

Original: It requires the objective function to be strongly convex to guarantee the uniqueness of the solution.
Revised: It requires the objective function to be strongly convex to guarantee the uniqueness of the solution.

Original: It ﬁrst adds L 2 regularization to obtain strong convexity if the original objective is not strongly convex.
Revised: It ﬁrst adds L 2 regularization to obtain strong convexity if the original objective is not strongly convex.

Original: Then it perturbs the objective with a random linear term.
Revised: Then it perturbs the objective with a random linear term.

Original: The sensitivity of objective perturbation is the maximum change of the minimizer that one record can produce.
Revised: The sensitivity of objective perturbation is the maximum change of the minimizer that one record can produce.

Original: Chaudhuri et al.
Revised: Chaudhuri et al.

Original: (2011) and Kifer et al.
Revised: (2011) and Kifer et al.

Original: (2012) use the largest and the smallest eigenvalue (i.e. the smooth and strongly convex coeﬃcient) of the objective’s Hessian matrix to upper bound such change.
Revised: (2012) use the largest and the smallest eigenvalue (i.e. the smooth and strongly convex coeﬃcient) of the objective’s Hessian matrix to upper bound such change.

Original: In comparison, gradient perturbation is more ﬂexible than output/objective perturbation.
Revised: In comparison, gradient perturbation is more ﬂexible than output/objective perturbation.

Original: For example, to bound the sensitivity, gradient perturbation only requires Lipschitz coeﬃcient which can be easily obtained by using the gradient clipping technique.
Revised: For example, to bound the sensitivity, gradient perturbation only requires Lipschitz coeﬃcient which can be easily obtained by using the gradient clipping technique.

Original: However, both output and objective perturbation further need to compute the smooth coeﬃcient, which is hard for some common objectives such as softmax regression.
Revised: However, both output and objective perturbation further need to compute the smooth coeﬃcient, which is hard for some common objectives such as softmax regression.

Original: More critically, output/objective perturbation cannot utilize the expected curvature condition because their training process does not contain perturbation noise.
Revised: More critically, output/objective perturbation cannot utilize the expected curvature condition because their training process does not contain perturbation noise.

Original: Moreover, they have to consider the worst performance of learning algorithm.
Revised: Moreover, they have to consider the worst performance of learning algorithm.

Original: That is because DP makes the worst case assumption on query function and output/objective perturbation treat the whole learning algorithm as a single query to private dataset.
Revised: That is because DP makes the worst case assumption on query function and output/objective perturbation treat the whole learning algorithm as a single query to private dataset.

Original: This explains why their utility guarantee depends on the worst curvature of the objective.
Revised: This explains why their utility guarantee depends on the worst curvature of the objective.

Original:  Experiment
Revised:  Experiment

Original: In this section, we evaluate the performance of DP-GD and DP-SGD on multiple real world datasets.
Revised: In this section, we evaluate the performance of DP-GD and DP-SGD on multiple real world datasets.

Original: We use the benchmark datasets provided by Iyengar et al.
Revised: We use the benchmark datasets provided by Iyengar et al.

Original: Objective functions are logistic regression and softmax regression for binary and multi-class datasets, respectively.
Revised: Objective functions are logistic regression and softmax regression for binary and multi-class datasets, respectively.

Original: Datasets.
Revised: Datasets.

Original: The benchmark datasets includes two multi-class datasets (MNIST, Covertype) and ﬁve binary datasets, and three of them are high dimensional (Gisette, Real-sim, RCV1). 
Revised: The benchmark datasets includes two multi-class datasets (MNIST, Covertype) and ﬁve binary datasets, and three of them are high dimensional (Gisette, Real-sim, RCV1). [Table] [Figure]

Revised: NP represents non-private baseline.

Revised: Detailed description about evaluated datasets can be found in Table 3.

Original: Following Iyengar et al.
Revised: Following Iyengar et al.

Original: (2019), we use 80% data for training and the rest for testing.
Revised: (2019), we use 80% data for training and the rest for testing.

Original: Detailed description of datasets can be found in Appendix B
Revised: Detailed description of datasets can be found in Appendix B

Original: Implementation details.
Revised: Implementation details.

Original: We track Rényi diﬀerentialy privacy (RDP) (Mironov, 2017) and convert it to ( (cid:15), δ ) -DP.
Revised: We track Rényi diﬀerentialy privacy (RDP) (Mironov, 2017) and convert it to ( (cid:15), δ ) -DP.

Original: Running step T is chosen from { 50 , 200 , 800 } for both DP-GD [Table] [Figure]
Revised: Running step T is chosen from { 50 , 200 , 800 } for both DP-GD and DP-SGD.

Original: NP represents non-private baseline.

Original: Detailed description about evaluated datasets can be found in Table 3.

Original: and DP-SGD. For DP-SGD, we use moments accountant to track the privacy loss and the sampling ratio is set as 0 . 1 .
Revised: For DP-SGD, we use moments accountant to track the privacy loss and the sampling ratio is set as 0 . 1 .

Original: The standard deviation of the added noise σ is set to be the smallest value such that the privacy budget is allowable to run desired steps.
Revised: The standard deviation of the added noise σ is set to be the smallest value such that the privacy budget is allowable to run desired steps.

Original: We ensure each loss function is Lipschitz by clipping individual gradient.
Revised: We ensure each loss function is Lipschitz by clipping individual gradient.

Original: The method in Goodfellow (2015) allows us to clip individual gradient eﬃciently.
Revised: The method in Goodfellow (2015) allows us to clip individual gradient eﬃciently.

Original: Clipping threshold is set as 1 ( 0 .for high dimensional datasets because of the sparse gradient).
Revised: Clipping threshold is set as 1 ( 0 .for high dimensional datasets because of the sparse gradient).

Original: For DP-GD, learning rate is chosen from { 0 . 1 , 1 . 0 , 5 . 0 } ( { 0 . 2 , 2 . 0 , 10 . 0 } for high dimensional datasets).
Revised: For DP-GD, learning rate is chosen from { 0 . 1 , 1 . 0 , 5 . 0 } ( { 0 . 2 , 2 . 0 , 10 . 0 } for high dimensional datasets).

Original: The learning rate of DP-SGD is twice as large as DP-GD and it is divided by 2 at the middle of training.
Revised: The learning rate of DP-SGD is twice as large as DP-GD and it is divided by 2 at the middle of training.

Original: Privacy parameter δ is set as 1 n 2 .
Revised: Privacy parameter δ is set as 1 n 2 .

Original: The l 2 regularization coeﬃcient is set as 1 × 10 − 4 .
Revised: The l 2 regularization coeﬃcient is set as 1 × 10 − 4 .

Original: All reported numbers are averaged over 20 runs.
Revised: All reported numbers are averaged over 20 runs.

Original: Baseline algorithms.
Revised: Baseline algorithms.

Original: The baseline algorithms include state-of-the-art objective and output perturbation algorithms.
Revised: The baseline algorithms include state-of-the-art objective and output perturbation algorithms.

Original: For objective perturbation, we use Approximate Minima Perturbation (AMP)
Revised: For objective perturbation, we use Approximate Minima Perturbation (AMP)

Original: (Iyengar et al., 2019).
Revised: (Iyengar et al., 2019).

Original: For output perturbation, we use the algorithm in Wu et al.
Revised: For output perturbation, we use the algorithm in Wu et al.

Original: (2017) (Output perturbation SGD).
Revised: (2017) (Output perturbation SGD).

Original: We adopt the implementation and hyperparameters in Iyengar et al.
Revised: We adopt the implementation and hyperparameters in Iyengar et al.

Original: for both algorithms.
Revised: for both algorithms.

Original: For multi-class classiﬁcation tasks, Wu et al.
Revised: For multi-class classiﬁcation tasks, Wu et al.

Original: (2017) and Iyengar et al.
Revised: (2017) and Iyengar et al.

Original: (2019) divide the privacy budget evenly and train multiple binary classiﬁers because their algorithms need to compute smooth coeﬃcient before training and therefore are not directly applicable to softmax regression.
Revised: (2019) divide the privacy budget evenly and train multiple binary classiﬁers because their algorithms need to compute smooth coeﬃcient before training and therefore are not directly applicable to softmax regression.

Original: Experiment results.
Revised: Experiment results.

Original: The validation accuracy results for all evaluated algorithms with (cid:15) = 0 .
Revised: The validation accuracy results for all evaluated algorithms with (cid:15) = 0 .

Original: for multi-class datasets) are presented in Table 2.
Revised: for multi-class datasets) are presented in Table 2.

Original: We also plot the accuracy results with varying (cid:15) in Figure 3.
Revised: We also plot the accuracy results with varying (cid:15) in Figure 3.

Original: These results conﬁrm our theory in Section 3: gradient perturbation achieves better performance than other perturbation methods as it leverages the average curvature.
Revised: These results conﬁrm our theory in Section 3: gradient perturbation achieves better performance than other perturbation methods as it leverages the average curvature.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we show the privacy noise actually helps optimization analysis, which can be used to improve the utility guarantee of both DP-GD and DP-SGD.
Revised: In this paper, we show the privacy noise actually helps optimization analysis, which can be used to improve the utility guarantee of both DP-GD and DP-SGD.

Original: Our result theoretically justiﬁes the empirical superiority of gradient perturbation over other methods and advance the state of the art utility guarantee of DP-ERM algorithms.
Revised: Our result theoretically justiﬁes the empirical superiority of gradient perturbation over other methods and advance the state of the art utility guarantee of DP-ERM algorithms.

Original: Experiments on real world datasets corroborate our theoretical ﬁndings nicely.
Revised: Experiments on real world datasets corroborate our theoretical ﬁndings nicely.

Original: In the future, it is interesting to consider how to utilize the expected curvature condition to improve the utility guarantee of other gradient perturbation based algorithms.
Revised: In the future, it is interesting to consider how to utilize the expected curvature condition to improve the utility guarantee of other gradient perturbation based algorithms.

Original: Appendix A Proofs Related to DP-GD and DP-SGD
Revised: Appendix A Proofs Related to DP-GD and DP-SGD

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: Let x 1 , . . .
Revised: Let x 1 , . . .

Original: , x t be the path generated by optimization procedure.
Revised: , x t be the path generated by optimization procedure.

Original: Since x t contains Gaussian perturbation noise z
Revised: Since x t contains Gaussian perturbation noise z

Original: [Equation]
Revised: [Equation]

Original: 1 , Deﬁnition 3 gives us
Revised: 1 , Deﬁnition 3 gives us

Original: [Equation]
Revised: [Equation]

Original: Since F is β -smooth, we have
Revised: Since F is β -smooth, we have

Original: [Equation]
Revised: [Equation]

Original: Take linear combination of above inequalities,
Revised: Take linear combination of above inequalities,

Original: [Equation]
Revised: [Equation]

Original: Let r t
Revised: Let r t

Original: =
Revised: =

Original: (cid:107) x t − x ∗ (cid:107) be the solution error at step t .
Revised: (cid:107) x t − x ∗ (cid:107) be the solution error at step t .

Original: We have the following inequalities between r t +1 and r t .
Revised: We have the following inequalities between r t +1 and r t .

Original: [Equation]
Revised: [Equation]

Original: Take expectation with respect to z t , we have
Revised: Take expectation with respect to z t , we have

Original: [Equation]
Revised: [Equation]

Original: Further take expectation with respect to z t − 1 and use Eq 3, we have
Revised: Further take expectation with respect to z t − 1 and use Eq 2, we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Set θ = 12 and η ≤ 1 β ,
Revised: Set θ = 12 and η ≤ 1 β ,

Original: [Equation]
Revised: [Equation]

Original: Applying Eq (7) and taking expectation with respect to z t
Revised: Applying Eq (6) and taking expectation with respect to z t

Original: [Equation]
Revised: [Equation]

Original: z t − 1 , · · · , z 1 iteratively yields
Revised: z t − 1 , · · · , z 1 iteratively yields

Original: [Equation]
Revised: [Equation]

Original: Uniform privacy budget allocation scheme sets
Revised: Uniform privacy budget allocation scheme sets

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: Set T ≥ 2log( n ) ην , we have
Revised: Set T ≥ 2log( n ) ην , we have

Original: [Equation]
Revised: [Equation]

Original: Last inequality holds because 1 ην log(1 + ην 1 − ην ) > 1 for 1 ην ≥ β ν ≥ 1 .
Revised: Last inequality holds because 1 ην log(1 + ην 1 − ην ) > 1 for 1 ην ≥ β ν ≥ 1 .

Original: Therefore, for T ≥ 2log( n ) ην , we have the excepted solution error E [ r 2 T +1 ] satisﬁes
Revised: Therefore, for T ≥ 2log( n ) ην , we have the excepted solution error E [ r 2 T +1 ] satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Since F ( x ) is β -smooth, we have
Revised: Since F ( x ) is β -smooth, we have

Original: [Equation]
Revised: [Equation]

Original: Using Eq (11) and Eq (12), we have the excepted excess risk satisﬁes
Revised: Using Eq (10) and Eq (11), we have the excepted excess risk satisﬁes

Original: [Equation]
Revised: [Equation]

Original: for T ≥ 2log( n ) ην .
Revised: for T ≥ 2log( n ) ην .

Original: The utility bound is minimized when T = 2log( n ) ην .
Revised: The utility bound is minimized when T = 2log( n ) ην .

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: The smooth condition gives us,
Revised: The smooth condition gives us,

Original: [Equation]
Revised: [Equation]

Original: Take expectation with respect to z t and substitute η = 1 β ,
Revised: Take expectation with respect to z t and substitute η = 1 β ,

Original: [Equation]
Revised: [Equation]

Original: Subtract F ( x ∗ ) on both sides and use convexity,
Revised: Subtract F ( x ∗ ) on both sides and use convexity,

Original: [Equation]
Revised: [Equation]

Original: Substitute ∇ F
Revised: Substitute ∇ F

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Summing over t = 1 , . .
Revised: Summing over t = 1 , . .

Original: , T and take expectation with respect to z 1 , . .
Revised: , T and take expectation with respect to z 1 , . .

Original: , z T ,
Revised: , z T ,

Original: T (cid:88)
Revised: T (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Use convexity,
Revised: Use convexity,

Original: [Equation]
Revised: [Equation]

Original: Choose T = nβ(cid:15) √ p , we have
Revised: Choose T = nβ(cid:15) √ p , we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 3 and 4.
Revised: Proof of Theorem 3 and 4.

Original: We start by giving a useful lemma.
Revised: We start by giving a useful lemma.

Original: Lemma 2.
Revised: Lemma 2.

Original: Choose η t = 1 νt , the expected solution error of x t in Algorithm 2 for any t > 1 satisﬁes
Revised: Choose η t = 1 νt , the expected solution error of x t in Algorithm 2 for any t > 1 satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 2.
Revised: Proof of Lemma 2.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Take expectation with respect to perturbation noise z t and uniform sampling, we have
Revised: Take expectation with respect to perturbation noise z t and uniform sampling, we have

Original: [Equation]
Revised: [Equation]

Original: Further take expectation to z t − 1 and apply Deﬁnition 3,
Revised: Further take expectation to z t − 1 and apply Deﬁnition 3,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Now we use induction to conduct the proof.
Revised: Now we use induction to conduct the proof.

Original: Substitute η t = 1 tν into Eq 22, we have Lemma 2 hold for t = 2 .
Revised: Substitute η t = 1 tν into Eq 21, we have Lemma 2 hold for t = 2 .

Original: Assume E [ (cid:107) x t
Revised: Assume E [ (cid:107) x t

Original: [Equation]
Revised: [Equation]

Original: ∗
Revised: ∗

Original: [Equation]
Revised: [Equation]

Original: holds for t > 2 , then
Revised: holds for t > 2 , then

Original: [Equation]
Revised: [Equation]

Original: It’s easy to check that Eq 21 holds for arbitrary x rather than x ∗ .
Revised: It’s easy to check that Eq 20 holds for arbitrary x rather than x ∗ .

Original: Rearrange Eq 21 and take expectation, we have
Revised: Rearrange Eq 20 and take expectation, we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Let k be arbitrarily chosen from { 1 , . .
Revised: Let k be arbitrarily chosen from { 1 , . .

Original: , (cid:98) T/ 2 (cid:99)}
Revised: , (cid:98) T/ 2 (cid:99)}

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Substitute η t 
Revised: Substitute η t =

Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Now we bound E [ (cid:107) x t − x
Revised: Now we bound E [ (cid:107) x t − x

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: 2 ] for t ≥ T
Revised: 2 ] for t ≥ T

Original: [Equation]
Revised: [Equation]

Original: k ,
Revised: k ,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Substitute Eq 27 into Eq 26,
Revised: Substitute Eq 26 into Eq 25,

Original: [Equation]
Revised: [Equation]

Original: Let S k
Revised: Let S k

Original: = 1 k +1 (cid:80)
Revised: = 1 k +1 (cid:80)

Original: Tt =
Revised: Tt =

Original: T − k E [ F ( x t )] be the averaged expected values of the last k + 1 iterations.
Revised: T − k E [ F ( x t )] be the averaged expected values of the last k + 1 iterations.

Original: We are interested in S 0 − F ( x ∗ ) =
Revised: We are interested in S 0 − F ( x ∗ ) =

Original: E [ F ( x T )] − F ( x ∗ ) .
Revised: E [ F ( x T )] − F ( x ∗ ) .

Original: Now we derive an inequality between S k and S k − 1 .
Revised: Now we derive an inequality between S k and S k − 1 .

Original: By deﬁnition,
Revised: By deﬁnition,

Original: [Equation]
Revised: [Equation]

Original: Rearrange Eq 28{+
+} to upper bound − E [ x T − k ] ,
Revised: Rearrange Eq {+
+} to upper bound − E [ x T − k ] ,

Original: [Equation]
Revised: [Equation]

Original: t

Original: [Equation][Equation] [Equation] [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: − k t (cid:33) .
Revised: t (cid:33) .

Original: Summing over k = 1 , . .
Revised: Summing over k = 1 , . .

Original: k = (cid:98) T/ 2 (cid:99) ,
Revised: k = (cid:98) T/ 2 (cid:99) ,

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: (cid:98) T/ 2 (cid:99) (cid:88) k
Revised: (cid:98) [Equation] [Equation] (cid:99) (cid:88) k

Original: =  kT + (cid:98) T/ 2 (cid:99) (cid:88) k
Revised: =  kT + (cid:98) T/ 2 (cid:99) (cid:88) k

Original: =
Revised: =

Original: T (cid:88) t =
Revised: T (cid:88) t =

Original: T − k k ( k + 1) t 
Revised: T − k k ( k + 1) t 

Original:  .
Revised:  .

Original: Now we bound S (cid:98) T/ 2 (cid:99) − F ( x ∗ ) .
Revised: Now we bound S (cid:98) T/ 2 (cid:99) − F ( x ∗ ) .

Original: Choose x = x ∗ and η t = 1 tν in Eq 25 ,
Revised: Choose x = x ∗ and η t = 1 tν in Eq 24 ,

Original: T (cid:88) t
Revised: T (cid:88) t

Original: = (cid:100) T/ 2 (cid:101)
Revised: = (cid:100) T/ 2 (cid:101)

Original: E [ F ( x t ) − F ( x ∗ )] =
Revised: E [ F ( x t ) − F ( x ∗ )] =

Original: ν (cid:100) T/ 

Revised: ν (cid:100) T/

Original: (cid:101) E
Revised: (cid:101) E

Original: [ (cid:13)(cid:13) x (cid:100)
Revised: [ (cid:13)(cid:13) x (cid:100)

Original: T/ 2 (cid:101) − x ∗ (cid:13)(cid:13) 2 ]
Revised: T/ 2 (cid:101) − x ∗ (cid:13)(cid:13) 2 ]

Original: + ν
Revised: + ν

Original: T (cid:88) t
Revised: T (cid:88) t

Original: = (cid:100) T/ 2 (cid:101)
Revised: = (cid:100) T/ 2 (cid:101)

Original: +
Revised: +

Original: E [ (cid:107) x t − x ∗ (cid:107) 2 ] +
Revised: E [ (cid:107) x t − x ∗ (cid:107) 2 ] +

Original: [Equation]
Revised: [Equation]

Original: (cid:0)
Revised: (cid:0)

Original: [Equation]
Revised: [Equation]

Original: (cid:1)
Revised: (cid:1)

Original: T (cid:88) t
Revised: T (cid:88) t

Original: = (cid:100)
Revised: = (cid:100)

Original: [Equation]
Revised: [Equation]

Original: The second inequality uses Lemma 2.
Revised: The second inequality uses Lemma 2.

Original: The last inequality holds because the fact that (cid:80) Tt =
Revised: The last inequality holds because the fact that (cid:80) Tt =

Original: (cid:100)
Revised: (cid:100)

Original: T/ 2 (cid:101) 1 t ≤ log(2) .
Revised: T/ 2 (cid:101) 1 t ≤ log(2) .

Original: Dividing Eq 32 by (cid:100) T/ 2 (cid:101) ,
Revised: Dividing Eq 31 by (cid:100) T/ 2 (cid:101) ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We have (cid:80) (cid:98) T/ 2 (cid:99) k
Revised: We have (cid:80) (cid:98) T/ 2 (cid:99) k

Original: = 16 kT ≤ 16(
Revised: = 16 kT ≤ 16(

Original: [Equation]
Revised: [Equation]

Original: log ( T ))
Revised: log ( T ))

Original: [Equation]
Revised: [Equation]

Original: because it is harmonic sequence.
Revised: because it is harmonic sequence.

Original: Lastly, (cid:98) T/ 2 (cid:99) (cid:88) k
Revised: Lastly, (cid:98) T/ 2 (cid:99) (cid:88) k

Original: =
Revised: =

Original: T (cid:88) t =
Revised: T (cid:88) t =

Original: T − k k ( k + 1) t ≤ (cid:98) T/ 2 (cid:99) (cid:88) k
Revised: T − k k ( k + 1) t ≤ (cid:98) T/ 2 (cid:99) (cid:88) k

Original: = log(2)
Revised: = log(2)

Original: k ( k + 1) ≤ (cid:98) T/ 2 (cid:99) (cid:88) k
Revised: k ( k + 1) ≤ (cid:98) T/ 2 (cid:99) (cid:88) k

Original: = log(2)
Revised: = log(2)

Original: k≤ 2 log(2) .
Revised: k≤ 2 log(2) .

Original: Plugging these bounds into Eq 31, we have
Revised: Plugging these bounds into Eq 30, we have

Original: [Equation]
Revised: [Equation]

Original: (cid:19) .
Revised: (cid:19) .

Original: Choose σ 2 = Θ (cid:16) Tlog (1 /δ ) n 2 (cid:15) 2 (cid:17) to guarantee ( (cid:15), δ ) -DP.
Revised: Choose σ 2 = Θ (cid:16) Tlog (1 /δ ) n 2 (cid:15) 2 (cid:17) to guarantee ( (cid:15), δ ) -DP.

Original: Set T = n 2 (cid:15) 2 , we have
Revised: Set T = n 2 (cid:15) 2 , we have

Original: [Equation]
Revised: [Equation]

Original: (cid:19) .
Revised: (cid:19) .

Original: Set T = n(cid:15) √ p and assume p < n 2 , we have
Revised: Set T = n(cid:15) √ p and assume p < n 2 , we have

Original: [Equation]
Revised: [Equation]

Original: n(cid:15)ν
Revised: n(cid:15)ν

Original: [Equation]
Revised: [Equation]

Original: Appendix B Detailed description on benchmark datasets
Revised: Appendix B Detailed description on benchmark datasets

Original: [Table]
Revised: [Table]

Revised: Appendix C Comparison between Average and Minimum Curvatures on Different dataset

Revised: In this section we plot the average and minimum curvatures in Figure 4 for another dataset KDDCup99.

Revised: The objective function is still regularized logistic regression.

Revised: As shown in Figure 4, the average curvature is still larger than the minimum curvature (especially when the regularization term is small).

Revised: Despite this, the average curvature of KDDCup99 is smaller than Adult, this may be the reason why the improvement in Section 4 is larger for the Adult dataset.

Revised: [Figure]