Original: I MPROVED C ONVERGENCE OF D IFFERENTIAL P RIVATE SGD WITH G RADIENT C LIPPING
Revised: I MPROVED C ONVERGENCE OF D IFFERENTIAL P RIVATE SGD WITH G RADIENT C LIPPING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Differential private stochastic gradient descent (DP-SGD) with gradient clipping (DP-SGD-GC) is an effective optimization algorithm that can train machine learning models with a privacy guarantee.
Revised: Differential private stochastic gradient descent (DP-SGD) with gradient clipping (DP-SGD-GC) is an effective optimization algorithm that can train machine learning models with a privacy guarantee.

Original: Despite the popularity of DP-SGD-GC, its convergence in unbounded domain without the Lipschitz continuous assumption is less-understood; existing analysis of DP-SGD-GC either impose additional assumptions or end up with an utility bound that involves a non-vanishing bias term.
Revised: Despite the popularity of DP-SGD-GC, its convergence in unbounded domain without the Lipschitz continuous assumption is less-understood; existing analysis of DP-SGD-GC either impose additional assumptions or end up with an utility bound that involves a non-vanishing bias term.

Original: In this work, for smooth and unconstrained problems, we improve the current analysis and show that DP-SGD-GC can achieve a vanishing utility bound without any bias term.
Revised: In this work, for smooth and unconstrained problems, we improve the current analysis and show that DP-SGD-GC can achieve a vanishing utility bound without any bias term.

Original: Furthermore, when the noise generated from subsampled gradients is light-tailed, we prove that DP-SGD-GC can achieve nearly the same utility bound as DP-SGD applies to the Lipschitz continuous objectives.
Revised: Furthermore, when the noise generated from subsampled gradients is light-tailed, we prove that DP-SGD-GC can achieve nearly the same utility bound as DP-SGD applies to the Lipschitz continuous objectives.

Original: As a by-product, we propose a new clipping technique, called value clipping, to mitigate the computational overhead caused by the classic gradient clipping.
Revised: As a by-product, we propose a new clipping technique, called value clipping, to mitigate the computational overhead caused by the classic gradient clipping.

Original: Experiments on standard benchmark datasets are conducted to support our analysis.
Revised: Experiments on standard benchmark datasets are conducted to support our analysis.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Training machine learning models that can achieve decent prediction accuracy while preserving data privacy is fundamental in many modern machine learning applications.
Revised: Training machine learning models that can achieve decent prediction accuracy while preserving data privacy is fundamental in many modern machine learning applications.

Original: The concept of differential privacy (DP) from Dwork et al.
Revised: The concept of differential privacy (DP) from Dwork et al.

Original: (Dwork et al., 2006) offers an elegant mathematical framework to characterize the privacy-preserving ability of randomized algorithms.
Revised: (2006) offers an elegant mathematical framework to characterize the privacy-preserving ability of randomized algorithms.

Original: A huge line of works has been devoted to designing differential-private training algorithms for the empirical risk minimization (ERM) problem.
Revised: A huge line of works has been devoted to designing differential-private training algorithms for the empirical risk minimization (ERM) problem.

Original: Among many proposed methods, differential private stochastic gradient descent (DP-SGD) is an effective optimization algorithm that can solve ERM with a privacy guarantee and achieve a reasonable utility bound.
Revised: Among many proposed methods, differential private stochastic gradient descent (DP-SGD) is an effective optimization algorithm that can solve ERM with a privacy guarantee and achieve a reasonable utility bound.

Original: DP-SGD has received substantial interest in recent years due to its simplicity and effectiveness (Song et al., 2013; Bassily et al., 2014;
Revised: DP-SGD has received substantial interest in recent years due to its simplicity and effectiveness (Song et al., 2013; Bassily et al., 2014;

Original: Abadi et al., 2016; Wang et al., 2017; Bassily et al., 2019; Feldman et al., 2020; Asi et al., 2021).
Revised: Abadi et al., 2016; Wang et al., 2017; Bassily et al., 2019; Feldman et al., 2020; Asi et al., 2021).

Original: In the classic analysis of DP-SGD, the variance of the Gaussian noise used in each iteration of
Revised: In the classic analysis of DP-SGD, the variance of the Gaussian noise used in each iteration of

Original: DP-SGD relies crucially on the ℓ 2 -sensitivity of the loss function.
Revised: DP-SGD relies crucially on the ℓ 2 -sensitivity of the loss function.

Original: Therefore most early works on
Revised: Therefore most early works on

Original: DP-SGD assume each individual loss function to be Lipschitz continuous in its domain (Song et al., 2013; Bassily et al., 2014).
Revised: DP-SGD assume each individual loss function to be Lipschitz continuous in its domain (Song et al., 2013; Bassily et al., 2014).

Original: However, many real-world problems are only smooth but not globally
Revised: However, many real-world problems are only smooth but not globally

Original: Lipschitz continuous; for example, the unconstrained linear regression problem.
Revised: Lipschitz continuous; for example, the unconstrained linear regression problem.

Original: There are two techniques to circumvent the Lipschitz continuous assumption: (i) imposing an additional bounded domain constraint to the original problem; (ii) clipping each gradients in their 2-norm and using them to update the model (Abadi et al., 2016).
Revised: There are two techniques to circumvent the Lipschitz continuous assumption: (i) imposing an additional bounded domain constraint to the original problem; (ii) clipping each gradients in their 2-norm and using them to update the model (Abadi et al., 2016).

Original: In practice, the gradient clipping technique is usually more preferred than imposing a bounded domain constraint because the latter requires prior knowledge of the distance between initialization and solution, which is typically unavailable for unconstrained problems.
Revised: In practice, the gradient clipping technique is usually more preferred than imposing a bounded domain constraint because the latter requires prior knowledge of the distance between initialization and solution, which is typically unavailable for unconstrained problems.

Original: Indeed, the state-of-the-art implementations of DP-SGD including Opacus (Yousefpour et al., 2021) and TensorFlow Privacy 1 all advocate the gradient clipping technique.
Revised: Indeed, the state-of-the-art implementations of DP-SGD including Opacus (Yousefpour et al., 2021) and TensorFlow Privacy 1 all advocate the gradient clipping technique.

Original: Despite the popularity of DP-SGD with gradient clipping (DP-SGD-GC), the convergence of DP-SGDGC for unconstrained problems that are not globally Lipschitz continuous has not been well-studied.
Revised: Despite the popularity of DP-SGD with gradient clipping (DP-SGD-GC), the convergence of DP-SGDGC for unconstrained problems that are not globally Lipschitz continuous has not been well-studied.

Original: In fact, recent works (Chen et al., 2020; Song et al., 2021) have reported that DP-SGD-GC can suffer from a constant utility in the worst case.
Revised: In fact, recent works (Chen et al., 2020; Song et al., 2021) have reported that DP-SGD-GC can suffer from a constant utility in the worst case.

Original: With these negative results on the convergence of
Revised: With these negative results on the convergence of

Original: DP-SGD-GC, one may consider DP-SGD-GC as an algorithm with a fundamental non-convergence issue.
Revised: DP-SGD-GC, one may consider DP-SGD-GC as an algorithm with a fundamental non-convergence issue.

Original: In this work, we show that this is not the case.
Revised: In this work, we show that this is not the case.

Original: With a careful choice of the clipping threshold,
Revised: With a careful choice of the clipping threshold,

Original: [Table]
Revised: [Table]

Original: we prove that DP-SGD-GC can achieve the same utility bound as its non-clipped counterpart DP-SGD.
Revised: we prove that DP-SGD-GC can achieve the same utility bound as its non-clipped counterpart DP-SGD.

Original: Formally
Revised: Formally

Original: [Table]
Revised: [Table]

Original: we summarize our contributions as follows.
Revised: we summarize our contributions as follows.

Original: • For unconstrained problems that are convex and smooth but not necessarily globally Lipschitz continuous, we show that DP-SGD-GC can achieve a (cid:101) O ( √ d/ ( √ nϵ )) utility bound.
Revised: • For unconstrained problems that are convex and smooth but not necessarily globally Lipschitz continuous, we show that DP-SGD-GC can achieve a (cid:101) O ( √ d/ ( √ nϵ )) utility bound.

Original: When the noise generated from the subsampled gradients is light-tailed (Assumption 4.1) 2 , the utility bound can be improved to (cid:101) O ( √ d/ ( nϵ )) , which is the same as the utility bound of DP-SGD applies to the Lipschitz continuous problems.
Revised: When the noise generated from the subsampled gradients is light-tailed (Assumption 4.1) 2 , the utility bound can be improved to (cid:101) O ( √ d/ ( nϵ )) , which is the same as the utility bound of DP-SGD applies to the Lipschitz continuous problems.

Original: See Table 1 for a comparison to existing results.
Revised: See Table 1 for a comparison to existing results.

Original: Our convergence analysis of DP-SGD-GC for 
Revised: Our convergence analysis of DP-SGD-GC for convex, smooth and unconstrained problems

Original: convex, smooth and unconstrained problems, to our knowledge, provide the first utility bound without a non-vanishing bias term.
Revised: , to our knowledge, provide the first utility bound without a non-vanishing bias term.

Original: • We show that our analysis also applies to unconstrained smooth problems that can potentially be nonconvex.
Revised: • We show that our analysis also applies to unconstrained smooth problems that can potentially be nonconvex.

Original: Consequently, DP-SGD-GC can achieve a (cid:101) O ( √ d/ ( √ nϵ ))
Revised: Consequently, DP-SGD-GC can achieve a (cid:101) O ( √ d/ ( √ nϵ ))

Original: gradient norm bound for smooth problems.
Revised: gradient norm bound for smooth problems.

Original: Again, the bound can be improved to (cid:101) O ( √ d/ ( nϵ )) with an additional light-tail-noise assumption.
Revised: Again, the bound can be improved to (cid:101) O ( √ d/ ( nϵ )) with an additional light-tail-noise assumption.

Original: • This work is theoretical in essence but also includes a practical contribution (Section 5).
Revised: • This work is theoretical in essence but also includes a practical contribution (Section 5).

Original: We develop a novel value clipping technique for problems that satisfy the  growth condition (Definition 3.1).
Revised: We develop a novel value clipping technique for problems that satisfy the weak growth condition (Definition 3.1).

Original: The proposed value clipping technique can be implemented within one forwardbackward propagation on existing deep learning platforms and can alleviate the computation overhead caused by gradient clipping.
Revised: The proposed value clipping technique can be implemented within one forwardbackward propagation on existing deep learning platforms and can alleviate the computation overhead caused by gradient clipping.

Original: The efficiency of value clipping is demonstrated on real-world datasets.
Revised: The efficiency of value clipping is demonstrated on real-world datasets.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: DP-SGD with gradient clipping was initially proposed by Abadi et al.
Revised: DP-SGD with gradient clipping was initially proposed by Abadi et al.

Original: (Abadi et al., 2016).

Original: Gradient clipping and its variants have been widely adopted by many privacy-aware training algorithms (Yousefpour et al., 2021; Andrew et al., 2021).
Revised: Gradient clipping and its variants have been widely adopted by many privacy-aware training algorithms (Yousefpour et al., 2021; Andrew et al., 2021).

Original: Despite the popularity of gradient clipping, the convergence rate of DP-SGD-GC without the Lipschitz continuous and bounded domain assumptions remains a challenging task; see (Wang et al., 2022, Remark 5) for a short discussion on the hardness of removing the bounded domain assumption.
Revised: Despite the popularity of gradient clipping, the convergence rate of DP-SGD-GC without the Lipschitz continuous and bounded domain assumptions remains a challenging task; see (Wang et al., 2022, Remark 5) for a short discussion on the hardness of removing the bounded domain assumption.

Original: This challenging research question was not carefully studied until the recent works from Chen et al.
Revised: This challenging research question was not carefully studied until the recent works from Chen et al.

Original: (Chen et al., 2020) and Song et al.
Revised: (2020) and Song et al.

Original: (Song et al., 2021), who provided counter-examples showing that DP-SGD-GC can suffer from a constant utility in the worst case.
Revised: (2021), who provided counter-examples showing that DP-SGD-GC can suffer from a constant utility in the worst case.

Original: Chen et al.
Revised: Chen et al.

Original: (Chen et al., 2020) studied the convergence of DP-SGD-GC to a stationary point in the nonconvex setting and showed that an additional assumption on gradient distribution is sufficient to derive a meaningful utility bound.
Revised: (2020) studied the convergence of DP-SGD-GC to a stationary point in the nonconvex setting and showed that an additional assumption on gradient distribution is sufficient to derive a meaningful utility bound.

Original: Song et al.
Revised: Song et al.

Original: (Song et al., 2021) showed that DP-SGD-GC converges to a perturbed objective function for the generalized linear model and can suffer from a constant utility for the original objective in the worst case.
Revised: (2021) showed that DP-SGD-GC converges to a perturbed objective function for the generalized linear model and can suffer from a constant utility for the original objective in the worst case.

Original: Note there are some other recent works that study the convergence of DP-SGD-GC for smooth objective Du et al. 
Revised: Note there are some other recent works that study the convergence of DP-SGD-GC for smooth objective (Du et al., 2021; Wu et al., 2021; Yang et al., 2022), the rates in these works usually involve a bias term due to clipping. A concurrent work from Bu et al. (2022) suggests that a small clipping threshold can yield promising performance for DP-SGD-GC in certain scenarios, such as training language models. Their empirical discovery contrasts with the theoretical analysis in this work as our proof technique relies on a large clipping threshold.

Original: Wu et al.
Revised: Bu et al.

Original: (2021), the rates in these works usually involve a bias term due to clipping. Another line of work is using gradient clipping to avoid gradient explosion when training neural networks (Zhang et al., 2020b;a).
Revised: ’s experiments indicate that the analysis in this work may be further improvable; rigorous theoretical justification for the phenomenon described by Bu et al. (2022) is worth future investigation.

Original: These works clip the averaged gradient instead of clipping each individual gradient in the sampled batch, which is different from the gradient clipping used by DP-SGD when the batch size is greater than one.
Revised: Another concurrent work from Yang et al.

Original: The convergence  analysis  of clipping averaged gradient can not be applied directly to DP-SGD and does not overlap with this work.
Revised: (2022) studied the convergence of DP-SGD-GC under the generalized smooth condition, their analysis relies on a different set of assumptions and do not overlap with this work.

Original: On the practical side, the original implementation of gradient clipping technique was inefficient as one needs to calculate the norm of each individual sample in every iteration.
Revised: On the practical side, the original implementation of gradient clipping technique was inefficient as one needs to calculate the norm of each individual sample in every iteration.

Original: Many works (Goodfellow, 2015; Abadi et al., 2016; Rochette et al., 2019; Bu et al., 2021; Subramani et al., 2021) have been carried on to improve the efficiency of DP-SGD-GC from either engineering or algorithmic perspective.
Revised: Many works (Goodfellow, 2015; Abadi et al., 2016; Rochette et al., 2019; Bu et al., 2021; Subramani et al., 2021) have been carried on to improve the efficiency of DP-SGD-GC from either engineering or algorithmic perspective.

Original: Our proposed value clipping technique can be viewed as an alternative to the classic gradient clipping technique.
Revised: Our proposed value clipping technique can be viewed as an alternative to the classic gradient clipping technique.

Original: Notation Throughout this paper, for any positive integer n , we denote [ n ] := { 1 , 2 , . . . , n } .
Revised: Notation Throughout this paper, for any positive integer n , we denote [ n ] := { 1 , 2 , . . . , n } .

Original: We denote ∥ · ∥ to be the vector 2-norm or matrix operator norm if not otherwise specified.
Revised: We denote ∥ · ∥ to be the vector 2-norm or matrix operator norm if not otherwise specified.

Original: We use the notation (cid:101) O ( · ) to hide poly-logarithmic terms.
Revised: We use the notation (cid:101) O ( · ) to hide poly-logarithmic terms.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We consider the empirical-risk minimization (ERM) problem
Revised: We consider the empirical-risk minimization (ERM) problem

Original: [Equation]
Revised: [Equation]

Original: where n is the number of training samples, f i ’s are differentiable functions and w is the model we wish to train.
Revised: where n is the number of training samples, f i ’s are differentiable functions and w is the model we wish to train.

Original: Throughout the paper, we assume that f is bounded below and its minimum is attainable.
Revised: Throughout the paper, we assume that f is bounded below and its minimum is attainable.

Original: We let W ∗ to be the set of solutions for problem (P), and denote the optimal function value as f ∗ .
Revised: We let W ∗ to be the set of solutions for problem (P), and denote the optimal function value as f ∗ .

Original: We assume that a lower bound of f ∗ is known as a prior.
Revised: We assume that a lower bound of f ∗ is known as a prior.

Original: Note that this assumption holds in many realistic settings, for example the ERM problems are usually lower bounded by zero.
Revised: Note that this assumption holds in many realistic settings, for example the ERM problems are usually lower bounded by zero.

Original: Next we introduce the  growth condition (GC), which serves as the cornerstone for our analysis.
Revised: Next we introduce the weak growth condition (WGC), which is the cornerstone for our analysis.

Original: Definition 3.1.
Revised: Definition 3.1.

Original: A function h : R d → R is ( β 1 , β 2 ) -GC for some β 1 > 0 , β 2 ≥ 

Revised: A function h : R d → R is ( β 1 , β 2 ) -WGC for some β 1 > 0 , β 2 ≥

Original: if
Revised: if

Original: [Equation]

Original: ( w )

Original: [Equation]
Revised: [Equation]

Original: The  growth condition bounds the norm of the gradient by a linear function of the objective value.
Revised: The weak growth condition bounds the norm of the gradient by a linear function of the objective value.

Original: GC is gaining increasing interest in recent years as multiple works have demonstrated that GC and its variants can improve the classic analysis of SGD-type algorithms (Schmidt & Le Roux, 2013; Needell et al., 2016; Vaswani et al., 2019; Gower et al., 2019; Stich, 2019; Khaled & Richtárik, 2020; Fang et al., 2021; Gower et al., 2021).
Revised: WGC is gaining increasing interest in recent years as multiple works have demonstrated that WGC and its variants can improve the classic analysis of SGD-type algorithms (Schmidt & Le Roux, 2013; Needell et al., 2016; Vaswani et al., 2019; Gower et al., 2019; Stich, 2019; Khaled & Richtárik, 2020; Fang et al., 2021; Gower et al., 2021).

Original: It is easy to show that smooth functions that are bounded below necessarily satisfy GC; the following Lemma makes this precise.
Revised: It is easy to show that smooth functions that are bounded below necessarily satisfy WGC; the following Lemma makes this precise.

Original: Lemma 3.2.
Revised: Lemma 3.2.

Original: If a function h : R d → R is L -smooth for some L > 0 and bounded below e.g., inf w ∈ R d h ( w ) >
Revised: If a function h : R d → R is L -smooth for some L > 0 and bounded below e.g., inf w ∈ R d h ( w ) >

Original: −∞ .
Revised: −∞ .

Original: Then h is (2 L, 0) -GC.
Revised: Then h is (2 L, 0) -WGC.

Original: We stress that GC is not a strong assumption.
Revised: We stress that WGC is not a strong assumption.

Original: In fact, a wide range of nonconvex and nonsmooth problems arising from the empirical-risk minimization problem also satisfy GC, for example the
Revised: In fact, a wide range of nonconvex and nonsmooth problems arising from the empirical-risk minimization problem also satisfy WGC, for example the

Original: Lipschitz continuous model with smooth and convex loss function; see Fang et al.
Revised: Lipschitz continuous model with smooth and convex loss function; see Fang et al.

Original: (Fang et al., 2021, § 4.1) and Section E for more details.
Revised: (Fang et al., 2021, § 4.1) and Section E for more details.

Original: We recall the standard definition of differential privacy (DP).
Revised: We recall the standard definition of differential privacy (DP).

Original: Definition 3.3 (Dwork et al., 2006) .
Revised: Definition 3.3 (Dwork et al., 2006) .

Original: A randomized algorithm A is ( ϵ, δ ) -differentially private if for all neighboring datasets D, D ′ and for all events S in the output space of A , we have
Revised: A randomized algorithm A is ( ϵ, δ ) -differentially private if for all neighboring datasets D, D ′ and for all events S in the output space of A , we have

Original: [Equation]
Revised: [Equation]

Original: If δ = 0 , then A is said to be ϵ -differentially private.
Revised: If δ = 0 , then A is said to be ϵ -differentially private.

Original: The detailed algorithm of DP-SGD-GC is shown in Algorithm 1.
Revised: The detailed algorithm of DP-SGD-GC is shown in Algorithm 1.

Original: It has been shown that DP-SGD-GC is ( ϵ, δ ) -DP
Revised: It has been shown that DP-SGD-GC is ( ϵ, δ ) -DP

Original: as long as the noise level σ is larger than certain threshold (Theorem 3.4).
Revised: as long as the noise level σ is larger than certain threshold (Theorem 3.4).

Original: Theorem 3.4 (Abadi et al., 2016, Theorem 1) .
Revised: Theorem 3.4 (Abadi et al., 2016, Theorem 1) .

Original: Let q = B/n , where B is the batch size and n is the number of data points.
Revised: Let q = B/n , where B is the batch size and n is the number of data points.

Original: There exist constants c 1 and c 2 , such that for any ϵ < c 1 q 2 T , Algorithm 1 is ( ϵ, δ ) -DP for any δ > 

Revised: There exist constants c 1 and c 2 , such that for any ϵ < c 1 q 2 T , Algorithm 1 is ( ϵ, δ ) -DP for any δ >

Original: if
Revised: if

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  M AIN THEORETICAL RESULTS
Revised:  M AIN THEORETICAL RESULTS

Original: We present our main theoretical contributions in this section.
Revised: We present our main theoretical contributions in this section.

Original: Denote w priv as the output of Algorithm 1.
Revised: Denote w priv as the output of Algorithm 1.

Original: We are interested in the upper bound of the excess empirical risk f ( w priv ) − f ∗ and the gradient norm square ∥∇ f
Revised: We are interested in the upper bound of the excess empirical risk f ( w priv ) − f ∗ and the gradient norm square ∥∇ f

Original: ( w priv ) ∥ 2 without assuming f i ’s to be Lipschitz continuous in R d .
Revised: ( w priv ) ∥ 2 without assuming f i ’s to be Lipschitz continuous in R d .

Original: Part of our analysis relies on assuming the noise generated from subsampled gradients is “light-tailed” (sub-Gaussian).
Revised: Part of our analysis relies on assuming the noise generated from subsampled gradients is “light-tailed” (sub-Gaussian).

Original: Formally, we introduce the following assumption 3 .
Revised: Formally, we introduce the following assumption 3 .

Original: Assumption 4.1.
Revised: Assumption 4.1.

Original: There exist ρ > 0 such that
Revised: There exist ρ > 0 such that

Original: [Equation]
Revised: [Equation]

Original: We note that the above light-tail assumption is a widely used assumption for the analysis of high probability utility bound of SGD (Nemirovski et al., 2009; Juditsky & Nesterov, 2014; Ghadimi &
Revised: We note that the above light-tail assumption is a widely used assumption for the analysis of high probability utility bound of SGD (Nemirovski et al., 2009; Juditsky & Nesterov, 2014; Ghadimi &

Original: Lan, 2013).
Revised: Lan, 2013; Harvey et al., 2019; Feldman et al., 2020).

Original: Assumption 4.1 does not imply f i ’s to be globally Lipschitz continuous on R d , and therefore will not trivialize our analysis.
Revised: Assumption 4.1 does not imply f i ’s to be globally Lipschitz continuous on R d , and therefore will not trivialize our analysis.

Revised: There is a recent trend on analyzing SGD and its variants with heavy-tail gradient noise (Gürbüzbalaban et al., 2021).

Revised: When the gradient noise is heavy tailed, we obtain a weaker utility bound (Theorem 4.6) compared with bound with light-tailed noise.

Original: The idea of our proof is concise.
Revised: The idea of our proof is concise.

Original: We first summarize the proof sketch as follows, and then explain the technical details step by step.
Revised: We first summarize the proof sketch as follows, and then explain the technical details step by step.

Original: Proof sketch:
Revised: Proof sketch:

Original: • Assuming that the initial objective gap f ( w (0) ) − f ∗ is bounded.
Revised: • Assuming that the initial objective gap f ( w (0) ) − f ∗ is bounded.

Original: We are able to prove that, with high probability, the iterates generated from SGD have bounded objective values that only logarithmically depend on T .
Revised: We are able to prove that, with high probability, the iterates generated from SGD have bounded objective values that only logarithmically depend on T .

Original: • The  growth condition allows us to convert the objective value upper bound to the gradient norm upper bound.
Revised: • The weak growth condition allows us to convert the objective value upper bound to the gradient norm upper bound.

Original: Therefore, with high probability, gradient clipping will never happen during the optimization process of DP-SGD if the clipping threshold C is chosen appropriately.
Revised: Therefore, with high probability, gradient clipping will never happen during the optimization process of DP-SGD if the clipping threshold C is chosen appropriately.

Original: • Finally, we can apply the classic convergence analysis of SGD (without gradient clipping) and obtain a non-trivial excess empirical risk or gradient norm upper bound.
Revised: • Finally, we can apply the classic convergence analysis of SGD (without gradient clipping) and obtain a non-trivial excess empirical risk or gradient norm upper bound.

Original: To begin with, we develop a uniform upper bound on the objective values, e.g., f ( w ( t ) ) where w ( t ) ’s are the iterates generated from the vanilla SGD algorithm (Algorithm 2) with sub-Gaussian noise.
Revised: To begin with, we develop a uniform upper bound on the objective values, e.g., f ( w ( t ) ) where w ( t ) ’s are the iterates generated from the vanilla SGD algorithm (Algorithm 2) with sub-Gaussian noise.

Original: Proposition 4.2 (Uniform upper bound on objective values of SGD with sub-Gaussian noise) .
Revised: Proposition 4.2 (Uniform upper bound on objective values of SGD with sub-Gaussian noise) .

Original: Assume f is L -smooth for some L > 

Revised: Assume f is L -smooth for some L >

Original: and there exist ˜ σ > 0 such that E [exp( ∥ ζ ( t ) ∥ 2 / ˜ σ 2 )] ≤ e for any t ∈ N .
Revised: and there exist ˜ σ > 0 such that E [exp( ∥ ζ ( t ) ∥ 2 / ˜ σ 2 )] ≤ e for any t ∈ N .

Original: Denote { w ( t ) } [-Tt =
-] as the iterates generated from Algorithm 2 with η ≤ min (cid:110) 12 L , 1 ˜ σ √ T (cid:111) .
Revised: Denote { w ( t ) } [-Tt =
-]T t =0 as the iterates generated from Algorithm 2 with η ≤ min (cid:110) 12 L , 1 ˜ σ √ T (cid:111) .

Original: Then for any δ ∈ (0 ,1) ,
Revised: Then for any δ ∈ (0 ,

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − δ 
Revised: with probability at least 1 − δ .

Original: The conclusion stated in Proposition 4.2 may seem obvious at first glance as the training algorithm is expected to produce almost decreasing objective values during the optimization process.
Revised: The conclusion stated in Proposition 4.2 may seem obvious at first glance as the training algorithm is expected to produce almost decreasing objective values during the optimization process.

Original: However, we note that deriving a nearly constant upper bound of f ( w ( t ) )
Revised: However, we note that deriving a nearly constant upper bound of f ( w ( t ) )

Original: − f ∗ that holds uniformly over t ∈ { 0 , 1 . .
Revised: − f ∗ that holds uniformly over t ∈ { 0 , 1 . .

Original: , T } without assuming Lipschitz continuity or bounded domain is nontrivial.
Revised: , T } without assuming Lipschitz continuity or bounded domain is nontrivial.

Original: Our proof relies on a recently proposed technical tool called the generalized Freedman inequality (Harvey et al., 2019); see Section C.1 for the detailed proof of Proposition 4.2.
Revised: Our proof relies on a recently proposed technical tool called the generalized Freedman inequality (Harvey et al., 2019); see Section C.1 for the detailed proof of Proposition 4.2.

Original: We also remark that Proposition 4.
Revised: We also remark that Proposition 4.

Original: holds for the standard SGD algorithm without considering differential privacy, and thus may be of independent interest.
Revised: holds for the standard SGD algorithm without considering differential privacy, and thus may be of independent interest.

Original: Based on Proposition 4.2, we can further obtain an upper bound on each individual loss f i ( w ( t ) )
Revised: Based on Proposition 4.2, we can further obtain an upper bound on each individual loss f i ( w ( t ) )

Original: − f ∗ i under Assumption 4.1, and therefore also upper bound ∥∇ f i ( w ( t ) )
Revised: − f ∗ i under Assumption 4.1, and therefore also upper bound ∥∇ f i ( w ( t ) )

Original: ∥ via the  growth condition.
Revised: ∥ via the weak growth condition.

Original: Proposition 4.3.
Revised: Proposition 4.3.

Original: Assume f
Revised: Assume f

Original: i ’s are L -smooth for some L > 0 and there exist ˜ σ > 0 such that E [exp( ∥ ζ ( t ) ∥ 2 / ˜ σ 2 )] ≤ e for any t ∈ N .
Revised: i ’s are L -smooth for some L > 0 and there exist ˜ σ > 0 such that E [exp( ∥ ζ ( t ) ∥ 2 / ˜ σ 2 )] ≤ e for any t ∈ N .

Original: Denote { w ( t ) } Tt =

Revised: Denote { w ( t ) } Tt =

Original: as the iterates generated from Algorithmwith η ≤ min (cid:110) 12 L , 1 ˜ σ √ T (cid:111) .
Revised: as the iterates generated from Algorithmwith η ≤ min (cid:110) 12 L , 1 ˜ σ √ T (cid:111) .

Original: Then for any δ ∈ (0 , 1) ,
Revised: Then for any δ ∈ (0 , 1) ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: holds with probability at least 1 − δ .
Revised: holds with probability at least 1 − δ .

Original: Furthermore, when Assumption 4.1 is satisfied, 
Revised: Furthermore, when Assumption 4.1 is satisfied, max i ∈ [ n ] , [Equation] ∈

Original: [Equation]
Revised: [Equation]

Original: holds with probability at least 1 − δ for any δ ∈ (0 , 1) .
Revised: holds with probability at least 1 − δ for any δ ∈ (0 , 1) .

Original: When Assumption 4.1 holds, Proposition 4.3 suggests that the upper bound of the maximum gradient norm logarithmically depends on δ, T and n (eq.
Revised: When Assumption 4.1 holds, Proposition 4.3 suggests that the upper bound of the maximum gradient norm logarithmically depends on δ, T and n (eq.

Original: When Assumption 4.1 does not hold, there is an addtional √ n term appears in eq.
Revised: When Assumption 4.1 does not hold, there is an addtional √ n term appears in eq.

Original: (1), which eventually causes a √ n loss in the utility bound.
Revised: (1), which eventually causes a √ n loss in the utility bound.

Original: Now we are ready to present our excess empirical risk bound and gradient norm bound for DP-SGD in terms of conditional expectation.
Revised: Now we are ready to present our excess empirical risk bound and gradient norm bound for DP-SGD in terms of conditional expectation.

Original: Due to space limitation, we only present the proposition with
Revised: Due to space limitation, we only present the proposition with

Original: Assumption 4.1 in the main text and place the proposition without Assumption 4.1 in Appendix B.
Revised: Assumption 4.1 in the main text and place the proposition without Assumption 4.1 in Appendix B.

Original: Proposition 4.4 (Convergence on conditional expectation) .
Revised: Proposition 4.4 (Convergence on conditional expectation) .

Original: Assume that Assumption 4.1 holds.
Revised: Assume that Assumption 4.1 holds.

Original: Denote w priv as the output of Algorithm 1 and define E := {∥ f i ( w ( t ) )
Revised: Denote w priv as the output of Algorithm 1 and define E := {∥ f i ( w ( t ) )

Original: ∥ ≤ C ∀ i ∈ [ n ] , t ∈ { 0 , 1 . . .
Revised: ∥ ≤ C ∀ i ∈ [ n ] , t ∈ { 0 , 1 . . .

Original: , T }} as the event of no clipping happens during the training of Algorithm 1.
Revised: , T }} as the event of no clipping happens during the training of Algorithm 1.

Original: Let D f := f ( w (0) )
Revised: Let D f := f ( w (0) )

Original: − f ∗ .
Revised: − f ∗ .

Original: Given any ϵ > 0 and δ, δ ′ ∈ (0  , 1) .
Revised: Given any ϵ > 0 and δ, δ ′ ∈ (0 . 5 , 1) .

Original: • Assume f
Revised: • Assume f

Original: i ’s are convex and L -smooth for some L > 0 .
Revised: i ’s are convex and L -smooth for some L > 0 .

Original: set T > ϵ/c 1 ,
Revised: set T > ϵ/c 1 ,

Original: [Equation]
Revised: [Equation]

Original: where c 1 , c 2 , c 5 are some absolute constants and c 3 , c 4 are constants that depend on L and D f .
Revised: where c 1 , c 2 , c 5 are some absolute constants and c 3 , c 4 are constants that depend on L and D f .

Original: We have that Algorithm 1 is ( ϵ, δ ) -DP, Pr[ E ] ≥ 1 − δ ′ , and
Revised: We have that Algorithm 1 is ( ϵ, δ ) -DP, Pr[ E ] ≥ 1 − δ ′ , and

Original: [Equation]
Revised: [Equation]

Original: • Assume f
Revised: • Assume f

Original: i ’s are L -smooth for some L > 0 .
Revised: i ’s are L -smooth for some L > 0 .

Original: Setting ϵ, σ, C, η as in eq.
Revised: Setting ϵ, σ, C, η as in eq.

Original: It holds that Algorithmis ( ϵ, δ ) -DP, Pr[ E ] ≥ 1 − δ ′ , and
Revised: It holds that Algorithmis ( ϵ, δ ) -DP, Pr[ E ] ≥ 1 − δ ′ , and

Original: [Equation]
Revised: [Equation]

Original: While the bounds stated in Proposition 4.4 are close to our objective, we note that these bounds are expressed in terms of conditional expectation where the conditioning event happens with high probability, which is different from the classic notion of convergence in expectation.
Revised: While the bounds stated in Proposition 4.4 are close to our objective, we note that these bounds are expressed in terms of conditional expectation where the conditioning event happens with high probability, which is different from the classic notion of convergence in expectation.

Original: Fortunately, we show that it is easy to convert the bounds in Proposition 4.4 to expected utility bound when the random variable f
Revised: Fortunately, we show that it is easy to convert the bounds in Proposition 4.4 to expected utility bound when the random variable f

Original: ( w priv ) − f ∗ (or ∥∇ f ( w priv ) ∥ 2 ) is sub-exponential, which is true under Assumption 4.1.
Revised: ( w priv ) − f ∗ (or ∥∇ f ( w priv ) ∥ 2 ) is sub-exponential, which is true under Assumption 4.1.

Original: Lemma A.8 serves as the main technical tool for this conversion.
Revised: Lemma A.9 serves as the main technical tool for this conversion.

Original: Theorem 4.5 (Convergence on expectation) .
Revised: Theorem 4.5 (Convergence on expectation) .

Original: Suppose that Assumption 4.1 holds.
Revised: Suppose that Assumption 4.1 holds.

Original: Denote w priv as the output of Algorithm 1.
Revised: Denote w priv as the output of Algorithm 1.

Original: Let D f := f ( w (0) )
Revised: Let D f := f ( w (0) )

Original: − f ∗ .
Revised: − f ∗ .

Original: Given any ϵ > 0 and δ ∈ (0  , 1) .
Revised: Given any ϵ > 0 and δ ∈ (0 . 5 , 1) .

Original: Setting T, σ, η in the same way as eq.
Revised: Setting T, σ, η in the same way as eq.

Original: (3) and let C = (cid:

Revised: (3) and let C = (cid:

Original: ) c
Revised: ) c

Original: 3 + c 4 log( nT ) , where c 1 , c 2 , c 5 are some absolute constants and
Revised: 3 + c 4 log( nT ) , where c 1 , c 2 , c 5 are some absolute constants and

Original: c 3 , c 4 are constants that depend on L, D f .
Revised: c 3 , c 4 are constants that depend on L, D f .

Original: • Assume f
Revised: • Assume f

Original: i ’s are convex and L -smooth for some L > 0 .
Revised: i ’s are convex and L -smooth for some L > 0 .

Original: Then Algorithm 1 is ( ϵ, δ ) -DP
Revised: Then Algorithm 1 is ( ϵ, δ ) -DP

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Consequently, we have E[ f
Revised: Consequently we have E

Original: [Equation] w 
Revised: f ( w priv ) − f

Original: [Equation]
Revised: [Equation]

Revised: ] =

Original: [Equation]
Revised: [Equation]

Original: nϵ
Revised: O

Original: [Equation] 
Revised: [Equation] [Equation]

Revised: nϵ )

Original: [Equation]
Revised: [Equation]

Original:  by setting T =
Revised: 1 (cid:1) by setting T =

Original: Θ (cid:0) n 2 ϵ 2 d − 1 (cid:1) .
Revised: Θ (cid:0) n 2 ϵ 2 d − 1 (cid:1) .

Original: • Assume f
Revised: • Assume f

Original: i ’s are L -smooth for some L > 0 .
Revised: i ’s are L -smooth for some L > 0 .

Original: Then Algorithm 1 is ( ϵ, δ ) -DP
Revised: Then Algorithm 1 is ( ϵ, δ ) -DP

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Consequently, we have E (cid:2) ∥∇ f ( w priv ) ∥ 2 (cid:3) =
Revised: Consequently, we have E (cid:2) ∥∇ f ( w priv ) ∥ 2 (cid:3) =

Original: (cid:101) O (cid:0) d 1 / 2 ( nϵ ) − 1 (cid:1) by setting T =
Revised: (cid:101) O (cid:0) d 1 / 2 ( nϵ ) − 1 (cid:1) by setting T =

Original: Θ (cid:0) n 2 ϵ 2 d − 1 (cid:1) .
Revised: Θ (cid:0) n 2 ϵ 2 d − 1 (cid:1) .

Original: Theorem 4.6 (Convergence without light-tail noise, informal) .
Revised: Theorem 4.6 (Convergence without light-tail noise, informal) .

Original: Under the same conditions as
Revised: Under the same conditions as

Original: Theorem 4.5 but without Assumption 4.1.
Revised: Theorem 4.5 but without Assumption 4.1.

Original: We have E [ f ( w priv ) − f ∗ ] = (cid:101) O (cid:0) d 1 / 2 ( √ nϵ ) − 1 (cid:1) and
Revised: We have E [ f ( w priv ) − f ∗ ] = (cid:101) O (cid:0) d 1 / 2 ( √ nϵ ) − 1 (cid:1) and

Original: E (cid:2) ∥∇ f ( w priv ) [Equation]
Revised: E (cid:2) ∥∇ f ( w priv ) ∥ 2 (cid:3)

Original: (cid:3) = (cid:101) O (cid:0) [Equation] √ nϵ ) − 1 (cid:1) by setting T =
Revised: = (cid:101) O (cid:0) d 1 / 2 ( √ nϵ ) − 1 (cid:1) by setting T =

Original: Θ (cid:0) nϵ 2 d − 1 (cid:1) .
Revised: Θ (cid:0) nϵ 2 d − 1 (cid:1) .

Original: Remark 4.1.
Revised: Remark 4.1.

Original: Our results suggest that, when the problem is smooth, the Lipschitz continuous and bounded domain assumptions can be removed almost for free when analyzing DP-SGD-GC with light-tailed gradient noise; the only cost is some logarithmic terms.
Revised: Our results suggest that, when the problem is smooth, the Lipschitz continuous and bounded domain assumptions can be removed almost for free when analyzing DP-SGD-GC with light-tailed gradient noise; the only cost is some logarithmic terms.

Original: When the gradient noise is not necessarily light-tailed, there is a loss of √ n in the utility bound.
Revised: When the gradient noise is not necessarily light-tailed, there is a loss of √ n in the utility bound.

Original: Theorem 4.5 and Theorem 4.6 are the main theoretical contributions of this paper.
Revised: Theorem 4.5 and Theorem 4.6 are the main theoretical contributions of this paper.

Original: The bounds stated in Theorem 4.5 nearly match the rate O (log(1 /δ )
Revised: The bounds stated in Theorem 4.5 nearly match the rate O (log(1 /δ )

Original: d 1 / 2 ( nϵ ) − 1 ) , which is the best-known bound of DP-SGD with the Lipschitz continuous or bounded domain assumption (Bassily et al., 2014).
Revised: d 1 / 2 ( nϵ ) − 1 ) , which is the best-known bound of DP-SGD with the Lipschitz continuous or bounded domain assumption (Bassily et al., 2014).

Original: When the Assumption 4.1 does not hold, the bound stated in Theorem 4.6 is worse than the rate
Revised: When the Assumption 4.1 does not hold, the bound stated in Theorem 4.6 is worse than the rate

Original: O (log(1 /δ )
Revised: O (log(1 /δ )

Original: d 1 / 2 ( nϵ ) − 1 ) with a factor of √ n .
Revised: d 1 / 2 ( nϵ ) − 1 ) with a factor of √ n .

Original: To our knowledge, for unconstrained smooth problems, this is the first utility bound of DP-SGD-GC without a non-vanishing bias term.
Revised: To our knowledge, for unconstrained smooth problems, this is the first utility bound of DP-SGD-GC without a non-vanishing bias term.

Original: Note that existing lower bound analyses for DP-SGD either assume the domain is bounded (Bassily et al., 2014) or the loss is Lipschitz continuous (Song et al., 2021).
Revised: Note that existing lower bound analyses for DP-SGD either assume the domain is bounded (Bassily et al., 2014) or the loss is Lipschitz continuous (Song et al., 2021).

Original: Therefore those lower bounds are not comparable with the upper bounds stated in Theorem 4.5.
Revised: Therefore those lower bounds are not comparable with the upper bounds stated in Theorem 4.5.

Original: We leave the lower bound of DP-SGD-GC for unconstrained smooth problems as a future direction to explore.
Revised: We leave the lower bound of DP-SGD-GC for unconstrained smooth problems as a future direction to explore.

Original:  V ALUE CLIPPING
Revised:  V ALUE CLIPPING

Original: This section focuses on the practical side of DP-SGD-GC.
Revised: This section focuses on the practical side of DP-SGD-GC.

Original: A well-known implementation issue of
Revised: A well-known implementation issue of

Original: Algorithm 1 is that the gradient clipping step (line 4 of Algorithm 1) requires to access the norm of each individual gradient from the sampled batch, and naive implementation of DP-SGD-GC on current deep learning platforms cannot fully exploit the parallelism of GPU; see some attempts that try to mitigate this issue (Goodfellow, 2015; Rochette et al., 2019; Bu et al., 2021).
Revised: Algorithm 1 is that the gradient clipping step (line 4 of Algorithm 1) requires to access the norm of each individual gradient from the sampled batch, and naive implementation of DP-SGD-GC on current deep learning platforms cannot fully exploit the parallelism of GPU; see some attempts that try to mitigate this issue (Goodfellow, 2015; Rochette et al., 2019; Bu et al., 2021).

Original: The state-of-the-artw t ∗ β implementation of DP-SGD-GC is from Subramani et al.
Revised: The state-of-the-art implementation of DP-SGD-GC is from Subramani et al.

Original: (Subramani et al., 2021), who developed a highly engineered approach to exploit language primitives, compilation, and vectorization on certain deep learning platforms.
Revised: (2021), who developed a highly engineered approach to exploit language primitives, compilation, and vectorization on certain deep learning platforms.

Original: In this section, we propose a value clipping technique for functions that satisfy the growth condition (Definition 3.1).
Revised: In this section, we propose a value clipping technique for functions that satisfy the weak [Equation] [Equation] [Equation]

Revised: [Equation]

Original:  The proposed value clipping can be viewed as an alternative to the classic gradient clipping technique that is easy to implement on all existing deep learning platforms such as PyTorch (Paszke et al., 2019), Tensorflow (Abadi et al., 2015) and MXNet (Chen et al., 2015).
Revised: [Equation] [Equation] growth condition (Definition 3.1). The proposed value clipping can be viewed as an alternative to the classic gradient clipping technique that is easy to implement on all existing deep learning platforms such as PyTorch (Paszke et al., 2019), Tensorflow (Abadi et al., 2015) and MXNet (Chen et al., 2015).

Original: The intuition behind value clipping is simple — when f i ’s satisfy the  growth condition, the norm of their gradients can be bounded by a function of their objective values, e.g., ∥∇ f i ( w ) ∥ ≤ (cid:112) β 1 ( f i ( w ) − f ∗ i ) + β 2 , therefore scaling the gradient by (cid:112)
Revised: The intuition behind value clipping is simple — when f i ’s satisfy the weak growth condition, the norm of their gradients can be bounded by a function of their objective values, e.g., ∥∇ f i ( w ) ∥ ≤ (cid:112) β 1 ( f i ( w ) − f ∗ i ) + β 2 , therefore scaling the gradient by (cid:112)

Original: β 1 ( f i ( w ) − f ∗ i ) + β 2 ensures the scaled gradient has a bounded norm.
Revised: β 1 ( f i ( w ) − f ∗ i ) + β 2 ensures the scaled gradient has a bounded norm.

Original: Formally,
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original: The detailed algorithm of DP-SGD with value clipping, termed DP-SGD-VC, is shown in Algorithm 3.
Revised: The detailed algorithm of DP-SGD with value clipping, termed DP-SGD-VC, is shown in Algorithm 3.

Original: Note that DP-SGD-VC requires knowing the GC parameters β 1 , β 2 and a lower bound of f ∗ i ’s as its input.
Revised: Note that DP-SGD-VC requires knowing the WGC parameters β 1 , β 2 and a lower bound of f ∗ i ’s as its input.

Original: The GC parameters for simple models, including linear and logistic regression, are easy to calculate.
Revised: The WGC parameters for simple models, including linear and logistic regression, are easy to calculate.

Original: For feed-forward neural networks, the calculation of GC parameters is achievable but more involved; see Appendix E for details.
Revised: For feed-forward neural networks, the calculation of WGC parameters is achievable but more involved; see Appendix E for details.

Original: It is easy to verify that DP-SGD-VC is ( ϵ, δ ) -DP because the norm of the clipped gradient is guaranteed to be bounded by C ; the following corollary is a direct consequence of Theorem 3.4 and describes the DP property of DP-SGD-VC.
Revised: It is easy to verify that DP-SGD-VC is ( ϵ, δ ) -DP because the norm of the clipped gradient is guaranteed to be bounded by C ; the following corollary is a direct consequence of Theorem 3.4 and describes the DP property of DP-SGD-VC.

Original: Corollary 5.1.
Revised: Corollary 5.1.

Original: Assume f
Revised: Assume f

Original: i ’s are ( β 1 , β 2 ) -GC for some β 1 > 0 , β 2 ≥ 0 .
Revised: i ’s are ( β 1 , β 2 ) -WGC for some β 1 > 0 , β 2 ≥ 0 .

Original: Let q = B/n , where B is the batch size and n is the number of data points.
Revised: Let q = B/n , where B is the batch size and n is the number of data points.

Original: There exist constants c 1 and c 2 , such that for any ϵ < c 1 q 2 T , Algorithm 3 is ( ϵ, δ ) -DP for any δ > 0 if σ ≥ c 2 q (cid:112) T log(1 /δ )
Revised: There exist constants c 1 and c 2 , such that for any ϵ < c 1 q 2 T , Algorithm 3 is ( ϵ, δ ) -DP for any δ > 0 if σ ≥ c 2 q (cid:112) T log(1 /δ )

Original: ϵ − 1 .
Revised: ϵ − 1 .

Original: Remark 5.1.
Revised: Remark 5.1.

Original: DP-SGD-VC is easy to implement on existing auto-differentiation based deep learning platforms.
Revised: DP-SGD-VC is easy to implement on existing auto-differentiation based deep learning platforms.

Original: The value clipping step (line 4 of Algorithm 3) can be realized within one forwardbackward propagation if the GC parameters are given in advance.
Revised: The value clipping step (line 4 of Algorithm 3) can be realized within one forwardbackward propagation if the WGC parameters are given in advance.

Original: Therefore DP-SGD-VC can be as fast as the vanilla SGD algorithm.
Revised: Therefore DP-SGD-VC can be as fast as the vanilla SGD algorithm.

Original:  N UMERICAL STUDY
Revised:  N UMERICAL STUDY

Original: We conduct experiments on two standard image classification benchmark datasets: MNIST (LeCun & Cortes, 2011) and CIFAR10 (Krizhevsky, 2009).
Revised: We conduct experiments on two standard image classification benchmark datasets: MNIST (LeCun & Cortes, 2011) and CIFAR10 (Krizhevsky, 2009).

Original: In Appendix, we also present some experimental results on synthetic data with light-tailed noise.
Revised: In Appendix, we also present some experimental results on synthetic data with light-tailed noise.

Original: For MNIST, we train a linear classifier and a twolayer MLP with 128 hidden nodes respectively.
Revised: For MNIST, we train a linear classifier and a twolayer MLP with 128 hidden nodes respectively.

Original: For CIFAR10, to achieve decent accuracy, we use a pre-trained VGG16 network (Simonyan & Zisserman, 2015) to extract informative high-level features.
Revised: For CIFAR10, to achieve decent accuracy, we use a pre-trained VGG16 network (Simonyan & Zisserman, 2015) to extract informative high-level features.

Original: Based on the 512-dimensional extracted features, we train a linear classifier and a two-layer MLP with 128 hidden nodes respectively.
Revised: Based on the 512-dimensional extracted features, we train a linear classifier and a two-layer MLP with 128 hidden nodes respectively.

Original: Implementation details For all experiments, we set the batch size B = 128 , the noise level σ = 1 .and
Revised: Implementation details For all experiments, we set the batch size B = 128 , the noise level σ = 1 .and

Original: the confidence level δ = 10 − 5 .
Revised: the confidence level δ = 10 − 5 .

Original: For MNIST, we try learning rate in { 2 × 10 − 3 , 5 × 10 − 3 , 10 − 2 } for each experiment and report the best result.
Revised: For MNIST, we try learning rate in { 2 × 10 − 3 , 5 × 10 − 3 , 10 − 2 } for each experiment and report the best result.

Original: For CIFAR10 we fix the learning rate to be 0 .
Revised: For CIFAR10 we fix the learning rate to be 0 .

Original: All experiments are conducted on a server with 4 CPUs and one NVIDIA Tesla P100 GPU.
Revised: All experiments are conducted on a server with 4 CPUs and one NVIDIA Tesla P100 GPU.

Original: denote the per epoch.

Original: The left two figures present the results on the MNIST dataset, the algorithm is about (1 . 0379 , 10 − 5 )

Original: -DP.

Original: The right two figures show the results on the CIFAR10 dataset (feature extracted by a pretrained VGG16 network), the algorithm is about (0 . 9580 , 10 − 5 ) -DP

Original: 6. T HE EVOLUTION OF CLIPPING FREQUENCY DURING TRAINING
Revised: 6. T HE EVOLUTION OF CLIPPING FREQUENCY DURING TRAINING

Original: We run DP-SGD-GC with different clipping thresholds.
Revised: We run DP-SGD-GC with different clipping thresholds.

Original: In particular, we try C ∈ { 1 , 5 , 20 , 40 } and
Revised: In particular, we try C ∈ { 1 , 5 , 20 , 40 } and

Original: C ∈ { 0 . 1 , 0 . 2 , 0 . 4 , 1 . 0 } for MNIST and CIFAR10 respectively.
Revised: C ∈ { 0 . 1 , 0 . 2 , 0 . 4 , 1 . 0 } for MNIST and CIFAR10 respectively.

Original: The evolutions of training accuracy and clipping frequency per epoch with different clipping thresholds are shown in Figure 1.
Revised: The evolutions of training accuracy and clipping frequency per epoch with different clipping thresholds are shown in Figure 1.

Original: We can observe that, in most cases, the clipping frequency decreases as the training accuracy goes up.
Revised: We can observe that, in most cases, the clipping frequency decreases as the training accuracy goes up.

Original: This observation aligns with the growth condition as lower training loss implies a smaller average gradient norm and further results in lower clipping frequency.
Revised: This observation aligns with the WGC as lower training loss implies a smaller average gradient norm and further results in lower clipping frequency.

Original: We also see that, in most cases, the clipping frequency can become close to 0 when the clipping threshold is chosen appropriately; this observation is consistent with our theoretical analysis.
Revised: We also see that, in most cases, the clipping frequency can become close to 0 when the clipping threshold is chosen appropriately; this observation is consistent with our theoretical analysis.

Original: Another interesting observation is the result of training the two-layer neural network with MNIST and C ∈ { 20 , 40 } .
Revised: Another interesting observation is the result of training the two-layer neural network with MNIST and C ∈ { 20 , 40 } .

Original: The clipping frequency is small initially and becomes stable at 15% instead of decreasing to zero.
Revised: The clipping frequency is small initially and becomes stable at 15% instead of decreasing to zero.

Original: We conjecture that this phenomenon is because the growth condition parameters of the neural network grow as the training goes and thus prevent some gradient norms from being smaller than the clipping threshold.
Revised: We conjecture that this phenomenon is because the WGC parameters of the neural network grow as the training goes and thus prevent some gradient norms from being smaller than the clipping threshold.

Original: 6. T HE EVALUATION OF DP-SGD-VC
Revised: 6. T HE EVALUATION OF DP-SGD-VC

Original: We set f ∗ lb = 0 for all experiments with DP-SGD-VC.
Revised: We set f ∗ lb = 0 for all experiments with DP-SGD-VC.

Original: The calculation of the GC parameters for a feed-forward neural network with cross-entropy loss is given in Section E.{+E, where β 2 = 
+}
Revised: The calculation of the WGC parameters for a feed-forward neural network with cross-entropy loss is given in Section {+E, where β 2 = 
+}

Revised: and β 1 depends on the spectral norm of each layer of neural networks.

Original: Compared with vanilla 
Revised: Compared with vanilla SGD,

Original: SGD, DP-SGD-VC has an additional cost to calculate the spectral norm of each layer in each iteration.
Revised: DP-SGD-VC has an additional cost to calculate the spectral norm of each layer in each iteration.

Original: As shown in the following content, the overhead of calculating the spectral norm is not significant.[Table]
Revised: As shown in the following content, the overhead of calculating the spectral norm is not significant.

Original: Training and testing accuracy
Revised: Training and testing accuracy

Original: The training and testing accuracy of DP-SGD-GC and DP-SGDVC with different clipping thresholds are shown in Figure 2 and Figure 3.
Revised: The training and testing accuracy of DP-SGD-GC and DP-SGDVC with different clipping thresholds are shown in Figure 2 and Figure 3.

Original: We can observe that 
Revised: We can observe that accuracy.

Original:  DP-SGD-VC converges slightly slower than DP-SGD-GC in terms of the epoch.
Revised: [Table] DP-SGD-VC converges slightly slower than DP-SGD-GC in terms of the epoch.

Original: This observation should not be surprising as DP-SGD-VC uses an upper bound of the gradient norm for clipping and will result in a smaller effective learning rate than DP-SGD-GC.
Revised: This observation should not be surprising as DP-SGD-VC uses an upper bound of the gradient norm for clipping and will result in a smaller effective learning rate than DP-SGD-GC.

Original: For CIFAR10, the training and testing are easy as the model is trained on pre-trained features; both DP-SGD-VC and DP-SGD-GC can achieve similar accuracy at the end of training.
Revised: For CIFAR10, the training and testing are easy as the model is trained on pre-trained features; both DP-SGD-VC and DP-SGD-GC can achieve similar accuracy at the end of training.

Original: For MNIST, there is an unfortunate loss of training and testing accuracy.
Revised: For MNIST, there is an unfortunate loss of training and testing accuracy.

Original: For MNIST with the linear model, there is a ∼ 2% loss in training and testing accuracy.
Revised: For MNIST with the linear model, there is a ∼ 2% loss in training and testing accuracy.

Original: For MNIST with two-layer NN, there is a 2 ∼ 3% loss in training and testing  for C ∈ { 20 , 40 } and a ∼ 4% loss for C = 10 .
Revised: For MNIST with two-layer NN, there is a 2 ∼ 3% loss in training and testing accuracy for C ∈ { 20 , 40 } and a ∼ 4% loss for C = 10 .

Original: The gap between DP-SGD-GC and GP-SGD-VC is more obvious when the clipping threshold is small.
Revised: The gap between DP-SGD-GC and GP-SGD-VC is more obvious when the clipping threshold is small.

Original: We conjecture that the loss of training and testing accuracy is due to our estimation of f ∗ i .
Revised: We conjecture that the loss of training and testing accuracy is due to our estimation of f ∗ i .

Original: The estimation f ∗ lb = 0 is accurate for CIFAR as the model can almost perfectly fit all pre-trained data.
Revised: The estimation f ∗ lb = 0 is accurate for CIFAR as the model can almost perfectly fit all pre-trained data.

Original: However, the estimation is inaccurate for
Revised: However, the estimation is inaccurate for

Original: MNIST and thus results in a loose upper bound of the gradients’ norm.
Revised: MNIST and thus results in a loose upper bound of the gradients’ norm.

Original: Comparing the computational time per epoch We report the per epoch runtime of different algorithms in Table 2.
Revised: Comparing the computational time per epoch We report the per epoch runtime of different algorithms in Table 2.

Original: All experiments are conducted on a server with one NVIDIA Tesla P100 GPU.
Revised: All experiments are conducted on a server with one NVIDIA Tesla P100 GPU.

Original: The vanilla SGD without privacy consideration is the baseline method and is the fastest among all algorithms.
Revised: The vanilla SGD without privacy consideration is the baseline method and is the fastest among all algorithms.

Original: Micro-batching is the naive implementation of DP-SGD-GC and is significantly slower than SGD.
Revised: Micro-batching is the naive implementation of DP-SGD-GC and is significantly slower than SGD.

Original: GC-Opacus is the implementation of DP-SGD-GC from the (highly optimized) Opacus package (Yousefpour et al., 2021); we can see that there is still a gap between the performance of
Revised: GC-Opacus is the implementation of DP-SGD-GC from the (highly optimized) Opacus package (Yousefpour et al., 2021); we can see that there is still a gap between the performance of

Original: GC-Opacus and the standard non-private SGD algorithm.
Revised: GC-Opacus and the standard non-private SGD algorithm.

Original: DP-SGD-VC is our implementation of
Revised: DP-SGD-VC is our implementation of

Original: DP-SGD with the proposed value clipping technique.
Revised: DP-SGD with the proposed value clipping technique.

Original: We can observe that DP-SGD-VC is slightly slower than the standard SGD algorithm and faster than other private training methods.
Revised: We can observe that DP-SGD-VC is slightly slower than the standard SGD algorithm and faster than other private training methods.

Original: Limitation Despite the efficiency of DP-SGD-VC, it also has certain limitations: (i) as shown in the experiments, DP-SGD-VC may cause some loss in training/testing accuracy if our estimation for the GC parameters is loose; (ii) DP-SGD-VC requires calculating the GC parameters, which we show is available for feed-forward neural works with cross-entropy loss.
Revised: Limitations Despite the efficiency of DP-SGD-VC, it also has certain limitations: (i) as shown in the experiments, DP-SGD-VC may cause some loss in training/testing accuracy if our estimation for the WGC parameters is loose; (ii) DP-SGD-VC requires calculating the WGC parameters, which we show is available for feed-forward neural works with cross-entropy loss.

Original: However, VC technique can not be applied to more complicated neural network architectures with arbitrary loss in a black-box manner.
Revised: However, VC technique can not be applied to more complicated neural network architectures with arbitrary loss in a black-box manner, e.g., transformers with ranking loss.

Original: Overall, DP-SGD-VC is more efficient than DP-SGD-GC and can perform similarly to DP-SGD-GC in certain scenarios but also possibly worse in other situations.
Revised: Overall, DP-SGD-VC is more efficient than DP-SGDGC and can perform similarly to DP-SGD-GC in certain scenarios but also possibly worse in other situations.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper studied the convergence behavior of a widely used privacy-preserving learning algorithm called DP-SGD-GC.
Revised: This paper studied the convergence behavior of a widely used privacy-preserving learning algorithm called DP-SGD-GC.

Original: Our analysis extended the convergence of DP-SGD-GC to smooth and unconstrained problems without assuming the objective to be globally Lipschitz continuous.
Revised: Our analysis extended the convergence of DP-SGD-GC to smooth and unconstrained problems without assuming the objective to be globally Lipschitz continuous.

Original: We believe that our theoretical results improved the current understanding of DP-SGD-GC and provided new insights for practitioners and researchers to use DP-SGD-GC and design new algorithms.
Revised: We believe that our theoretical results improved the current understanding of DP-SGD-GC and provided new insights for practitioners and researchers to use DP-SGD-GC and design new algorithms.

Original: Our analysis can potentially be used for other privacy-preserving learning algorithms such as adaptive DP-SGD (Asi et al., 2021) and DP-SGD with subspace identification (Zhou et al., 2021; Song et al., 2021).
Revised: Our analysis can potentially be used for other privacy-preserving learning algorithms such as adaptive DP-SGD (Asi et al., 2021) and DP-SGD with subspace identification (Zhou et al., 2021; Song et al., 2021).

Original: Our work also suggests some future directions.
Revised: Our work also suggests some future directions.

Original: The light-tail noise condition may not hold in some machine learning applications (Gürbüzbalaban et al., 2021).
Revised: The light-tail noise condition may not hold in some machine learning applications (Gürbüzbalaban et al., 2021).

Original: In these scenarios, our utility bound of DP-SGD-GC (Theorem 4.5) is √ n worse than the best-known bound of DP-SGD-GC applied to Lipschitz functions.
Revised: In these scenarios, our utility bound of DP-SGD-GC (Theorem 4.5) is √ n worse than the best-known bound of DP-SGD-GC applied to Lipschitz functions.

Original: Whether it is possible to further improve the utility bound is an interesting question from the perspective of theory.
Revised: Whether it is possible to further improve the utility bound is an interesting question from the perspective of theory.

Original: Another direction is to explore the lower bound of DP-SGD-GC with a carefully tuned clipping threshold for smooth and unconstrained problems.
Revised: Another direction is to explore the lower bound of DP-SGD-GC with a carefully tuned clipping threshold for smooth and unconstrained problems.

Original: E THIC STATEMENT
Revised: E THIC STATEMENT

Original: This paper studied a class of private-preserving machine learning algorithms called differential private
Revised: This paper studied a class of private-preserving machine learning algorithms called differential private

Original: SGD.
Revised: SGD.

Original: We provide improved convergence analysis and some new theoretical insights into DP-SGD.
Revised: We provide improved convergence analysis and some new theoretical insights into DP-SGD.

Original: We are not aware of any potential negative social impact of this work.
Revised: We are not aware of any potential negative social impact of this work.

Original: The datasets used for the experiments do not contain personally identifiable information or offensive content.
Revised: The datasets used for the experiments do not contain personally identifiable information or offensive content.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: The code to reproduce our experimental results in Section 6 is included as supplementary material in the submission; we will make our code publicly available upon acceptance
Revised: The code to reproduce our experimental results in Section 6 is included as supplementary material in the submission; we will make our code publicly available upon acceptance

Original:   A. S TANDARD FACTS
Revised:  A PPENDIX A PPENDIX A L EMMAS A. S TANDARD FACTS

Original: Lemma A.1.
Revised: Lemma A.1.

Original: For any a, b ∈ R , it is true that ( a + b ) 2 ≤ 

Revised: For any a, b ∈ R , it is true that ( a + b ) 2 ≤

Original: a 

Revised: a

Original: + 2 b 2 .
Revised: + 2 b 2 .

Original: Lemma A.2.
Revised: Lemma A.2.

Original: It holds that (cid:80) ∞ i =1 ( i/ 2 i )
Revised: It holds that (cid:80) ∞ i =1 ( i/ 2 i )

Original: = 2 .
Revised: = 2 .

Revised: Lemma A.3.

Revised: Given n random events A 1 , A 2 , . . .

Revised: , A n .

Revised: It holds that

Revised: [Equation]

Original: A. S UB -G AUSSIAN AND SUB - EXPONENTIAL PROPERTIES
Revised: A. S UB -G AUSSIAN AND SUB - EXPONENTIAL PROPERTIES

Original: Lemma A.3 (Sub-Gaussian properties, Vershynin, 2018, Proposition 2.5.2) .
Revised: Lemma A.4 (Sub-Gaussian properties, Vershynin, 2018, Proposition 2.5.2) .

Original: Let X be a random variable.
Revised: Let X be a random variable.

Original: Then the following properties are equivalent; the parameters K i , i = 1 , . .
Revised: Then the following properties are equivalent; the parameters K i , i = 1 , . .

Original: , 4 differ from each other by at most an absolute constant factor.
Revised: , 4 differ from each other by at most an absolute constant factor.

Original: • The tails of X satisfy
Revised: • The tails of X satisfy

Original: [Equation]
Revised: [Equation]

Original: • The moment generating function (MGF) of X 2 satisfies
Revised: • The moment generating function (MGF) of X 2 satisfies

Original: [Equation]
Revised: [Equation]

Original: • The MGF of X 2 is bounded at some point, e.g.,
Revised: • The MGF of X 2 is bounded at some point, e.g.,

Original: [Equation]
Revised: [Equation]

Original: • E [ X ] = 0 and the MGF of X satisfies
Revised: • E [ X ] = 0 and the MGF of X satisfies

Original: [Equation]
Revised: [Equation]

Original: Lemma A.4 (Harvey et al., 2019, Lemma A.4) .
Revised: Lemma A.5 (Harvey et al., 2019, Lemma A.4) .

Original: Let X 1 , X 2 , . . .
Revised: Let X 1 , X 2 , . . .

Original: , X n be random variables.
Revised: , X n be random variables.

Original: Assume that there exist K 1 , K 2 , . .
Revised: Assume that there exist K 1 , K 2 , . .

Original: . , K n > 0 such that E
Revised: . , K n > 0 such that E

Original: [exp( λX i )] ≤ exp( λK i ) for all 0 ≤ λ ≤ 1 /K i .
Revised: [exp( λX i )] ≤ exp( λK i ) for all 0 ≤ λ ≤ 1 /K i .

Original: Then E [exp( λ (cid:80) ni
Revised: Then E [exp( λ (cid:80) ni

Original: =1 X i )] ≤ exp( λ (cid:80) ni
Revised: =1 X i )] ≤ exp( λ (cid:80) ni

Original: =1 K i ) for all 0 ≤ λ ≤ 1 / (cid:80) ni
Revised: =1 K i ) for all 0 ≤ λ ≤ 1 / (cid:80) ni

Original: =1 K i .
Revised: =1 K i .

Original: Lemma A.5 (Harvey et al., 2019, Claim A.7) .
Revised: Lemma A.6 (Harvey et al., 2019, Claim A.7) .

Original: Suppose X is a random variable such that there exists constants c and K such that E
Revised: Suppose X is a random variable such that there exists constants c and K such that E

Original: [exp( λX )] ≤ c exp( λK ) for all 0 ≤ λ ≤ 1 /K .
Revised: [exp( λX )] ≤ c exp( λK ) for all 0 ≤ λ ≤ 1 /K .

Original: Then for any δ ∈ (0 , 1) , Pr[ X ≥ K log(1 /δ )] ≤ ceδ .
Revised: Then for any δ ∈ (0 , 1) , Pr[ X ≥ K log(1 /δ )] ≤ ceδ .

Original: Note that the original lemmas from Harvey et al.
Revised: Note that the original lemmas from Harvey et al.

Original: (Harvey et al., 2019) assume E [exp( λX )] ≤ exp( λK ) for all λ ≤ 1 /K .
Revised: (Harvey et al., 2019) assume E [exp( λX )] ≤ exp( λK ) for all λ ≤ 1 /K .

Original: It is easy to check that their results also hold for X such that E [exp( λX )] ≤ exp( λK ) for all
Revised: It is easy to check that their results also hold for X such that E [exp( λX )] ≤ exp( λK ) for all

Original: 0 ≤ λ ≤ 1 /K .
Revised: 0 ≤ λ ≤ 1 /K .

Original: Lemma A.6.
Revised: Lemma A.7.

Original: Let X 1 , X 2 , . . .
Revised: Let X 1 , X 2 , . . .

Original: , X T be random variables such that E
Revised: , X T be random variables such that E

Original: [exp( λX i )] ≤ exp( λK ) ∀ λ ∈ [0 , K ] for all i ∈ { 1 , 2 , . . .
Revised: [exp( λX i )] ≤ exp( λK ) ∀ λ ∈ [0 , K ] for all i ∈ { 1 , 2 , . . .

Original: , T } .
Revised: , T } .

Original: Then there exists absolute positive constant c such that
Revised: Then there exists absolute positive constant c such that

Original: Pr (cid:104)(cid:80) Ti =

Original: [Equation]

Original: i ≥ TK log(

Original: [Equation]
Revised: [Equation]

Original: (cid:105) ≤ ceδ for any δ ∈ (0 , 1) .
Revised: ≤ ceδ for any δ ∈ (0 , 1) .

Original: Proof.
Revised: Proof.

Original: By Lemma A.4,
Revised: By Lemma A.5,

Original: [Equation]
Revised: [Equation]

Original: Then Lemma A.5 gives
Revised: Then Lemma A.6 gives

Original: [Equation]
Revised: [Equation]

Original: The above finishes the proof.
Revised: The above finishes the proof.

Original: Lemma A.7 (Tail bound for the maximum of sub-Gaussian variables) .
Revised: Lemma A.8 (Tail bound for the maximum of sub-Gaussian variables) .

Original: Let X 1 , X 2 , . . .
Revised: Let X 1 , X 2 , . . .

Original: , X n be random variables.
Revised: , X n be random variables.

Original: Assume that there exist K > 0 such that E [exp( λX i )] ≤ exp( λ 2 K 2 ) for all λ ∈ R and i ∈ [ n ] .
Revised: Assume that there exist K > 0 such that E [exp( λX i )] ≤ exp( λ 2 K 2 ) for all λ ∈ R and i ∈ [ n ] .

Original: Then there exist some absolute positive constant c such that
Revised: Then there exist some absolute positive constant c such that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By Lemma A.3, there exist some absolute constant c > 0 such that
Revised: By Lemma A. [Equation] there exist some absolute constant c > 0 such that

Original: [Equation]
Revised: [Equation]

Original: Then we simply apply union bound to obtain the conclusion.
Revised: Then we simply apply union bound to obtain the conclusion.

Original: For any t ≥ 0 ,
Revised: For any t ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: which yields the desired result
Revised: which yields the desired result

Original: [Equation]
Revised: [Equation]

Original: Lemma A.8.
Revised: Lemma A.9.

Original: Let X be a random variable and E be a random event such that E [ | X | | E ] ≤ α and
Revised: Let X be a random variable and E be a random event such that E [ | X | | E ] ≤ α and

Original: Pr[ E ] ≥ 1 − δ for some α > 0 , δ ∈ (0 , 1) .
Revised: Pr[ E ] ≥ 1 − δ for some α > 0 , δ ∈ (0 , 1) .

Original: If X satisfies Pr[ | X | − β ≥ t ] ≤ 2 exp( − t/K ) ∀ t
Revised: If X satisfies Pr[ | X | − β ≥ t ] ≤ 2 exp( − t/K ) ∀ t

Original: ≥for some β, K > 0 .
Revised: ≥for some β, K > 0 .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let Q : [0 , 1] → R ∪ {∞} be the quantile function for the random variable | X | , e.g.,
Revised: Let Q : [0 , 1] → R ∪ {∞} be the quantile function for the random variable | X | , e.g.,

Original: [Equation]
Revised: [Equation]

Original: By the assumption that | X | − β is sub-exponential, we have that
Revised: By the assumption that | X | − β is sub-exponential, we have that

Original: [Equation]
Revised: [Equation]

Original: Now we are ready to prove the conclusion.
Revised: Now we are ready to prove the conclusion.

Original: First notice that
Revised: First notice that

Original: [Equation]
Revised: [Equation]

Original: ) where 1 E c is the indicator function with the event E c .
Revised: where 1 E c is the indicator function with the event E c .

Original: The remaining is to bound E [ | X | · 1 E c ] .
Revised: The remaining is to bound E [ | X | · 1 E c ] .

Original: Define a new set of events A i :=
Revised: Define a new set of events A i :=

Original: (cid:8) | X | ∈ (cid:2) Q [-(cid:0) 1 − δ 
-]
Revised: (cid:8) | X | ∈ [Equation] Q [-(cid:0) 1 − δ 
-][Equation]

Original: i − 1 (cid:1) , Q (cid:0) 1 − δ 2 i (cid:1)(cid:3)(cid:9) , i = 1 , 2 , . . .
Revised: 1 − δ [Equation] [Equation] , Q

Revised: [Equation]

Original: {+1 − δ 
+}
Revised: {+1 − δ 
+}

Revised: i (cid:1)(cid:3)(cid:9) , i = 1 , 2 , . . .

Original: , and denote the probability measure space as (Ω , F , µ ) .
Revised: , and denote the probability measure space as (Ω , F , µ ) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where (i) is by eq.
Revised: where (i) is by eq.

Original: (4) and (ii) is by the standard scalar inequality (Lemma A.2).
Revised: (4) and (ii) is by the standard scalar inequality (Lemma A.2).

Original: The above together with eq.
Revised: The above together with eq.

Original: (5) yields the desired result.
Revised: (5) yields the desired result.

Original: A. T HE GENERALIZED F REEDMAN INEQUALITY
Revised: A. T HE GENERALIZED F REEDMAN INEQUALITY

Original: We restate the generalized Freedman inequality and its corollary from Harvey et al.(Harvey et al., 2019).
Revised: We restate the generalized Freedman inequality and its corollary from Harvey et al.(Harvey et al., 2019).

Original: Lemma A.9 (Generalized Freedman Inequality, Harvey et al., 2019, Theorem 3.2) .
Revised: Lemma A.10 (Generalized Freedman Inequality, Harvey et al., 2019, Theorem 3.2) .

Original: Let { d i , F i }
Revised: Let { d i , F i }

Original: Ti
Revised: Ti

Original: = be a martingale difference sequence.
Revised: = be a martingale difference sequence.

Original: Suppose v i − 1 ≥ 0 , ∀ i ∈
Revised: Suppose v i − 1 ≥ 0 , ∀ i ∈

Original: [ T ] are F i − 1 -measurable random variables such that E
Revised: [ T ] are F i − 1 -measurable random variables such that E

Original: [exp( λd i ) | F i − 1 ] ≤ exp( λ 2 2 v
Revised: [exp( λd i ) | F i − 1 ] ≤ exp( λ 2 2 v

Original: i − 1 ) for all i ∈
Revised: i − 1 ) for all i ∈

Original: [ T ] , λ > 0 .
Revised: [ T ] , λ > 0 .

Original: Let S t = (cid:80) ti =1 d i and V t = (cid:80) ti =
Revised: Let S t = (cid:80) t i = d i and V t = (cid:80) ti =

Original: 1 v
Revised: 1 v

Original: i − 1 .
Revised: i − 1 .

Original: Let α i ≥ 0 and α = max i ∈
Revised: Let α i ≥ 0 and α = max i ∈

Original: [ T ] α i .
Revised: [ T ] α i .

Original: Then
Revised: Then

Original: Pr

Original: [Equation]
Revised: [Equation]

Original: The following lemma is an immediate consequence from the generalized Freedman inequality.
Revised: The following lemma is an immediate consequence from the generalized Freedman inequality.

Original: Lemma A.10.
Revised: Lemma A.11.

Original: Let { d i , F i } Ti =1 be a martingale difference sequence.
Revised: Let { d i , F i } Ti =1 be a martingale difference sequence.

Original: Suppose v i − 1 ≥ 0 , ∀ i ∈
Revised: Suppose v i − 1 ≥ 0 , ∀ i ∈

Original: [ T ] are F i − 1 -measurable random variables such that E
Revised: [ T ] are F i − 1 -measurable random variables such that E

Original: [exp( λd i ) | F i − 1 ] ≤ exp( λ 2 2 v
Revised: [exp( λd i ) | F i − 1 ] ≤ exp( λ 2 2 v

Original: i − 1 ) for all i ∈
Revised: i − 1 ) for all i ∈

Original: [ T ] , λ > 0 .
Revised: [ T ] , λ > 0 .

Original: Let S t = (cid:80) t i = d i and V t = (cid:80) t i =1 v i − 1 .
Revised: Let S t = (cid:80) ti =1 d i and V t = (cid:80) ti = 1 v

Original:  Let δ ∈ (0 , 1) and suppose there are positive values R ( δ ) > 0 and non-negative values { α ( t ) i , i = 1 , 2 , . .
Revised: i − 1 . Let δ ∈ (0 , 1) and suppose there are positive values R ( δ ) > 0 and non-negative values { α ( t ) i , i = 1 , 2 , . .

Original: , T, t = 1 , 2 , . . .
Revised: , T, t = 1 , 2 , . . .

Original: , T } such that
Revised: , T } such that

Original: Pr

Original: [Equation]
Revised: [Equation]

Original: (cid:41)(cid:35)

Original: [Equation]

Original: 1 − δ.

Original: Let α = max i ∈
Revised: Let α = max i ∈

Original: [ T ] ,t ∈
Revised: [ T ] ,t ∈

Original: [ T ] α ( t ) i .
Revised: [ T ] α ( t ) i .

Original: Then
Revised: Then

Original: Pr (cid:34) T (cid:91) t = { S t ≥ x } (cid:35) ≤ δ + T exp (cid:18) − x 4 α + 8 R ( δ ) /x (cid:19) ∀ x, β > 0 .
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Given δ ∈ (0 , 1) , x ∈ R .
Revised: Given δ ∈ (0 , 1) , x ∈ R .

Original: Define the events A t := { S t ≥ x } and B t :=
Revised: Define the events A t := { S t ≥ x } and B t :=

Original: (cid:110)
Revised: (cid:110)

Original: V t ≤ (cid:80) ti =
Revised: V t ≤ (cid:80) ti =

Original: 1 α ( t ) i
Revised: 1 α ( t ) i

Original: d i + R ( δ ) (cid:111) .
Revised: d i + R ( δ ) (cid:111) .

Original: Then
Revised: Then

Original: Pr (cid:34) T (cid:91) t
Revised: Pr (cid:34) T (cid:91) t

Original: = { S t ≥ x } (cid:35) =
Revised: = { S t ≥ x } (cid:35) =

Original: Pr (cid:34) T (cid:91) t =
Revised: Pr (cid:34) T (cid:91) t =

Original: A t (cid:35) =
Revised: A t (cid:35) =

Original: Pr (cid:34)(cid:32) T (cid:91) t =
Revised: Pr (cid:34)(cid:32) T (cid:91) t =

Original: A t (cid:33) (cid:92) (cid:32) T (cid:92) t =
Revised: A t (cid:33) (cid:92) (cid:32) T (cid:92) t =

Original: B t (cid:33)(cid:35) + Pr (cid:34)(cid:32) T (cid:91) t =
Revised: B t (cid:33)(cid:35) + Pr (cid:34)(cid:32) T (cid:91) t =

Original: A t (cid:33) (cid:92) (cid:32) T (cid:92) t =
Revised: A t (cid:33) (cid:92) (cid:32) T (cid:92) t =

Original: B t (cid:33) c (cid:35) (i) ≤
Revised: B t (cid:33) c (cid:35) (i) ≤

Original: Pr (cid:34)(cid:32) T (cid:91) t =
Revised: Pr (cid:34)(cid:32) T (cid:91) t =

Original: A t (cid:33) (cid:92) (cid:32) T (cid:92) t = 
Revised: A t (cid:33) (cid:92) (cid:32) T (cid:92) t = B t (cid:33)(cid:35) + δ ≤ Pr (cid:34) T (cid:91) t = (cid:32) A t (cid:92) (cid:32) T (cid:92) i = B i (cid:33)(cid:33)(cid:35) + δ ≤ Pr (cid:34) T (cid:91) t = (cid:16) A t (cid:92) B t (cid:17)(cid:35) + δ

Original:  B t (cid:33)(cid:35) + δ ≤ Pr (cid:34)
Revised: Pr (cid:104) A t (cid:92) B t (cid:105) + δ (iii)

Original: [Equation]
Revised: [Equation]

Original: where (i) is by the assumption Pr (cid:2)
Revised: + T exp (cid:18) [Equation]

Original:  ∩ Tt =1 B t (cid:3) ≥ 1 − δ , (ii) is by union bound (note that we can not directly use Lemma A.9 since V t relies on α ( t ) 1 , . .
Revised: x 4 α + 8 R ( δ ) /x ∀ x, β > 0 , where (i) is by the assumption Pr (cid:2) ∩ Tt =1 B t (cid:3) ≥ 1 − δ , (ii) is by union bound (note that we can not directly use Lemma A.10 since V t relies on α ( t ) 1 , . .

Original: α ( t ) t instead of α 1 , . .
Revised: α ( t ) t instead of α 1 , . .

Original: . , α t ), (iii) is by applying
Revised: . , α t ), (iii) is by applying

Original: Lemma A.9 for T times.
Revised: Lemma A.10 for T times.

Original: A
Revised: A. O THER LEMMAS

Original: O THER LEMMAS

Original: Recall Lemma 3.2.
Revised: Recall Lemma 3.2.

Original: Proof of Lemma 3.2.
Revised: Proof of Lemma 3.2.

Original: By the smoothness of h , we have
Revised: By the smoothness of h , we have

Original: [Equation]
Revised: [Equation]

Original: Making the identification that v = u − ∇ h ( u ) /L .
Revised: Making the identification that v = u − ∇ h ( u ) /L .

Original: We obtain
Revised: We obtain

Original: [Equation]
Revised: [Equation]

Original: Next, we review the standard convergence rate of SGD algorithm in the following lemma.
Revised: Next, we review the standard convergence rate of SGD algorithm in the following lemma.

Original: Lemma A.11 (Ghadimi & Lan, 2013, Theorem 2.1) .
Revised: Lemma A.12 (Ghadimi & Lan, 2013, Theorem 2.1) .

Original: Denote ˆ w as the output of Algorithm 4.
Revised: Denote ˆ w as the output of Algorithm 4.

Original: Assume that f i ’s are L -smooth for some L > 0 , η ≤ 1 /L and there exist σ ≥ 0 such that E [ ∥ ξ ( t ) ∥ 2 ] ≤ σfor all t ∈ N .
Revised: Assume that f i ’s are L -smooth for some L > 0 , η ≤ 1 /L and there exist σ ≥ 0 such that E [ ∥ ξ ( t ) ∥ 2 ] ≤ σfor all t ∈ N .

Original: • If f i ’s are further convex, then
Revised: • If f i ’s are further convex, then

Original: [Equation]
Revised: [Equation]

Original: where W ∗ is the set of solutions.
Revised: where W ∗ is the set of solutions.

Original: • If f i ’s are not necessarily convex, then
Revised: • If f i ’s are not necessarily convex, then

Original: [Equation]
Revised: [Equation]

Original: The following lemma connects the GC parameters of f i ’s to the GC parameters of f .
Revised: The following lemma connects the WGC parameters of f i ’s to the WGC parameters of f .

Original: Lemma A.12.
Revised: Lemma A.13.

Original: Assume that f i ’s are ( β 1 , β 2 ) -GC for all i ∈ [ n ] .
Revised: Assume that f i ’s are ( β 1 , β 2 ) -WGC for all i ∈ [ n ] .

Original: Then f := 1 n (cid:80) ni
Revised: Then f := 1 n (cid:80) ni

Original: =

Revised: =

Original: f i is ( β 1 , β 2 + Γ β 1 ) -GC, where Γ = 1 n (cid:80) ni
Revised: f i is ( β 1 , β 2 + Γ β 1 ) -WGC, where Γ = 1 n (cid:80) ni

Original: =

Revised: =

Original: ( f ∗ − f ∗ i ) .
Revised: ( f ∗ − f ∗ i ) .

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: where (i) is by the convexity of ∥ · ∥ 2 and (ii) is by the assumption that f i ’s are ( β 1 , β 2 )
Revised: where (i) is by the convexity of ∥ · ∥ 2 and (ii) is by the assumption that f i ’s are ( β 1 , β 2 )

Original: -GC.
Revised: -WGC.

Original: A PPENDIX B M ISSING PROPOSITIONS AND THEOREMS
Revised: A PPENDIX B M ISSING PROPOSITIONS AND THEOREMS

Original: The convergence of DP-SGD-GC with Assumption 4.1 is given in Section 4.
Revised: The convergence of DP-SGD-GC with Assumption 4.1 is given in Section 4.

Original: Here we present the propositions and theorems without Assumption 4.1.
Revised: Here we present the propositions and theorems without Assumption 4.1.

Original: Proposition B.1 (Convergence on conditional expectation) .
Revised: Proposition B.1 (Convergence on conditional expectation) .

Original: Denote w priv as the output of Algorithmand define E := {∥ f i ( w ( t ) )
Revised: Denote w priv as the output of Algorithmand define E := {∥ f i ( w ( t ) )

Original: ∥ ≤ C ∀ i ∈ [ n ] , t ∈ { 0 , 1 . . .
Revised: ∥ ≤ C ∀ i ∈ [ n ] , t ∈ { 0 , 1 . . .

Original: , T }} as the event of no clipping happens during the training of Algorithm 1.
Revised: , T }} as the event of no clipping happens during the training of Algorithm 1.

Original: Let D f := f ( w (0) )
Revised: Let D f := f ( w (0) )

Original: − f ∗ .
Revised: − f ∗ .

Original: Given any ϵ > 0 and δ, δ ′ ∈ (0 , 1) .
Revised: Given any ϵ > 0 and δ, δ ′ ∈ (0 , 1) .

Original:  Assume 
Revised: • Assume f

Original: f i ’s are convex and L -smooth for some L > 0 .
Revised: i ’s are convex and L -smooth for some L > 0 .

Original: set T > ϵ/c 1 ,
Revised: set T > ϵ/c 1 ,

Original: [Equation]
Revised: [Equation]

Original: where c 1 , c 2 , c 5 are some absolute constants and c 3 , c 4 are constants that depend on L and D f .
Revised: where c 1 , c 2 , c 5 are some absolute constants and c 3 , c 4 are constants that depend on L and D f .

Original: We have that Algorithm 1 is ( ϵ, δ ) -DP, Pr[ E ] ≥ 1 − δ ′ , and
Revised: We have that Algorithm 1 is ( ϵ, δ ) -DP, Pr[ E ] ≥ 1 − δ ′ , and

Original: [Equation]
Revised: [Equation]

Original: • Assume f
Revised: • Assume f

Original: i ’s are L -smooth for some L > 
Revised: i ’s are L -smooth for some L > 0 .

Original: Setting ϵ, σ, C, η as in eq.
Revised: Setting ϵ, σ, C, η as in eq.

Original: It holds that Algorithmis ( ϵ, δ ) -DP, Pr[ E ] ≥ 1 − δ ′ , and
Revised: It holds that Algorithmis ( ϵ, δ ) -DP, Pr[ E ] ≥ 1 − δ ′ , and

Original: [Equation]
Revised: [Equation]

Original: Theorem B.2 (Convergence on expectation) .
Revised: Theorem B.2 (Convergence on expectation) .

Original: Denote w priv as the output of Algorithm 1.
Revised: Denote w priv as the output of Algorithm 1.

Original: Let
Revised: Let

Original: D f := f ( w (0) )
Revised: D f := f ( w (0) )

Original: − f ∗ .
Revised: − f ∗ .

Original: Given any ϵ > 0 and δ ∈ (0 , 1) .
Revised: Given any ϵ > 0 and δ ∈ (0 , 1) .

Original: Setting T, σ, η in the same way as eq.
Revised: Setting T, σ, η in the same way as eq.

Original: (6) and let C = (cid:112) c
Revised: (6) and let C = (cid:112) c

Original: 3 n + c 4 n log( T ) , where c 1 , c 2 , c 5 are some absolute constants and c 3 , c 4 are constants that depend on L, D f .
Revised: 3 n + c 4 n log( T ) , where c 1 , c 2 , c 5 are some absolute constants and c 3 , c 4 are constants that depend on L, D f .

Original: • Assume f
Revised: • Assume f

Original: i ’s are convex and L -smooth for some L > 0 .
Revised: i ’s are convex and L -smooth for some L > 0 .

Original: Then Algorithm 1 is ( ϵ, δ ) -DP
Revised: Then Algorithm 1 is ( ϵ, δ ) -DP

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Consequently, we have E [ f ( w priv ) − f ∗ ] = (cid:101)O (cid:0) d 1 / 2 ( nϵ ) − 1 (cid:1) by setting T =
Revised: Consequently, we have E [ f ( w priv ) − f ∗ ] = (cid:101)

Revised: O

Revised: (cid:0)

Revised: d

Revised: nϵ

Original:  Θ (cid:0) nϵ2 d − 1 (cid:1) .
Revised: ) − 1 (cid:1) by setting T = Θ (cid:0) nϵ d − 1 (cid:1) .

Original: • Assume f
Revised: • Assume f

Original: i ’s are L -smooth for some L > 0 .
Revised: i ’s are L -smooth for some L > 0 .

Original: Then Algorithm 1 is ( ϵ, δ ) -DP
Revised: Then Algorithm 1 is ( ϵ, δ ) -DP

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Revised: priv

Original:  Consequently, we have E (cid:2) ∥∇ f ( w priv ) ∥ 2 (cid:3) =
Revised: [Equation] O [Equation] Consequently, we have E (cid:2) ∥∇ f ( w priv ) ∥ 2 (cid:3) =

Original: (cid:101) O (cid:0) d 1 / 2 ( nϵ ) − 1 (cid:1) by setting T =
Revised: (cid:101) O (cid:0) d 1 / 2 ( nϵ ) − 1 (cid:1) by setting T =

Original: Θ (cid:0) nϵ 2 d − 1 (cid:1) .
Revised: Θ (cid:0) nϵ 2 d − 1 (cid:1) .

Original: A PPENDIX C P ROOFS FOR S ECTIONC. P ROOF OF P ROPOSITION 4. 
Revised: A PPENDIX C P ROOFS FOR S ECTIONC. P ROOF OF P ROPOSITION 4.

Original: Proof.
Revised: Proof.

Original: We begin with the smoothness of f ,
Revised: We begin with the smoothness of f ,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Let
Revised: Let

Revised: Z t := t (cid:88) i =

Revised: z i ,

Revised: U

Original: [Equation]
Revised: [Equation]

Revised: :=

Revised: [Equation]

Revised: i = u i .

Original: Recursively apply eq.
Revised: Recursively apply eq.

Original: (7) gives
Revised: (7) gives

Original: [Equation]
Revised: [Equation]

Original: All we need is to show that Z T − 1 and U t − 1 are bounded above with high probability.
Revised: All we need is to show that Z T − 1 and U t − 1 are bounded above with high probability.

Original: First, we bound
Revised: First, we bound

Original: Z T − 1 .
Revised: Z T − 1 .

Original: Notice that ζ ( t ) ’s are sub-Gaussian variables, therefore Z T − 1 is sub-exponential.
Revised: Notice that ζ ( t ) ’s are sub-Gaussian variables, therefore Z T − 1 is sub-exponential.

Original: We apply
Revised: We apply

Original: Lemma A.6, there exist some absolute constant c 1 > 0 such that for any δ ′ ∈ (0 , 0 .5)
Revised: Lemma A.7, there exist some absolute constant c 1 > 0 such that for any δ ′ ∈ (0 , 0 .

Original: [Equation]
Revised: [Equation]

Original: definition

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − δ ′ .
Revised: with probability at least 1 − δ ′ .

Original: Next we bound the term U t − 1  by the generalized Freedman inequality (Lemma A.9).
Revised: Next we bound the term U t − 1 . Noticing that { U t } ∞ t =0 is a martingale sequence, we bound it by the generalized Freedman inequality (Lemma A.10).

Original: Denote F t −to be the σ -Algebra generated from { w (1) , . . .
Revised: Denote F t − 1 to be the σ -Algebra generated from { w (1) , . . .

Original: , w ( t ) } .
Revised: , w ( t ) } .

Original: Then by the definition of ζ ( t ) , we have
Revised: Then by the definition of ζ ( t ) , we have

Original: [Equation]
Revised: [Equation]

Revised: where we use the inequality ∥ u t ∥ 2 ≤ ( Lη 2 − η ) 2 ∥∇ f ( w ( t ) )

Revised: ∥ 2 ∥ ζ ( t ) ∥ 2 .

Original: By the properties of sub-Gaussian variable (Lemma A.3), there exist some absolute constant c 2 >such that
Revised: By the properties of subGaussian variable (Lemma A.4), there exist some absolute constant c 2 > 0 such that

Original: [Equation]
Revised: [Equation]

Original: Let v t − 1 :
Revised: Let v t − 1 :

Original: =
Revised: =

Original: c 2 ( Lη 2 − η ) 2 ˜ σ 2 ∥∇ f ( w ( t ) )
Revised: c 2 ( Lη 2 − η ) 2 ˜ σ 2 ∥∇ f ( w ( t ) )

Original: ∥ 2 .
Revised: ∥ 2 .

Original: Then v t

Original: [Equation]

Original: ≤

Original: [Equation]

Original: c

Original: [Equation]

Original: Lη

Original: [Equation]

Original:  ˜ σ 2 ( f ( w ( t ) ) [Equation] f ∗ ) (By the  growth condition)
Revised: Then v t − 1 ≤ 2 c 2 Lη 2 ˜ σ 2 ( f ( w ( t ) ) − f ∗ ) (By the weak growth condition)

Original: [Equation]
Revised: [Equation]

Original: (By η≤
Revised: (By η

Original: [Equation]
Revised: [Equation]

Original: (˜ σ √ T ) ) .
Revised: T ) ) .

Original: Furthermore, for any t = 0 , 1 , . .
Revised: Furthermore, for any t = 0 , 1 , . .

Original: , T − 1 ,
Revised: , T − 1 ,

Original: V t :=

Original: t (cid:88) i = v i −≤ ct (cid:88)

Original: i

Original: [Equation]

Original: [Equation]

Original: ≤

Original: [Equation]
Revised: [Equation]

Original: Combining  with eq.
Revised: Combining the above with eq.

Original: (9), with probability 1 − δ ′
Revised: (9), with probability 1 − δ ′

Original: V t ≤ 2 c 2 L (cid:16) f

Original: ( w (0) )

Original: − f ∗ + c 1 L log(1 /δ ′ ) / 2 (cid:17) + 2 c 2 L t (cid:88) i = t − i T u i ∀ t = 0 , 1 , . . . , T − 1 .
Revised: [Equation]

Original: Now we are ready to apply Lemma A.10.
Revised: Now we are ready to apply Lemma A.11.

Original: Making the identification
Revised: Making the identification

Original: [Equation]
Revised: [Equation]

Original: and apply Lemma A.10 gives 
Revised: and apply Lemma A.11 gives Pr

Original: [Equation]
Revised: [Equation]

Original:  It is easy to verify that with the choice x = max (cid:110) 4 (cid:112) R ( δ ′ ) log( T [Equation] δ ′ ) , 8 α log( T/δ ′ )
Revised: [Equation] It is easy to verify that with the choice x = max (cid:110) 4 (cid:112) R ( δ ′ ) log( T/δ ′ ) , 8 α log( T/δ ′ )

Original: (cid:111) , we have
Revised: (cid:111) , we have

Revised: Pr

Original: [Equation]
Revised: [Equation]

Original: Therefore, with probability at least 1 − 2 δ ′ ,max t =0 , 1 ,...,T [Equation] [Table] [Equation]
Revised: Therefore, with probability at least 1 − 2 δ ′ ,

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: where (i) is by eq.
Revised: where (i) is by eq.

Original: (8) and eq.
Revised: (8) and eq.

Original: (10), (ii) is by the fact that max { a, b } ≤ a + b and √ a + b ≤ √ a + √ b for any a, b ≥ 0 , (iii) is by Lemma A.1.
Revised: (10), (ii) is by the fact that max { a, b } ≤ a + b and √ a + b ≤ √ a + √ b for any a, b ≥ 0 , (iii) is by Lemma A.1.

Original: Substitute δ ′ with δ/ 2 , we obtain the desired result.
Revised: Substitute δ ′ with δ/ 2 , we obtain the desired result.

Original: C. P ROOF OF P ROPOSITION 4. 
Revised: C. P ROOF OF P ROPOSITION 4.

Original: Proof.
Revised: Proof.

Original: By the assumption that f i ’s are L -smooth, we know that f is also L -smooth.
Revised: By the assumption that f i ’s are L -smooth, we know that f is also L -smooth.

Original: Therefore f is (2 L, 0) -GC
Revised: Therefore f is (2 L, 0) -WGC

Original: (By Lemma 3.2), e.g.,
Revised: (By Lemma 3.2), e.g.,

Original: [Equation]
Revised: [Equation]

Original: Apply Proposition 4.2.
Revised: Apply Proposition 4.2.

Original: We obtain that
Revised: We obtain that

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − δ for some absolute positive constants c 1 , c 2 .
Revised: with probability at least 1 − δ for some absolute positive constants c 1 , c 2 .

Original: The remaining is to bound the individual gradient norm ∥∇ f i ( w ( t ) )
Revised: The remaining is to bound the individual gradient norm ∥∇ f i ( w ( t ) )

Original: ∥ .
Revised: ∥ .

Original: When Assumption 4.1 holds, we have that
Revised: When Assumption 4.1 holds, we have that

Original: [Equation]
Revised: [Equation]

Original: By Assumption 4.1, we know that ∥∇ f i ( w ) − ∇ f ( w ) ∥ is sub-Gaussian for all w ∈ R d .
Revised: By Assumption 4.1, we know that ∥∇ f i ( w ) − ∇ f ( w ) ∥ is sub-Gaussian for all w ∈ R d .

Original: Therefore we can apply Lemma A.7 to bound the maximum of sub-Gaussian variables, which gives
Revised: Therefore we can apply Lemma A.8 to bound the maximum of sub-Gaussian variables, which gives

Original: [Equation]
Revised: [Equation]

Original: i

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: for some absolute positive constant c 3 > 0 and any δ ∈ (0 , 1) .
Revised: for some absolute positive constant c 3 > 0 and any δ ∈ (0 , 1) .

Original: Combining eq.
Revised: Combining eq.

Original: (11) and eq.
Revised: (11) and eq.

Original: We obtain that
Revised: We obtain that

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − 2 δ , which yields the desired result eq.
Revised: with probability at least 1 − 2 δ , which yields the desired result eq.

Original: (2) under Assumption 4.1.
Revised: (2) under Assumption 4.1.

Original: When Assumption 4.1 does not hold.
Revised: When Assumption 4.1 does not hold.

Original: [Equation]
Revised: [Equation]

Original: where the last line is by Proposition 4.2.
Revised: where the last line is by Proposition 4.2.

Original: This completes the proof for eq.
Revised: This completes the proof for eq.

Revised: [Equation]

Original: C. P ROOF OF P ROPOSITION 4. 
Revised: C. P ROOF OF P ROPOSITION 4.

Original: Proof.
Revised: Proof.

Original: We first consider the case that f i ’s are convex and smooth.
Revised: We first consider the case that f i ’s are convex and smooth.

Original: By Lemma 3.2, we know that f i ’s are (2 L, 0) -GC in this scenario.
Revised: By Lemma 3.2, we know that f i ’s are (2 L, 0) -WGC in this scenario.

Original: Setting
Revised: Setting

Original: [Equation]
Revised: [Equation]

Original: where c 1 , c 2 , c 3 correspond to the absolute positive constants that appeared in the proof of Proposition 4.2, c 4 is some positive constant that appeared in Theorem 3.4.
Revised: where c 1 , c 2 , c 3 correspond to the absolute positive constants that appeared in the proof of Proposition 4.2, c 4 is some positive constant that appeared in Theorem 3.4.

Original: Next we need to apply Proposition 4.3.
Revised: Next we need to apply Proposition 4.3.

Original: Denote B t as the batch sampled at the t -th iteration and let
Revised: Denote B t as the batch sampled at the t -th iteration and let

Original: [Equation]
Revised: [Equation]

Original: By the definition of ξ ( t ) and Assumption 4.1, ζ ( t ) ’s satisfy
Revised: By the definition of ξ ( t ) and Assumption 4.1, ζ ( t ) ’s satisfy

Original: [Equation]
Revised: [Equation]

Original:  where c 5 is some positive absolute constant.
Revised: (by the property of Poisson sampling) , where c 5 is some positive absolute constant.

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: Then Proposition 4.3 tells us that Algorithm 2 with the above setup of ˜ σ and η will produce iterates such that
Revised: Then Proposition 4.3 tells us that Algorithm 2 with the above setup of ˜ σ and η will produce iterates such that

Original: [Equation]
Revised: [Equation]

Original: with probability 1 − δ ′ .
Revised: with probability 1 − δ ′ .

Original: Therefore Algorithm 1 with theabove parameter setup (eq. (13) and η defined as in eq. 
Revised: The above analysis is based on Algorithm 2. Next, we draw the connection between Algorithmwith the above setup of parameters and Algorithm 1 with the parameter setup in eq. (13) and η defined as in eq. We introduce some new notation to help our analysis.

Original: (14)) will be equivalent to Algorithm 2 (with ˜ σ, η defined as in eq.
Revised: To distinguish between Algorithm 2 (SGD) and Algorithm 1 (DP-SGD-GC), we denote { ˜ w ( t ) } Tt =0 as the iterates from Algorithm 2 and { w ( t ) }

Original: (14)) with probability at least 1 − δ ′ .
Revised: Tt =0 as the iterates from Algorithm 1. We further let (cid:101) E t and E t as the event {∥∇ f i ( ˜ w ( t ) ) ∥ ≤ C, ∀ i ∈ [ n ] } and {∥∇ f i ( w ( t ) ) ∥ ≤ C, ∀ i ∈ [ n ] } respectively.

Revised: For two random variables A and B , we denote A ∼ B if A and B are independent and identically distributed.

Revised: Consider two independent runs of Algorithm 2 and Algorithm 1 with ˜ w (0)

Revised: =

Revised: w (0) , we are going to show that

Revised: • [ ˜ w (0) , . . .

Revised: , ˜ w ( T ) ] conditioned on the event ∩

Revised: Tt =0 (cid:101) E t has the same distribution as [ w (0) , . . .

Revised: , w ( T ) ] conditional on the event ∩

Original: {+Tt =
+}
Revised: {+Tt =
+}

Original: {+E t ; • Pr[ ∩ Tt =
+}
Revised: {+E t ; • Pr[ ∩ Tt =
+}

Revised: (cid:101) E t ] =

Original: {+Pr[ ∩ Tt =
+}
Revised: {+Pr[ ∩ Tt =
+}

Revised: E t ] .

Revised: The first conclusion should be obvious.

Revised: Given ˜ w (0) =

Revised: w (0) , we can conclude that ˜ w (1) ∼ w (1) conditioned on (cid:101) E 0 and E 0 (both gradients are bounded), which further implies ˜ w (2) ∼ w (2) conditioned on (cid:101) E 1 and E 1 and so on.

Revised: For the second conclusion, we prove by induction.

Revised: Given ˜ w (0) =

Revised: w (0) , the base case

Revised: Pr[ (cid:101) E 0 ]

Revised: =

Original: Thus it is straightforward that Pr[  E  ] ≥ 1 − δ ′ (recall that{+= Pr[ ∩ mt =
+} E is the event of no gradient clipping happens) .{+t ] , we are going to prove Pr[ ∩ m +1 t =
+}
Revised: Pr[ E 0 ] is obviously true. Next, given Pr[ ∩ mt =0 (cid:101) E t ] {+= Pr[ ∩ mt =
+} E {+t ] , we are going to prove Pr[ ∩ m +1 t =
+}

Revised: (cid:101) E t ] =

Revised: Pr[ ∩ m +1 t =0 E t ] .

Revised: Knowing that

Revised: [Equation]

Revised: (by Lemma A.3) .

Revised: We only need to prove that Pr[ ∩ pt =0 (cid:101) E

Revised: [Equation]

Revised: ∩

Revised: [Equation]

Revised: E cp +1 ] =

Original: {+Pr[ ∩ pt =
+}
Revised: {+Pr[ ∩ pt =
+}

Revised: E t ∩ E cp +1 ] ∀ p = 0 , 1 , . . .

Original: {+, m , which is equivalent to Pr[ (cid:101) E cp +
+}
Revised: {+, m , which is equivalent to Pr[ (cid:101) E cp +
+}

Revised: | ∩ pt =0 (cid:101) E t ]

Revised: Pr[ ∩ pt =0 (cid:101) E t ] =

Revised: Pr[ E cp +1 | ∩ pt =0 E t ]

Revised: Pr[ ∩ pt =0 E t ] .

Original: {+By induction, suppose Pr[ ∩ pt =
 (cid:101) E t ] = Pr[ ∩ pt =0 E t ] holds.+} Conditioning on the event E , Algorithm 1 reduces to Algorithm 2.{+two events ∩ pt =
+}
Revised: {+By induction, suppose Pr[ ∩ pt =
 (cid:101) E t ] = Pr[ ∩ pt =0 E t ] holds.+} Conditioning on the {+two events ∩ pt =
+}

Original: Notice that the term E [ ∥ ξ ( {+p +1) and therefore Pr[ (cid:101) E cp +
 | ∩ pt =0 (cid:101) E+} t) ∥ 2 ] in
Revised: (cid:101) E t and ∩ pt =0 E t , it is obvious that ˜ w ( p +1) ∼ w ( {+p +1) and therefore Pr[ (cid:101) E cp +
 | ∩ pt =0 (cid:101) E+} t ] = Pr[ E cp +1 | ∩ pt =0 E t ] .

Revised: Combining the above together, we prove that Pr[ ∩ m +1 t =0 E t ] and finish the induction.

Revised: With the above two conclusions, we can transfer the convergence of Algorithm 2 conditioned on the event ∩

Original: Lemma A.11 is bounded by ˜ σ 2 , then the classic{+event ∩ Tt =
 E t . Next, the conditional+} convergencerate of SGD (Lemma A.11) gives that
Revised: Tt =0 (cid:101) E t to Algorithm 1 conditioned on the {+event ∩ Tt =
 E t . Next, the conditional+} convergence of SGD.

Revised: The proof is simply based on the law of total expectation.

Revised: Denote ˜ w out as the output of Algorithm 2 and (cid:101) E =

Revised: ∩

Revised: Tt =0 (cid:101) E t .

Original: [Equation]
Revised: [Equation]

Revised: where (i) is by the fact that Pr[ (cid:101) E ] ≥ 1 − δ ′ ≥ 0 .

Revised: Now we can transfer the convergence of conditional SGD to conditional DP-SGD-GC.

Revised: Notice that the term E [ ∥ ξ ( t ) ∥ 2 ] in Lemma A.12 is bounded by ˜ σ 2 , then the classic convergence rate of SGD (Lemma A.12) gives that

Original: [Equation]
Revised: [Equation]

Original:  where (i) is by the definition of η , (ii) is by the definition of ˜ σ , (iii) is by the definition of σ (eq.
Revised: D [Equation] where (i) is by the definition of η , (ii) is by the definition of ˜ σ , (iii) is by the definition of σ (eq.

Original: (13)), (iv) is by the definition of q = B/n , (v) is by noticing C
Revised: (13)), (iv) is by the definition of q = B/n , (v) is by noticing C

Original: =
Revised: =

Original: O
Revised: O

Original: (log(1 /δ ′ )
Revised: (log(1 /δ ′ )

Original: + log( n ) + log( T )) from eq.
Revised: + log( n ) + log( T )) from eq.

Original: When f i ’s are L -smooth but not necessarily convex.
Revised: When f i ’s are L -smooth but not necessarily convex.

Original: We set the parameters the same as for the convex case.
Revised: We set the parameters the same as for the convex case.

Original: Apply the convergence rate of SGD for smooth but not necessarily convex functions (Lemma A.11).
Revised: Apply the convergence rate of SGD for smooth but not necessarily convex functions (Lemma A.12).

Original: Following the same proof template as for the convex case, can we obtain the desired result.
Revised: Following the same proof template as for the convex case, can we obtain the desired result.

Original: C. P ROOF OF P ROPOSITION B. 
Revised: C. P ROOF OF P ROPOSITION B.

Original: Proof.
Revised: Proof.

Original: The proof is almost idential as the proof of Proposition 4.4, the difference is that we need to set
Revised: The proof is almost idential as the proof of Proposition 4.4, the difference is that we need to set

Original: [Equation]
Revised: [Equation]

Original: where c ′ is some positive absolute constant.
Revised: where c ′ is some positive absolute constant.

Original: Note that the clipping threshold C now involves an additional √ n term.
Revised: Note that the clipping threshold C now involves an additional √ n term.

Original: The following the same proof template as Proposition 4.4, we get obtain
Revised: The following the same proof template as Proposition 4.4, we get obtain

Original: [Equation]
Revised: [Equation]

Original: which yields the desired result.
Revised: which yields the desired result.

Original: Follow the same proof template, the above bound also holds for E [ ∥ f ( w priv ) ∥ 2 ] when f is nonconvex.
Revised: Follow the same proof template, the above bound also holds for E [ ∥ f ( w priv ) ∥ 2 ] when f is nonconvex.

Original: C. P ROOF OF T HEOREM 4. 
Revised: C. P ROOF OF T HEOREM 4.

Original: Proof.
Revised: Proof.

Original: The outline of the proof is summarized as follows.
Revised: The outline of the proof is summarized as follows.

Original: • First we show that the excess empirical risk e.g., f ( w priv ) − f ∗ or gradient norm square
Revised: • First we show that the excess empirical risk e.g., f ( w priv ) − f ∗ or gradient norm square

Original: e.g., ∥∇ f ( w priv ) ∥ 2 are sub-exponential random variables.
Revised: e.g., ∥∇ f ( w priv ) ∥ 2 are sub-exponential random variables.

Original: We also derive a pessimistic upper bound of their sub-exponential parameters (Lemma C.1) 4 .
Revised: We also derive a pessimistic upper bound of their sub-exponential parameters (Lemma C.1) 4 .

Original: • Next, we develop a technical tool to convert conditional expected error bound (the conditioning event happens with high probability) to expected error bound for sub-exponential random variables (Lemma A.8).
Revised: • Next, we develop a technical tool to convert conditional expected error bound (the conditioning event happens with high probability) to expected error bound for sub-exponential random variables (Lemma A.9).

Original: • Finally, by tuning δ ′ (the probability that gradient clipping happens during training; see Proposition 4.4), Lemma A.8 together with Lemma C.1 and Proposition 4.4 yield the desired result.
Revised: • Finally, by tuning δ ′ (the probability that gradient clipping happens during training; see Proposition 4.4), Lemma A.9 together with Lemma C.1 and Proposition 4.4 yield the desired result.

Original: Now we proceed to the detailed proof.
Revised: Now we proceed to the detailed proof.

Original: Consider the case that f i ’s are convex and L -smooth.
Revised: Consider the case that f i ’s are convex and L -smooth.

Original: Let
Revised: Let

Original: T, σ, C, η satisfy eq.
Revised: T, σ, C [Equation] η satisfy eq.

Original: We apply Lemma A.8 by making the identification
Revised: We apply Lemma A.9 by making the identification

Original: [Equation]
Revised: [Equation]

Original: where the choice of α is by Proposition 4.4 and the choice of β and K is by Lemma C.1.
Revised: where the choice of α is by Proposition 4.4 and the choice of β and K is by Lemma C.1.

Original: Then
Revised: Then

Original: Lemma A.8 gives that
Revised: Lemma A.9 gives that

Original: [Equation]
Revised: [Equation]

Original: where (i) is true by setting
Revised: where (i) is true by setting

Original: [Equation]
Revised: [Equation]

Original: Note that the above proof template works as long as we can construct β, K that polynomially depends on T, n, d, ϵ — we can tune δ ′ to convert all polynomial terms in β and K to polylogarithm terms by
Revised: Note that the above proof template works as long as we can construct β, K that polynomially depends on T, n, d, ϵ — we can tune δ ′ to convert all polynomial terms in β and K to polylogarithm terms by

Original: Lemma A.8.
Revised: Lemma A.9.

Original: We find Lemma A.8 a quite convenient technical tool for our analysis and we are not aware if this result exist in the literature.
Revised: We find Lemma A.9 a quite convenient technical tool for our analysis and we are not aware if this result exist in the literature.

Original: When f i ’s are smooth but not necessarily convex.
Revised: When f i ’s are smooth but not necessarily convex.

Original: The proof follows exactly the same as the convex case (just need to let X := ∥∇ f ( w priv ) ∥ 2 , β :=
Revised: The proof follows exactly the same as the convex case (just need to let X := ∥∇ [Equation]

Original: 4 L ( f ( w (0) )
Revised: w priv

Original: − f ∗ )
Revised: [Equation]

Original: + 2 T 2 C 2 , K = 2 T 2 C 2 dσ 2 c 1 Band apply eq.
Revised: and apply eq.

Original: (16) instead of eq.
Revised: (17) instead of eq.

Original: We omit the details to avoid tedious repetition.
Revised: We omit the details to avoid tedious repetition.

Original: Lemma C.1 (A pessimistic bound for DP-SGD-GC) .
Revised: Lemma C.1 (A pessimistic bound for DP-SGD-GC) .

Original: Assume that f i ’s are L -smooth.
Revised: Assume that f i ’s are L -smooth.

Original: Let w priv be the output from Algorithm 1 with η ≤ 1 /L .
Revised: Let w priv be the output from Algorithm 1 with η ≤ 1 /L .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Furthermore,
Revised: Furthermore,

Original: [Equation]
Revised: [Equation]

Revised: The

Original: Proof.
Revised: Proof.

Original: We begin with the smoothness of f , for any t ∈ { 0 , 1 , . . .
Revised: We begin with the smoothness of f , for any t ∈ { 0 , 1 , . . .

Original: , T }
Revised: , T }

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] [Table] 
Revised: [Equation] [Table] [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where (i) is by the fact that ⟨ a, b ⟩ ≤ 12 λ ∥ a ∥ 2 + λ 2 ∥
Revised: where (i) is by the fact that ⟨ a, b ⟩ ≤ 12 λ ∥ a ∥ 2 + λ 2 ∥

Original: b ∥ 2 ∀ λ > 0 , (ii) is by the convexity of ∥ · ∥ 2 , (iii) is because of ∥ ˜ g
Revised: b ∥ 2 ∀ λ > 0 , (ii) is by the convexity of ∥ · ∥ 2 , (iii) is because of ∥ ˜ g

Original: ( i ) j ∥ ≤ C due to gradient clipping, (iv) is by the  growth condition, (v) is by the assumption on learning rate ( η ≤ 1 /L ) 
Revised: ( i ) j ∥ ≤ C due to gradient clipping, (iv) is by the weak growth condition, (v) is by the assumption on learning rate ( η ≤ 1 /L ) .

Original: [Equation]

Original: By the definition of ξ ( i ) , Z T follows from the sub-exponential distribution.
Revised: By the definition of ξ ( i ) , Z T follows from the sub-exponential distribution.

Original: By Lemma A.3, there exist some absolute constant c 1 > 0 such that
Revised: By Lemma A.4, there exist some absolute constant c 1 > 0 such that

Original: [Equation]
Revised: [Equation]

Original: Combining with eq.
Revised: Combining with eq.

Original: [Equation] we have that
Revised: (18), we have that

Original: Pr

Original: [Equation]
Revised: [Equation]

Original: which finishes the proof for eq.
Revised: which finishes the proof for eq.

Original: By the  growth condition [Equation] we further have that
Revised: By the weak growth condition, we further have that

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: which finishes the prooffor eq.
Revised: which finishes the proof

Original: C. P ROOF OF T HEOREM 4. 
Revised: C. P ROOF OF T HEOREM 4.

Original: Proof.
Revised: Proof.

Original: The proof follows the same proof template of Theorem 4.5, we only need to change the setting of C and ˜ σ .
Revised: The proof follows the same proof template of Theorem 4.5, we only need to change the setting of C and ˜ σ .

Original: A PPENDIX D M ISSING EXPERIMENTS  ON SYNTHETIC DATA
Revised: A PPENDIX D M ISSING EXPERIMENTS D. E XPERIMENTS ON SYNTHETIC DATA

Revised: The light-tail-noise assumption may not hold when training neural networks on real-world data.

Original: We conduct experiments on synthetic data in this section.
Revised: To keep consitent with our theory, we conduct experiment with synthetic data and artificial Gaussian noise.

Original: The synthetic data has 10 , 000 samples and each sample is generated from a 256-dimensional standard Gaussian distribution and a ground truth linear model.
Revised: Our synthetic data has 10 , 000 samples and each sample is generated from a 256-dimensional standard Gaussian distribution and a ground truth linear model.

Original: To simulate stochastic gradient with light-tailed noise, we perform linear regression and add standard Gaussian noise (light-tailed) for the true gradient.
Revised: To simulate stochastic gradient with light-tailed noise, we perform linear regression and add standard Gaussian noise (light-tailed) for the true gradient.

Original: The experimental results are shown in Figure 4.
Revised: The experimental results are shown in Figure 4.

Original: The conclusion is the same as in Section 6: the evolution of clipping frequency has a decreasing trend and our value clipping technique can achieve similar accuracy as gradient clipping.
Revised: The conclusion is the same as in Section 6: the evolution of clipping frequency has a decreasing trend and our value clipping technique can achieve similar accuracy as gradient clipping.

Revised: The experiment in this part indicates that training neural networks on MNIST and CIFAR also enjoys light-tail-noise or the light-tail-noise assumption is removable.

Original: A PPENDIX E M ORE  ON THE GROWTH CONDITION
Revised: D. M ORE EXPERIMENTS ON TESTING ACCURACY

Revised: In this section, we compare the testing accuracy of DP-SGD-GC (the baseline method) and DP-SGDVC with different setup of ϵ and δ .

Revised: We set the noise level σ ∈ { 0 . 5 , 1 , 2 , 4 , 8 } and confidence level δ ∈ { 10 − 3 , 10 − 4 , 10 − 5 } ; each ( σ, δ ) pair decides a privacy parameter ϵ , which can be calculated by the Opacus package.

Revised: The result are shown in Figure 5.

Revised: We can observe that the performance gap between WGC and VC grows as the privacy level ϵ becomes larger.

Revised: For linear model, the testing accuracy gap is ∼ 2% when ϵ ∼ 10 and the gap is ∼ 5% when ϵ ∼ 0 .

Revised: For two-layer neural networks, the performance gap is larger, the testing accuracy gap is ∼ 3% when ϵ ∼ 10 and the gap is about ∼ 9% when ϵ ∼ 0 .

Revised: The experimental result should not be too surprising; imposing smaller ϵ implies adding more noise to the gradient, which will further increase the gradient norm and clipping frequency, and will eventually amplify the error created by VC.

Revised: A PPENDIX E M ORE ON THE WEAK GROWTH CONDITION

Original: We review some existing results and describe the  growth condition for feed-forward neural networks with cross-entropy loss.
Revised: We review some existing results and describe the weak growth condition for feed-forward neural networks with cross-entropy loss.

Original: First, we review the generalized growth condition (Fang et al., 2021, Proposition 4.1).
Revised: First, we review the generalized growth condition (Fang et al., 2021, Proposition 4.1).

Original: Consider the objective
Revised: Consider the objective

Original: [Equation]
Revised: [Equation]

Original: where ℓ : R → R ≥ 0 is a nonnegative 1-dimensional loss function that is convex , 1 -smooth , and satisfies inf ℓ = 0 .
Revised: where ℓ : R → R ≥ 0 is a nonnegative 1-dimensional loss function that is convex , 1 -smooth , and satisfies inf ℓ = 0 .

Original: The functions h i ’s are assumed to be β -Lipschitz continuous for some β > 0 .
Revised: The functions h i ’s are assumed to be β -Lipschitz continuous for some β > 0 .

Original: Then f i ’s and f are shown to satisfy the following  growth condition.
Revised: Then f i ’s and f are shown to satisfy the following weak growth condition.

Original: Lemma E.1 (Fang et al., 2021, Proposition 4.1) .
Revised: Lemma E.1 (Fang et al., 2021, Proposition 4.1) .

Original: For any w ∈ R d ,

Original: [Equation] , and [Equation] where ∂f ( w ) is the Clarke’s generalized gradient (Clarke, 1981).
Revised: For any w ∈ R d , ∥ ∂f i ( w ) ∥ 2 ≤ 2 β 2 f i ( w ) ∀ i ∈ [ n ] , and ∥ ∂f ( w ) ∥ 2 ≤ 2 β 2 f ( w ) , where ∂f ( w ) is the Clarke’s generalized gradient (Clarke, 1981).

Original: We refer readers to Fang et al.
Revised: We refer readers to Fang et al.

Original: (Fang et al., 2021) for more details on the derivation of the above lemma.
Revised: (Fang et al., 2021) for more details on the derivation of the above lemma.

Original: Note that f i ’s and f in eq.
Revised: Note that f i ’s and f in eq.

Original: (18) can be nonconvex, and the above lemma suggests that the  growth condition holds for a certain class of nonconvex ERM.
Revised: (19) can be nonconvex, and the above lemma suggests that the weak growth condition holds for a certain class of nonconvex ERM.

Original: E. G ROWTH CONDITION FOR FEED - FORWARD NEURAL NETWORKS WITH CROSS - ENTROPY LOSS
Revised: E. W EAK GROWTH CONDITION FOR FEED - FORWARD NEURAL NETWORKS WITH CROSS - ENTROPY LOSS

Original: We give a simple extension of Lemma E.1 to feed-forward neural networks with cross-entropy loss.
Revised: We give a simple extension of Lemma E.1 to feed-forward neural networks with cross-entropy loss.

Original: We denote the number of classes as K .
Revised: We denote the number of classes as K .

Original: For simplicity, we consider a feed-forward neural network with fixed width m .
Revised: For simplicity, we consider a feed-forward neural network with fixed width m .

Original: We denote the parameter of a H -layer feed-forward neural network as W :=
Revised: We denote the parameter of a H -layer feed-forward neural network as W :=

Original: ( W 1 , . . .
Revised: ( W 1 , . . .

Original: , W H ) , where W 1 ∈ R m × d ,
Revised: , W H ) , where W 1 ∈ R m × d ,

Original: W H ∈ R K
Revised: W H ∈ R K

Original: × m and W i ∈ R m × m , i = 2 , 3 , . .
Revised: × m and W i ∈ R m × m , i = 2 , 3 , . .

Original: , H − 1 .
Revised: , H − 1 .

Original: We further denote ∥ W ∥ 2 F = (cid:80) Hi =1 ∥ W i ∥ 2 F .
Revised: We further denote ∥ W ∥ 2 F = (cid:80) Hi =1 ∥ W i ∥ 2 F .

Original: With a little abuse of notation, we denote σ as the ReLU activation, e.g., σ ( x ) =
Revised: With a little abuse of notation, we denote σ as the ReLU activation, e.g., σ ( x ) =

Original: (max { x 1 , 0 } , . . .
Revised: (max { x 1 , 0 } , . . .

Original: , max { x m , 0 } ) .
Revised: , max { x m , 0 } ) .

Original: Note that neural networks with the ReLU activation is not differentiable everywhere on its domain.
Revised: Note that neural networks with the ReLU activation is not differentiable everywhere on its domain.

Original: We define the “gradient” of a ReLU neural networks in the same way as in (Allen-Zhu et al., 2019, Fact 2.6).
Revised: We define the “gradient” of a ReLU neural networks in the same way as in (Allen-Zhu et al., 2019, Fact 2.6).

Original: We consider a single training sample x (which is enough for our purpose) and define the architecture of the H -layer feed-forward neural network as
Revised: We consider a single training sample x (which is enough for our purpose) and define the architecture of the H -layer feed-forward neural network as

Original: [Equation]
Revised: [Equation]

Original: where h j is the hidden variables of the j -th layer, ˆ y is the prediction produced by the network.
Revised: where h j is the hidden variables of the j -th layer, ˆ y is the prediction produced by the network.

Original: Without loss of generality, we assume that the label for our training sample is the first class.
Revised: Without loss of generality, we assume that the label for our training sample is the first class.

Original: Then by the definition of the cross-entropy loss, we have that
Revised: Then by the definition of the cross-entropy loss, we have that

Original: [Equation]
Revised: [Equation]

Original: Making the identification 
Revised: Making the identification ℓ ( α

Original: [Equation]
Revised: [Equation]

Original:  and
Revised: = log(1 + exp( α )) and

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: It is obvious that ℓ satisfy the assumptions made by Lemma E.1.
Revised: It is obvious that ℓ satisfy the assumptions made by Lemma E.1.

Original: In order to apply Lemma E.1, all the remaining is to bound (cid:13)(cid:13)(cid:13) ∂g
Revised: In order to apply Lemma E.1, all the remaining is to bound (cid:13)(cid:13)

Original: (ˆ y )
Revised: [Equation]

Original: ∂ W (cid:13)(cid:13)(cid:13) .
Revised: ∂ W (cid:13)(cid:13) (cid:13) .

Original: First, we notice that
Revised: First, we notice that

Original: [Equation]
Revised: [Equation]

Original: = ⇒ ∥∇ ˆ y g
Revised: = ⇒ ∥∇ ˆ y g

Original: (ˆ y ) ∥ 

Revised: (ˆ y ) ∥

Original: ==
Revised: ==

Original: ⇒ ∥∇ ˆ y g
Revised: ⇒ ∥∇ ˆ y g

Original: (ˆ y ) ∥ 2 ≤ 2 .
Revised: (ˆ y ) ∥ 2 ≤ 2 .

Original: [Equation]

Original: Then we need to bound (cid:13)(cid:13)(cid:13) ∂ ˆ y ∂ W (cid:13)(cid:13)(cid:13) .
Revised: Then we need to bound (cid:13)(cid:13)(cid:13) ∂ ˆ y ∂ W (cid:13)(cid:13)(cid:13) .

Original: Directly apply (Allen-Zhu et al., 2019, Fact 2.6) gives that
Revised: Directly apply (Allen-Zhu et al., 2019, Fact 2.6) gives that

Original: [Equation]
Revised: [Equation]

Original:  for all i ∈ [ H ] .
Revised: (21) for all i ∈ [ H ] .

Original: Combining Lemma E.1, eq.
Revised: Combining Lemma E.1, eq.

Original: (19) and eq.
Revised: (20) and eq.

Original: (20), we obtain the following  growth condition
Revised: (21), we obtain the following weak growth condition

Original: [Equation]
Revised: [Equation]

Original:  Furthermore, by ∥∇ ˆ y g (ˆ y ) ∥ 2 ≤ 2 and (Allen-Zhu et al., 2019, Fact 2.6).
Revised: W [Equation] Furthermore, by ∥∇ ˆ y g (ˆ y ) ∥ 2 ≤ 2 and (Allen-Zhu et al., 2019, Fact 2.6).

Original: We also have ∥ 
Revised: We also have ∥ ∂f

Original: [Equation]
Revised: [Equation]

Revised: W

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: (cid:88) i =

Revised: [Equation]

Revised: (cid:89)

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: =

Revised: [Equation]

Original: ∥ 22 .
Revised: ∥ 22 .

Original: To sum up, we obtain ∥ ∂f (W ) ∥
Revised: To sum up, we obtain ∥ ∂f (

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: To access the GC-parameters, we need to compute ∥W j ∥ 2 , j ∈
Revised: To access the WGC-parameters, we need to compute ∥

Original: [ H ] in each iteration, the overhead can be amortized by increasing batch size.