Original: Stability of SGD: Tightness Analysis and Improved Bounds
Revised: Stability of SGD: Tightness Analysis and Improved Bounds

Original: Abstract
Revised: Abstract

Original: Stochastic Gradient Descent (SGD) based methods have been widely used for training large-scale machine learning models that also generalize well in practice.
Revised: Stochastic Gradient Descent (SGD) based methods have been widely used for training large-scale machine learning models that also generalize well in practice.

Original: Several explanations have been offered for this generalization performance, a prominent one being algorithmic stability [Hardt et al., 2016].
Revised: Several explanations have been offered for this generalization performance, a prominent one being algorithmic stability [Hardt et al., 2016].

Original: However, there are no known examples of smooth loss functions for which the analysis can be shown to be tight.
Revised: However, there are no known examples of smooth loss functions for which the analysis can be shown to be tight.

Original: Furthermore, apart from properties of the loss function, data distribution has also been shown to be an important factor in generalization performance.
Revised: Furthermore, apart from properties of the loss function, data distribution has also been shown to be an important factor in generalization performance.

Original: This raises the question: is the stability analysis of [Hardt et al., 2016] tight for smooth functions, and if not, for what kind of loss functions and data distributions can the stability analysis be improved?
Revised: This raises the question: is the stability analysis of [Hardt et al., 2016] tight for smooth functions, and if not, for what kind of loss functions and data distributions can the stability analysis be improved?

Original: In this paper we first settle open questions regarding tightness of bounds in the data-independent setting: we show that for general datasets, the existing analysis for convex and strongly-convex loss functions is tight, but it can be improved for non-convex loss functions.
Revised: In this paper we first settle open questions regarding tightness of bounds in the data-independent setting: we show that for general datasets, the existing analysis for convex and strongly-convex loss functions is tight, but it can be improved for non-convex loss functions.

Original: Next, we give novel and improved data-dependent bounds: we show stability upper bounds for a large class of convex regularized loss functions, with negligible regularization parameters, and improve existing data-dependent bounds in the non-convex setting.
Revised: Next, we give novel and improved data-dependent bounds: we show stability upper bounds for a large class of convex regularized loss functions, with negligible regularization parameters, and improve existing data-dependent bounds in the non-convex setting.

Original: We hope that our results will initiate further efforts to better understand the data-dependent setting under non-convex loss functions, leading to an improved understanding of the generalization abilities of deep networks.
Revised: We hope that our results will initiate further efforts to better understand the data-dependent setting under non-convex loss functions, leading to an improved understanding of the generalization abilities of deep networks.

Original:  INTRODUCTION
Revised:  INTRODUCTION

Original: Stochastic gradient descent (SGD) has gained great popularity in solving machine learning optimization problems [Kingma and Ba, 2014, Johnson and Zhang, 2013].
Revised: Stochastic gradient descent (SGD) has gained great popularity in solving machine learning optimization problems [Kingma and Ba, 2014, Johnson and Zhang, 2013].

Original: SGD leverages the finite-sum structure of the objective function, avoids the expensive computation of exact gradients, and thus provides a feasible and efficient optimization solution in large-scale settings [Bottou, 2012].
Revised: SGD leverages the finite-sum structure of the objective function, avoids the expensive computation of exact gradients, and thus provides a feasible and efficient optimization solution in large-scale settings [Bottou, 2012].

Original: The convergence and the optimality of SGD have been thoroughly studied [Ge et al., 2015, Rakhlin et al., 2012, Reddi et al., 2018,
Revised: The convergence and the optimality of SGD have been thoroughly studied [Ge et al., 2015, Rakhlin et al., 2012, Reddi et al., 2018,

Original: Zhou and Gu, 2019, Carmon et al., 2019a,b, Shamir and
Revised: Zhou and Gu, 2019, Carmon et al., 2019a,b, Shamir and

Original: Zhang, 2013].
Revised: Zhang, 2013].

Original: In recent years, new research questions have been raised regarding SGD’s impact on a model’s generalization power.
Revised: In recent years, new research questions have been raised regarding SGD’s impact on a model’s generalization power.

Original: The seminal work [Hardt et al., 2016] tackled the problem using the algorithmic stability of SGD, i.e., the progressive sensitivity of the trained model w.r.t.
Revised: The seminal work [Hardt et al., 2016] tackled the problem using the algorithmic stability of SGD, i.e., the progressive sensitivity of the trained model w.r.t.

Original: the replacement of a single (test) datum in the training set.
Revised: the replacement of a single (test) datum in the training set.

Original: They showed that the generalization error of an SGD-trained model is upper bounded by a uniform stability parameter ε stab , and relate ε stab to the divergence of the two parameter vectors obtained by training on twin datasets.
Revised: They showed that the generalization error of an SGD-trained model is upper bounded by a uniform stability parameter ε stab , and relate ε stab to the divergence of the two parameter vectors obtained by training on twin datasets.

Original: This stability-based analysis of the generalization gap allows one to bypass classical model capacity theorems
Revised: This stability-based analysis of the generalization gap allows one to bypass classical model capacity theorems

Original: [Vapnik, 1998, Koltchinskii and Panchenko, 2000] or weight-based complexity theorems [Neyshabur et al., 2017, Bartlett et al., 2017, Arora et al., 2018].
Revised: [Vapnik, 1998, Koltchinskii and Panchenko, 2000] or weight-based complexity theorems [Neyshabur et al., 2017, Bartlett et al., 2017, Arora et al., 2018].

Original: This framework also provides theoretical insights into many phenomena observed in practice, e.g., the “train faster, generalize better” phenomenon, the power of regularization techniques such as weight decay [Krogh and Hertz, 1992], dropout [Srivastava et al., 2014], and gradient clipping.
Revised: This framework also provides theoretical insights into many phenomena observed in practice, e.g., the “train faster, generalize better” phenomenon, the power of regularization techniques such as weight decay [Krogh and Hertz, 1992], dropout [Srivastava et al., 2014], and gradient clipping.

Original: Other works have developed the stability notion with advanced analysis [Bassily et al., 2020, Feldman and Vondrak, 2019, Kuzborskij and Lampert, 2018, Lei and Ying, 2020b, Lei et al., 2021a, Lei and Ying, 2020a] and adapted it into more sophisticated settings such as Stochastic Gradient Langevin Dynamics and momentum
Revised: Other works have developed the stability notion with advanced analysis [Bassily et al., 2020, Feldman and Vondrak, 2019, Kuzborskij and Lampert, 2018, Lei and Ying, 2020b, Lei et al., 2021a, Lei and Ying, 2020a] and adapted it into more sophisticated settings such as Stochastic Gradient Langevin Dynamics and momentum

Original: SGD [Mou et al., 2018, Chaudhari et al., 2019, Chen et al.,
Revised: SGD [Mou et al., 2018, Chaudhari et al., 2019, Chen et al.,

Original: 2018, Li et al., 2020, Lei et al., 2021b].
Revised: 2018, Li et al., 2020, Lei et al., 2021b].

Original: Despite the promises of this stability-based analysis, it remains open whether the analysis in [Hardt et al., 2016] can be further improved to reveal the full potential of the stability method, either in general or for specific data-distributions.
Revised: Despite the promises of this stability-based analysis, it remains open whether the analysis in [Hardt et al., 2016] can be further improved to reveal the full potential of the stability method, either in general or for specific data-distributions.

Original: Our results: We provide three kinds of results (see Table 1) that complement each other: a) tight lower bounds that show settings where stability analysis cannot be improved further for general datasets, b) weaker lower bounds that hint at a possible improvement, along with complementary improved upper bounds, also for general datasets and c) in settings where existing data-independent analysis cannot be improved, we derive improved data-dependent bounds.
Revised: Our results: We provide three kinds of results (see Table 1) that complement each other: a) tight lower bounds that show settings where stability analysis cannot be improved further for general datasets, b) weaker lower bounds that hint at a possible improvement, along with complementary improved upper bounds, also for general datasets and c) in settings where existing data-independent analysis cannot be improved, we derive improved data-dependent bounds.

Original: Below we summarize some of the existing open questions in this line of research, grouped according to properties of the loss function, along with our results addressing these problems.
Revised: Below we summarize some of the existing open questions in this line of research, grouped according to properties of the loss function, along with our results addressing these problems.

Original: 1.1 CONVEX AND STRONGLY CONVEX LOSS
Revised: 1.1 CONVEX AND STRONGLY CONVEX LOSS

Original: The following are the main results presented in [Hardt et al., 2016] for convex and strongly-convex loss functions (with certain Lipschitz and smoothness conditions), when optimized using SGD.
Revised: The following are the main results presented in [Hardt et al., 2016] for convex and strongly-convex loss functions (with certain Lipschitz and smoothness conditions), when optimized using SGD.

Original: Here n denotes the size of the sample,
Revised: Here n denotes the size of the sample,

Original: T the number of steps in SGD, and α t the size of the SGD step in the t -th iteration.
Revised: T the number of steps in SGD, and α t the size of the SGD step in the t -th iteration.

Original: For convex loss functions, the stability is upper bounded by (cid:80)
Revised: For convex loss functions, the stability is upper bounded by (cid:80)

Original: Ti =1 α t
Revised: Ti =1 α t

Original: /n .
Revised: /n .

Original: The smaller the number of iterations T is, the lower this upper bound.
Revised: The smaller the number of iterations T is, the lower this upper bound.

Original: Hence “train faster, generalize better".
Revised: Hence “train faster, generalize better".

Original: In practice
Revised: In practice

Original: , one
Revised: , one

Original: often
Revised: often

Original: uses constant step size: α t = α .
Revised: uses constant step size: α t = α .

Original: For convex loss functions the upper bound would then scale linearly in the number of iterations T , which seems to be too pessimistic.
Revised: For convex loss functions the upper bound would then scale linearly in the number of iterations T , which seems to be too pessimistic.

Original: [Hardt et al., 2016] show that by adding a µ 2 || w || 22 regularization term to the convex loss function, where w is the vector of weights and µ ∈ Θ(1) is a small constant, one gets much better stability upper bound for constant step size that does not depend on T , and is O (1 /n ) .
Revised: [Hardt et al., 2016] show that by adding a µ 2 || w || 22 regularization term to the convex loss function, where w is the vector of weights and µ ∈ Θ(1) is a small constant, one gets much better stability upper bound for constant step size that does not depend on T , and is O (1 /n ) .

Original: This gives rise to the following questions:
Revised: This gives rise to the following questions:

Original: Question 1: Are the upper bounds of [Hardt et al., 2016] for convex and strongly-convex functions tight?
Revised: Question 1: Are the upper bounds of [Hardt et al., 2016] for convex and strongly-convex functions tight?

Original: That is, can one construct loss functions that satisfy the hypotheses and exhibit the claimed worst-case stability performance?
Revised: That is, can one construct loss functions that satisfy the hypotheses and exhibit the claimed worst-case stability performance?

Original: We remark that, to the best of our knowledge, the only construction available in the literature is [Bassily et al., 2020].
Revised: We remark that, to the best of our knowledge, the only construction available in the literature is [Bassily et al., 2020].

Original: The authors analyze the stability of a loss function in order to derive lower bounds, but unfortunately, the loss function is not smooth and therefore does not satisfy the hypothesis in [Hardt et al., 2016].
Revised: The authors analyze the stability of a loss function in order to derive lower bounds, but unfortunately, the loss function is not smooth and therefore does not satisfy the hypothesis in [Hardt et al., 2016].

Original: Question 2: How important is the regularization term in order to make the transition from convex to strongly-convex, and therefore the improvement from an O ( T/n ) upper bound to an O (1 /n ) upper bound for constant step-size
Revised: Question 2: How important is the regularization term in order to make the transition from convex to strongly-convex, and therefore the improvement from an O ( T/n ) upper bound to an O (1 /n ) upper bound for constant step-size

Original: SGD?
Revised: SGD?

Original: We provide the following answers to the above questions:
Revised: We provide the following answers to the above questions:

Original: Result 1: The answer to question 1 is yes, i.e., there exist smooth, convex and strongly-convex loss functions that achieve the worst-case stability upper bound,
Revised: Result 1: The answer to question 1 is yes, i.e., there exist smooth, convex and strongly-convex loss functions that achieve the worst-case stability upper bound,

Original: In Theorem 1, we construct a Huber function which is quadratic in a certain area and linear outside.
Revised: In Theorem 1, we construct a Huber function which is quadratic in a certain area and linear outside.

Original: Under certain restricted assumptions, we proved the tightness of upper bounds in [Hardt et al., 2016] for convex loss which strengthens the lower bound of [Bassily et al., 2020] for the non-smooth case.
Revised: Under certain restricted assumptions, we proved the tightness of upper bounds in [Hardt et al., 2016] for convex loss which strengthens the lower bound of [Bassily et al., 2020] for the non-smooth case.

Original: In Theorem 2, our construction shows the tightness of upper bounds in [Hardt et al., 2016] for strongly convex loss.
Revised: In Theorem 2, our construction shows the tightness of upper bounds in [Hardt et al., 2016] for strongly convex loss.

Original: Result 2: (Data-dependent bounds)
Revised: Result 2: (Data-dependent bounds)

Original: We answer question 2 by introducing Theorem 3.
Revised: We answer question 2 by introducing Theorem 3.

Original: In Theorem 3, we derive an upper bound on the stability for linear model loss function that is independent of T (the number of iterations), even when the weight µ of the regularization term is very small (of the order of 1 /n 4 ), as long as the data satisfies a natural condition related to the Second Moment.
Revised: In Theorem 3, we derive an upper bound on the stability for linear model loss function that is independent of T (the number of iterations), even when the weight µ of the regularization term is very small (of the order of 1 /n 4 ), as long as the data satisfies a natural condition related to the Second Moment.

Original: Sharing a similar spirit with [Kuzborskij and Lampert, 2018], our result suggests that the property of distribution plays an important role in generalization of SGD, and nice properties of the data can almost replace the need for regularization.
Revised: Sharing a similar spirit with [Kuzborskij and Lampert, 2018], our result suggests that the property of distribution plays an important role in generalization of SGD, and nice properties of the data can almost replace the need for regularization.

Original: 1.2 NON-CONVEX LOSS
Revised: 1.2 NON-CONVEX LOSS

Original: [Hardt et al., 2016] also prove an upper bound for nonconvex loss functions, and one wonders again whether the bound is tight.
Revised: [Hardt et al., 2016] also prove an upper bound for nonconvex loss functions, and one wonders again whether the bound is tight.

Original: After only being able to prove a slightly weaker lower bound, we realized that this was because one can actually improve the analysis in [Hardt et al., 2016]!
Revised: After only being able to prove a slightly weaker lower bound, we realized that this was because one can actually improve the analysis in [Hardt et al., 2016]!

Original: Result 3: We provide matching lower ( Theorem 4) and upper bounds ( Theorem 5) on the stability of SGD for non-convex functions, that are tighter than the upper bound in [Hardt et al., 2016] for a wide and interesting range of values of T (e.g., when n < T < n 10 ).
Revised: Result 3: We provide matching lower ( Theorem 4) and upper bounds ( Theorem 5) on the stability of SGD for non-convex functions, that are tighter than the upper bound in [Hardt et al., 2016] for a wide and interesting range of values of T (e.g., when n < T < n 10 ).

Original: In the non-convex setting, the bounds in both [Hardt et al., 2016] and our Result 3 assume a decreasing step-size α t ∝ 1 /t in SGD.
Revised: In the non-convex setting, the bounds in both [Hardt et al., 2016] and our Result 3 assume a decreasing step-size α t ∝ 1 /t in SGD.

Original: However, in practice the constant step-size case is very important.
Revised: However, in practice the constant step-size case is very important.

Original: Although it is not derived formally, the techniques in [Hardt et al., 2016] can be employed to show an exponential upper bound for non-convex loss functions minimized using SGD with constant-size step, raising the question of the existence of better analysis.
Revised: Although it is not derived formally, the techniques in [Hardt et al., 2016] can be employed to show an exponential upper bound for non-convex loss functions minimized using SGD with constant-size step, raising the question of the existence of better analysis.

Original: Result 4: Also by Theorem 7, we show that without any additional assumptions on either the loss function or the data distribution, improving on this analysis is hopeless by providing a lower bound that is exponential in T .
Revised: Result 4: Also by Theorem 7, we show that without any additional assumptions on either the loss function or the data distribution, improving on this analysis is hopeless by providing a lower bound that is exponential in T .

Original: Data-dependent bounds:
Revised: Data-dependent bounds:

Original: This naturally raises the question
Revised: This naturally raises the question

Original: [Table]
Revised: [Table]

Original: of deriving data-dependent bounds on stability in the nonconvex setting.
Revised: of deriving data-dependent bounds on stability in the nonconvex setting.

Original: The work in [Kuzborskij and Lampert, 2018] took the first step in this direction by analyzing SGD using concept of “average stability” from [Bousquet and Elisseeff, 2002, Shalev-Shwartz et al., 2010], and deriving upper bounds on it.
Revised: The work in [Kuzborskij and Lampert, 2018] took the first step in this direction by analyzing SGD using concept of “average stability” from [Bousquet and Elisseeff, 2002, Shalev-Shwartz et al., 2010], and deriving upper bounds on it.

Original: Finally, we show:
Revised: Finally, we show:

Original: Result 5: The improved analysis for uniform stability of
Revised: Result 5: The improved analysis for uniform stability of

Original: SGD on non-convex and smooth loss functions can also be applied to improve on the result in [Kuzborskij and Lampert, 2018] and obtain a tighter bound for the average stability of
Revised: SGD on non-convex and smooth loss functions can also be applied to improve on the result in [Kuzborskij and Lampert, 2018] and obtain a tighter bound for the average stability of

Original: SGD.
Revised: SGD.

Original: We present Theorem 6 as the data-dependent version of Theorem 5.
Revised: We present Theorem 6 as the data-dependent version of Theorem 5.

Original: In summary, we essentially close the open questions of tightness in data-independent settings for all three classes of functions, and improve upper bounds in the data-dependent setting.
Revised: In summary, we essentially close the open questions of tightness in data-independent settings for all three classes of functions, and improve upper bounds in the data-dependent setting.

Original: We hope that our results will initiate further efforts to better understand the data-dependent setting under non-convex loss functions and analyze the conditions under which one can expect better upper bounds on stability and generalization of SGD.
Revised: We hope that our results will initiate further efforts to better understand the data-dependent setting under non-convex loss functions and analyze the conditions under which one can expect better upper bounds on stability and generalization of SGD.

Original:  RELATED WORKS
Revised:  RELATED WORKS

Original: The stability framework suggests that a stable machine learning algorithm results in models with good generalization performance [Kearns and Ron, 1999, Bousquet and Elisseeff, 2002, Elisseeff et al., 2005, Shalev-Shwartz et al., 2010, Devroye and Wagner, 1979a,b, Rogers and Wagner, 1978, Bousquet and Elisseeff, 2002].
Revised: The stability framework suggests that a stable machine learning algorithm results in models with good generalization performance [Kearns and Ron, 1999, Bousquet and Elisseeff, 2002, Elisseeff et al., 2005, Shalev-Shwartz et al., 2010, Devroye and Wagner, 1979a,b, Rogers and Wagner, 1978, Bousquet and Elisseeff, 2002].

Original: It serves as a mechanism for provable learnability when uniform convergence fails [Shalev-Shwartz et al., 2010, Nagarajan and Kolter, 2019].
Revised: It serves as a mechanism for provable learnability when uniform convergence fails [Shalev-Shwartz et al., 2010, Nagarajan and Kolter, 2019].

Original: The concept of uniform stability was introduced in order to derive high probability bounds on the generalization error [Bousquet and Elisseeff, 2002].
Revised: The concept of uniform stability was introduced in order to derive high probability bounds on the generalization error [Bousquet and Elisseeff, 2002].

Original: Uniform stability describes the worst-case change in the loss of a model trained on an algorithm when a single data point in the dataset is replaced.
Revised: Uniform stability describes the worst-case change in the loss of a model trained on an algorithm when a single data point in the dataset is replaced.

Original: In [Hardt et al., 2016], a uniform stability analysis for iterative algorithms is proposed to analyze SGD, generalizing the one-shot version in [Bousquet and Elisseeff, 2002].
Revised: In [Hardt et al., 2016], a uniform stability analysis for iterative algorithms is proposed to analyze SGD, generalizing the one-shot version in [Bousquet and Elisseeff, 2002].

Original: Algorithmic uniform stability is widely used in analyzing the generalization performance of SGD [Mou et al., 2018,
Revised: Algorithmic uniform stability is widely used in analyzing the generalization performance of SGD [Mou et al., 2018,

Original: Feldman and Vondrak, 2019, Chen et al., 2018].
Revised: Feldman and Vondrak, 2019, Chen et al., 2018].

Original: The worstcase leave-one-out type bounds also closely connect uniform stability with differential private learning [Feldman et al., 2018, 2020, Dwork et al., 2006, Wu et al., 2017], where the uniform stability can lead to provable privacy guarantees.
Revised: The worstcase leave-one-out type bounds also closely connect uniform stability with differential private learning [Feldman et al., 2018, 2020, Dwork et al., 2006, Wu et al., 2017], where the uniform stability can lead to provable privacy guarantees.

Original: Beside uniform stability, [Liu et al., 2017] proposed argument stability to capture stability of selected hypothesis function space.
Revised: Beside uniform stability, [Liu et al., 2017] proposed argument stability to capture stability of selected hypothesis function space.

Original: While the upper bounds of algorithmic stability of SGD have been extensively studied, the tightness of those bounds remains open.
Revised: While the upper bounds of algorithmic stability of SGD have been extensively studied, the tightness of those bounds remains open.

Original: In addition to uniform stability, an average stability of the SGD is studied in [Kuzborskij and Lampert, 2018] where the authors provide data-dependent upper bounds on stability 1 .
Revised: In addition to uniform stability, an average stability of the SGD is studied in [Kuzborskij and Lampert, 2018] where the authors provide data-dependent upper bounds on stability 1 .

Original: Our analysis framework for deriving improved bounds in [Hardt et al., 2016] can also be applied to improve the data-dependent stability results in [Kuzborskij and Lampert, 2018].
Revised: Our analysis framework for deriving improved bounds in [Hardt et al., 2016] can also be applied to improve the data-dependent stability results in [Kuzborskij and Lampert, 2018].

Original: In [Bassily et al., 2020], a lower bound on the stability of
Revised: In [Bassily et al., 2020], a lower bound on the stability of

Original: SGD for nonsmooth convex losses is proposed.
Revised: SGD for nonsmooth convex losses is proposed.

Original: The lower bound is designed to illustrate the tightness of the stability analysis without smoothness assumptions.
Revised: The lower bound is designed to illustrate the tightness of the stability analysis without smoothness assumptions.

Original: In this work, we report for the first time lower bounds on the uniform stability of SGD for smooth loss functions.
Revised: In this work, we report for the first time lower bounds on the uniform stability of SGD for smooth loss functions.

Original: Our tightness analysis suggests the necessity of additional assumptions for analyzing the generalization of SGD for deep learning.
Revised: Our tightness analysis suggests the necessity of additional assumptions for analyzing the generalization of SGD for deep learning.

Original:  PRELIMINARIES
Revised:  PRELIMINARIES

Original: In this section we introduce the notion of uniform stability and establish notation.
Revised: In this section we introduce the notion of uniform stability and establish notation.

Original: We first introduce the quantities empirical risk , population risk , and generalization gap .
Revised: We first introduce the quantities empirical risk , population risk , and generalization gap .

Original: Given an unknown distribution D on labeled sample space
Revised: Given an unknown distribution D on labeled sample space

Original: Z =
Revised: Z =

Original: X × R , let S
Revised: X × R , let S

Original: =
Revised: =

Original: { z 1 , ..., z n } denote a set of n samples z i =
Revised: { z 1 , ..., z n } denote a set of n samples z i =

Original: ( x i , y i ) drawn i.i.d.
Revised: ( x i , y i ) drawn i.i.d.

Original: from D .
Revised: from D .

Original: Let w ∈ R
Revised: Let w ∈ R

Original: d be the parameter(s) of a model that predicts y given x , and let f be a loss function where f ( w ; z ) denotes the loss of the model with parameter(s) w on sample z .
Revised: d be the parameter(s) of a model that predicts y given x , and let f be a loss function where f ( w ; z ) denotes the loss of the model with parameter(s) w on sample z .

Original: Let f ( w ; S ) denote the empirical risk f ( w ; S )
Revised: Let f ( w ; S ) denote the empirical risk f ( w ; S )

Original: =
Revised: =

Original: E z ∼ S
Revised: E z ∼ S

Original: [ f ( w ; z )]
Revised: [ f ( w ; z )]

Original: = 1 n (cid:80) ni
Revised: = 1 n (cid:80) ni

Original: =1 f ( w ; z i ) with corresponding population risk E z
Revised: =1 f ( w ; z i ) with corresponding population risk E z

Original: ∼D
Revised: ∼D

Original: [ f ( w ; z )] .
Revised: [ f ( w ; z )] .

Original: The generalization error of the model with parameter(s) w is defined as the difference between the empirical and population risks: | E z ∼D [ f
Revised: The generalization error of the model with parameter(s) w is defined as the difference between the empirical and population risks: | E z ∼D [ f

Original: [Equation]
Revised: [Equation]

Original: w ; z )
Revised: w ; z )

Original: [Equation]
Revised: [Equation]

Original: E z ∼ S
Revised: E z ∼ S

Original: [ f ( w ; z )] | .
Revised: [ f ( w ; z )] | .

Original: Next we introduce stochastic gradient descent (SGD).
Revised: Next we introduce stochastic gradient descent (SGD).

Original: We follow the setting of [Hardt et al., 2016]: starting with initialization w 0 ∈ R d , an SGD update step takes the form
Revised: We follow the setting of [Hardt et al., 2016]: starting with initialization w 0 ∈ R d , an SGD update step takes the form

Original: [Equation]
Revised: [Equation]

Original: where i t is drawn from [ n ] = { 1 , 2 , · · · , n } uniformly and independently in each round.
Revised: where i t is drawn from [ n ] = { 1 , 2 , · · · , n } uniformly and independently in each round.

Original: Let W be a convex and compact set to be optimized over.
Revised: Let W be a convex and compact set to be optimized over.

Original: For projected SGD we let
Revised: For projected SGD we let

Original: [Equation]
Revised: [Equation]

Original: where Π W ( v ) =
Revised: where Π W ( v ) =

Original: arg min w ∈W ∥
Revised: arg min w ∈W ∥

Original: w − v ∥ .
Revised: w − v ∥ .

Original: The analysis of SGD requires the following crucial properties of the loss function f ( · , z ) at any fixed point z , viewed solely as a function of the parameter(s) w :
Revised: The analysis of SGD requires the following crucial properties of the loss function f ( · , z ) at any fixed point z , viewed solely as a function of the parameter(s) w :

Original: Definition 1 ( L -Lipschitz) .
Revised: Definition 1 ( L -Lipschitz) .

Original: A function f ( w ) is L -Lipschitz if ∀ u, v ∈ R d : | f ( u ) − f ( v ) | ≤ L ∥ u − v ∥ .
Revised: A function f ( w ) is L -Lipschitz if ∀ u, v ∈ R d : | f ( u ) − f ( v ) | ≤ L ∥ u − v ∥ .

Original: Definition 2 ( β -smooth) .
Revised: Definition 2 ( β -smooth) .

Original: A function f ( w ) is β -smooth if ∀ u, v ∈ R d : |∇ f ( u ) − ∇ f ( v ) |
Revised: A function f ( w ) is β -smooth if ∀ u, v ∈ R d : |∇ f ( u ) − ∇ f ( v ) |

Original: ≤ β ∥
Revised: ≤ β ∥

Original: u − v ∥ .
Revised: u − v ∥ .

Original: Definition 3 ( γ -strongly-convex) .
Revised: Definition 3 ( γ -strongly-convex) .

Original: A function f ( w ) is γstrongly-convex if ∀ u, v ∈ R d : f ( u ) >
Revised: A function f ( w ) is γstrongly-convex if ∀ u, v ∈ R d : f ( u ) >

Original: f ( v
Revised: f ( v

Original: [Equation]
Revised: [Equation]

Original: Definition 4 ( ρ -Lipschitz Hessian) .
Revised: Definition 4 ( ρ -Lipschitz Hessian) .

Original: A loss function f has a ρ -Lispchitz Hessian if ∀ u, v ∈ R d , ∥∇ 2 f ( u ) − ∇ 2 f ( v ) ∥ ≤ ρ ∥ u − v ∥ .
Revised: A loss function f has a ρ -Lispchitz Hessian if ∀ u, v ∈ R d , ∥∇ 2 f ( u ) − ∇ 2 f ( v ) ∥ ≤ ρ ∥ u − v ∥ .

Original: Algorithmic Stability: Next we define the key concept of algorithmic stability , which was introduced by [Bousquet and Elisseeff, 2002] and adopted by [Hardt et al., 2016].
Revised: Algorithmic Stability: Next we define the key concept of algorithmic stability , which was introduced by [Bousquet and Elisseeff, 2002] and adopted by [Hardt et al., 2016].

Original: Informally, an algorithm is stable if its output only varies slightly when we change a single sample in the input dataset.
Revised: Informally, an algorithm is stable if its output only varies slightly when we change a single sample in the input dataset.

Original: When this stability is uniform over all datasets differing at a single point, this leads to an upper bound on the generalization gap.
Revised: When this stability is uniform over all datasets differing at a single point, this leads to an upper bound on the generalization gap.

Original: We now flesh this out more formally.
Revised: We now flesh this out more formally.

Original: Definition 5.
Revised: Definition 5.

Original: Two sets of samples S, S ′ are twin datasets if they differ at a single entry, i.e., S = { z 1 , ...
Revised: Two sets of samples S, S ′ are twin datasets if they differ at a single entry, i.e., S = { z 1 , ...

Original: z i , ..., z n } and
Revised: z i , ..., z n } and

Original: S ′
Revised: S ′

Original: = { z 1 , ..., z ′ i , ..., z n } .
Revised: = { z 1 , ..., z ′ i , ..., z n } .

Original: Now, let A be a (possibly randomized) algorithm which is parameterized by a sample S of n datapoints as A ( S ) .
Revised: Now, let A be a (possibly randomized) algorithm which is parameterized by a sample S of n datapoints as A ( S ) .

Original: Definition 6.
Revised: Definition 6.

Original: (Stability)
Revised: (Stability)

Original: Define the algorithmic stability parameter ε stab ( A , n ) as inf
Revised: Define the algorithmic stability parameter ε stab ( A , n ) as inf

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The expectation E A factors in the possible randomness of
Revised: The expectation E A factors in the possible randomness of

Original: A .
Revised: A .

Original: For such an algorithm, one can define its expected generalization error as
Revised: For such an algorithm, one can define its expected generalization error as

Original: [Equation]
Revised: [Equation]

Original: We also define a data-dependent stability which is an average stability that was introduced by [Rakhlin et al., 2005,
Revised: We also define a data-dependent stability which is an average stability that was introduced by [Rakhlin et al., 2005,

Original: Shalev-Shwartz et al., 2010] and was applied for analyzing algorithmic stability of SGD by [Kuzborskij and Lampert, 2018].
Revised: Shalev-Shwartz et al., 2010] and was applied for analyzing algorithmic stability of SGD by [Kuzborskij and Lampert, 2018].

Original: Definition 7 (On-average stability) .
Revised: Definition 7 (On-average stability) .

Original: Let D be the data distribution and w 0 be the initialized weight.
Revised: Let D be the data distribution and w 0 be the initialized weight.

Original: A randomized algorithm A is (cid:98) ε stab
Revised: A randomized algorithm A is (cid:98) ε stab

Original: ( D , w 0 ) -on-average stable if
Revised: ( D , w 0 ) -on-average stable if

Original: [Equation]
Revised: [Equation]

Original: where S iid ∼ D m and S ′ is its copy with i -th example replaced by z iid ∼ D .
Revised: where S iid ∼ D m and S ′ is its copy with i -th example replaced by z iid ∼ D .

Original: Throughout this paper, we will write ε stab and (cid:98) ε stab omitting dependencies that are clear in context.
Revised: Throughout this paper, we will write ε stab and (cid:98) ε stab omitting dependencies that are clear in context.

Original: Stability and generalization: It was proved in [Hardt et al., 2016]
Revised: Stability and generalization: It was proved in [Hardt et al., 2016]

Original: that GE ( A , n ) ≤ ε stab ( A , n ) .
Revised: that GE ( A , n ) ≤ ε stab ( A , n ) .

Original: Furthermore, the authors observed that an L -Lipschitz condition on the loss function f enforces a uniform upper bound: sup z ∈ Z | f ( w ; z ) − f ( w ′ ; z ) | ≤ L ∥ w − w ′ ∥ .
Revised: Furthermore, the authors observed that an L -Lipschitz condition on the loss function f enforces a uniform upper bound: sup z ∈ Z | f ( w ; z ) − f ( w ′ ; z ) | ≤ L ∥ w − w ′ ∥ .

Original: This implies that for a Lipschitz loss, the algorithmic stability ε stab ( A , n ) (and hence the generalization error GE ( A , n ) ) can be bounded by obtaining bounds on ∥ w − w ′ ∥ .
Revised: This implies that for a Lipschitz loss, the algorithmic stability ε stab ( A , n ) (and hence the generalization error GE ( A , n ) ) can be bounded by obtaining bounds on ∥ w − w ′ ∥ .

Original: And in [Kuzborskij and Lampert, 2018] they have similar results in the notion of on-average stability.
Revised: And in [Kuzborskij and Lampert, 2018] they have similar results in the notion of on-average stability.

Original: Let w t and w ′ t be the parameters obtained by running SGD on twin datasets S, S ′ respectively for t iterations.
Revised: Let w t and w ′ t be the parameters obtained by running SGD on twin datasets S, S ′ respectively for t iterations.

Original: The divergence quantity is defined as δ t :=
Revised: The divergence quantity is defined as δ t :=

Original: E A || w t − w ′ t
Revised: E A || w t − w ′ t

Original: || .
Revised: || .

Original: While [Hardt et al., 2016] reports upper bounds on δ t for different loss functions, e.g., convex and non-convex loss functions, we investigate the tightness of those bounds.
Revised: While [Hardt et al., 2016] reports upper bounds on δ t for different loss functions, e.g., convex and non-convex loss functions, we investigate the tightness of those bounds.

Original:  MAIN RESULTS
Revised:  MAIN RESULTS

Original: In this section, we report our main results.
Revised: In this section, we report our main results.

Original: We first consider the convex case with constant step size, where we prove 1) that the existing bounds in [Hardt et al., 2016] are tight, and 2) for linear models, we report a data-dependent analysis to show that ε stab does not increase with t .
Revised: We first consider the convex case with constant step size, where we prove 1) that the existing bounds in [Hardt et al., 2016] are tight, and 2) for linear models, we report a data-dependent analysis to show that ε stab does not increase with t .

Original: Then we move on to the non-convex case, where a) for decreasing step size we report a lower bound suggests that within a wide range of T , existing bound in [Hardt et al., 2016] is not tight.
Revised: Then we move on to the non-convex case, where a) for decreasing step size we report a lower bound suggests that within a wide range of T , existing bound in [Hardt et al., 2016] is not tight.

Original: We prove a tighter upper bound which matches our lower bound thus, and b) for constant step size we give loss functions whose divergence δ t increases exponentially with t .
Revised: We prove a tighter upper bound which matches our lower bound thus, and b) for constant step size we give loss functions whose divergence δ t increases exponentially with t .

Original: 4.1 CONVEX CASE
Revised: 4.1 CONVEX CASE

Original: In this section we analyze the stability of SGD when the loss function is convex and smooth.
Revised: In this section we analyze the stability of SGD when the loss function is convex and smooth.

Original: We begin with a construction which shows that Theorem 3.8 in [Hardt et al., 2016] is tight.
Revised: We begin with a construction which shows that Theorem 3.8 in [Hardt et al., 2016] is tight.

Original: Our lower bound analysis will require the quadratic function
Revised: Our lower bound analysis will require the quadratic function

Original: [Equation]
Revised: [Equation]

Original: where A is a d × d matrix.
Revised: where A is a d × d matrix.

Original: In the construction of lower bounds, we carefully choose A and S so that the single data point replaced in the twin data set will cause the instability of SGD.
Revised: In the construction of lower bounds, we carefully choose A and S so that the single data point replaced in the twin data set will cause the instability of SGD.

Original: In particular, we will choose A to be a PSD matrix in the convex case in the construction of the lower bound and choose A to be an indefinite matrix with some strictly negative eigenvalues in the non-convex case.
Revised: In particular, we will choose A to be a PSD matrix in the convex case in the construction of the lower bound and choose A to be an indefinite matrix with some strictly negative eigenvalues in the non-convex case.

Original: We first begin with the following lemma which describes how ∥ w t − w ′ t
Revised: We first begin with the following lemma which describes how ∥ w t − w ′ t

Original: ∥ behaves for functions defined in Equation 1.
Revised: ∥ behaves for functions defined in Equation 1.

Original: Lemma 1 (Dynamics of divergence) .
Revised: Lemma 1 (Dynamics of divergence) .

Original: Let f ( w ; x ) = 12 w ⊤
Revised: Let f ( w ; x ) = 12 w ⊤

Original: Aw − yx .
Revised: Aw − yx .

Original: Suppose [ x i − x ′ i ] / ∥ x
Revised: Suppose [ x i − x ′ i ] / ∥ x

Original: i − x ′ i ∥ is an eigenvector of A , i.e., A
Revised: i − x ′ i ∥ is an eigenvector of A , i.e., A

Original: [ x i − x ′ i ]
Revised: [ x i − x ′ i ]

Original: = λ xx ′
Revised: = λ xx ′

Original: [ x i − x ′ i ] .
Revised: [ x i − x ′ i ] .

Original: Let ∆ t be w t − w ′ t , α t ≤ λ xx ′ be the step size of SGD and ∆ 

Revised: Let ∆ t be w t − w ′ t , α t ≤ λ xx ′ be the step size of SGD and ∆

Original: = 0 .
Revised: = 0 .

Original: If one runs SGD on f ( w, S ) and f ( w, S ′ ) where S, S ′ are twin datasets and x ′⊤ i x j = 0 , x ⊤ i x j = 0 , ∀ j ̸
Revised: If one runs SGD on f ( w, S ) and f ( w, S ′ ) where S, S ′ are twin datasets and x ′⊤ i x j = 0 , x ⊤ i x j = 0 , ∀ j ̸

Original: = i , then the dynamics of ∆ t are given by
Revised: = i , then the dynamics of ∆ t are given by

Original: [Equation]
Revised: [Equation]

Original: Remark 1.
Revised: Remark 1.

Original: In this work, we assume that the different entry data x i , x ′ i are orthogonal to all other samples.
Revised: In this work, we assume that the different entry data x i , x ′ i are orthogonal to all other samples.

Original: Such a restrictive setting serves as a corner case to prove the tightness, suggesting the necessity of additional assumptions to improve the upper bound.
Revised: Such a restrictive setting serves as a corner case to prove the tightness, suggesting the necessity of additional assumptions to improve the upper bound.

Original: Indeed in Theorem 5, we introduce more realistic assumptions to avoid such corner cases, and the upper bound can be improved accordingly.
Revised: Indeed in Theorem 5, we introduce more realistic assumptions to avoid such corner cases, and the upper bound can be improved accordingly.

Original: The next lemma recursively applies Lemma 1.
Revised: The next lemma recursively applies Lemma 1.

Original: We will carefully chose λ xx ′ in the following lemma for lower bound constructions in the convex and non-convex cases.
Revised: We will carefully chose λ xx ′ in the following lemma for lower bound constructions in the convex and non-convex cases.

Original: Lemma 2 (Lower bound on divergence) .
Revised: Lemma 2 (Lower bound on divergence) .

Original: Let f ( w ; x ) = 12 w ⊤
Revised: Let f ( w ; x ) = 12 w ⊤

Original: Aw − yx .
Revised: Aw − yx .

Original: Suppose [ x i − x ′ i ] / ∥ x
Revised: Suppose [ x i − x ′ i ] / ∥ x

Original: i − x ′ i ∥ is an eigenvector of A
Revised: i − x ′ i ∥ is an eigenvector of A

Original: where
Revised: where

Original: A
Revised: A

Original: [ x i − x ′ i ]
Revised: [ x i − x ′ i ]

Original: = λ xx ′
Revised: = λ xx ′

Original: [ x i − x ′ i ] .
Revised: [ x i − x ′ i ] .

Original: Let ∆ t be w t − w ′ t , α t ≤ λ xx ′ be the step size of SGD and ∆ 

Revised: Let ∆ t be w t − w ′ t , α t ≤ λ xx ′ be the step size of SGD and ∆

Original: = 0 .
Revised: = 0 .

Original: If one runs SGD on f ( w, S ) and f ( w, S ′ ) where S, S ′ are twin datasets and x ′⊤ i x j = 0 , x ⊤ i x j = 0 , ∀ j ̸
Revised: If one runs SGD on f ( w, S ) and f ( w, S ′ ) where S, S ′ are twin datasets and x ′⊤ i x j = 0 , x ⊤ i x j = 0 , ∀ j ̸

Original: = i , then we have
Revised: = i , then we have

Original: [Equation]
Revised: [Equation]

Original: Now we can present our tightness results.
Revised: Now we can present our tightness results.

Original: We begin with the convex case.
Revised: We begin with the convex case.

Original: The main idea of the construction is to leverage Equation 1 with specially designed A and S, S ′ to ensure that E A ∥
Revised: The main idea of the construction is to leverage Equation 1 with specially designed A and S, S ′ to ensure that E A ∥

Original: w T − w ′ T ∥ will diverge.
Revised: w T − w ′ T ∥ will diverge.

Original: However, quadratic function in general does not L -Lipschitz condition, which does not match the assumpition used to derive upper bound in Hardt et al.
Revised: However, quadratic function in general does not L -Lipschitz condition, which does not match the assumpition used to derive upper bound in Hardt et al.

Original: To obtain the L -Lipschitz condition, we trim f ( w ; S ) to mimic the Huber loss function [Huber, 1992] so that the smoothness is maintained for the piecewise function.
Revised: To obtain the L -Lipschitz condition, we trim f ( w ; S ) to mimic the Huber loss function [Huber, 1992] so that the smoothness is maintained for the piecewise function.

Original: Theorem 1 (Lower bound for convex losses) .
Revised: Theorem 1 (Lower bound for convex losses) .

Original: Let w t , w ′ t be the outputs of SGD on twin datasets S, S ′ respectively.
Revised: Let w t , w ′ t be the outputs of SGD on twin datasets S, S ′ respectively.

Original: Let ∆ t = w t − w ′ t
Revised: Let ∆ t = w t − w ′ t

Original: and α t be the step size of SGD.
Revised: and α t be the step size of SGD.

Original: There exists a function f which is convex, β -smooth, and L -Lipschitz, and twin datasets S, S ′ such that
Revised: There exists a function f which is convex, β -smooth, and L -Lipschitz, and twin datasets S, S ′ such that

Original: [Equation]
Revised: [Equation]

Original: The convex upper bound in Theorem 3.8 of [Hardt et al., 2016] states that E A ∥ ∆ T ∥ ≤ (cid:80)
Revised: The convex upper bound in Theorem 3.8 of [Hardt et al., 2016] states that E A ∥ ∆ T ∥ ≤ (cid:80)

Original: Ti =1 α t Ln , which implies that the divergence increases throughout training.
Revised: Ti =1 α t Ln , which implies that the divergence increases throughout training.

Original: The lower bound in Theorem 1 suggests the tightness of the upper bound.
Revised: The lower bound in Theorem 1 suggests the tightness of the upper bound.

Original: However, in practice, this is not commonly observed; the generalization performance does not deteriorate as the number of training iterations increases.
Revised: However, in practice, this is not commonly observed; the generalization performance does not deteriorate as the number of training iterations increases.

Original: Under the γstrongly-convex loss function condition, [Hardt et al., 2016] provides an O ( 1 n ) uniform stability bound, which fits better with empirical observations on classical convex losses.
Revised: Under the γstrongly-convex loss function condition, [Hardt et al., 2016] provides an O ( 1 n ) uniform stability bound, which fits better with empirical observations on classical convex losses.

Original: In the next theorem, we show the tightness of the O ( 1 n ) bound for strongly-convex losses.
Revised: In the next theorem, we show the tightness of the O ( 1 n ) bound for strongly-convex losses.

Original: Theorem 2 (Lower bound for strongly-convex losses) .
Revised: Theorem 2 (Lower bound for strongly-convex losses) .

Original: Let w t , w ′ t be the outputs of SGD on twin datasets S, S ′ respectively, ∆ t be w t − w ′ t and α = 12 β be the step size of SGD.
Revised: Let w t , w ′ t be the outputs of SGD on twin datasets S, S ′ respectively, ∆ t be w t − w ′ t and α = 12 β be the step size of SGD.

Original: There exists a function f which is γ -strongly-convex and β -smooth, and twin datasets S, S ′ such that the divergence and stability of the two SGD outputs satisfies
Revised: There exists a function f which is γ -strongly-convex and β -smooth, and twin datasets S, S ′ such that the divergence and stability of the two SGD outputs satisfies

Original: [Equation]
Revised: [Equation]

Original: Theorem 2 provides evidence for the tightness of the O ( 1 n ) stability bound on SGD.
Revised: Theorem 2 provides evidence for the tightness of the O ( 1 n ) stability bound on SGD.

Original: To obtain such stability, the loss function must satisfy ∇ 2 w f ( w ; z ) > γI d with γ = Ω(1) .
Revised: To obtain such stability, the loss function must satisfy ∇ 2 w f ( w ; z ) > γI d with γ = Ω(1) .

Original: In general this does not hold, e.g., the Hessian of an individual linear regression loss term is x j x ⊤ j which is not stronglyconvex.
Revised: In general this does not hold, e.g., the Hessian of an individual linear regression loss term is x j x ⊤ j which is not stronglyconvex.

Original: In practice one can incorporate a strongly-convex regularizer to impose strong convexity, often resulting in improved generalization performance in practice [ShalevShwartz et al., 2010, Bousquet and Elisseeff, 2002].
Revised: In practice one can incorporate a strongly-convex regularizer to impose strong convexity, often resulting in improved generalization performance in practice [ShalevShwartz et al., 2010, Bousquet and Elisseeff, 2002].

Original: However, an O (1) regularization term will bias the loss function away from achieving sufficiently low empirical risk.
Revised: However, an O (1) regularization term will bias the loss function away from achieving sufficiently low empirical risk.

Original: This motivates us to investigate a weaker condition than strong convexity which still can enforce an O (cid:0) 1 n (cid:1) stability, without substantially biasing the loss function.
Revised: This motivates us to investigate a weaker condition than strong convexity which still can enforce an O (cid:0) 1 n (cid:1) stability, without substantially biasing the loss function.

Original: In the remainder of this section, we restrict ourselves to a family of linear model loss functions and show that
Revised: In the remainder of this section, we restrict ourselves to a family of linear model loss functions and show that

Original: the
Revised: the

Original: O ( 1 n ) stability results can be obtained under the framework of average stability.
Revised: O ( 1 n ) stability results can be obtained under the framework of average stability.

Original: The results of Theorem 3 have a dependence on a property of the distribution, and are thus distribution-dependent.
Revised: The results of Theorem 3 have a dependence on a property of the distribution, and are thus distribution-dependent.

Original: We begin with the definition of a ξ -bounded Second Moment.
Revised: We begin with the definition of a ξ -bounded Second Moment.

Original: Essentially, a bounded Second
Revised: Essentially, a bounded Second

Original: Moment dataset requires an average linear dependence of
Revised: Moment dataset requires an average linear dependence of

Original: Span { x 1 , ..., x n } .
Revised: Span { x 1 , ..., x n } .

Original: Recall that the i -th sample is of the form
Revised: Recall that the i -th sample is of the form

Original: z i =
Revised: z i =

Original: ( x i , y i ) .
Revised: ( x i , y i ) .

Original: Definition 8.
Revised: Definition 8.

Original: A set S =
Revised: A set S =

Original: { ( x 1 , y 1 ) , ..., ( x n , y n ) } is defined to have ξ S - bounded Second Moment if ∀ v ∈
Revised: { ( x 1 , y 1 ) , ..., ( x n , y n ) } is defined to have ξ S - bounded Second Moment if ∀ v ∈

Original: Span { x 1 , ..., x n }
Revised: Span { x 1 , ..., x n }

Original: [Equation]
Revised: [Equation]

Original: A distribution D has a ( ξ, n, µ ) -inversely bounded Second
Revised: A distribution D has a ( ξ, n, µ ) -inversely bounded Second

Original: Moment if there exists a constant ξ > 0 such that
Revised: Moment if there exists a constant ξ > 0 such that

Original: [Equation]
Revised: [Equation]

Original: Remark 2.
Revised: Remark 2.

Original: The value of ξ S is always lower bounded by the minimum nonzero eigenvalue of 1 n (cid:80)
Revised: The value of ξ S is always lower bounded by the minimum nonzero eigenvalue of 1 n (cid:80)

Original: j x j x ⊤ j which is the empirical second moment of data with size n .
Revised: j x j x ⊤ j which is the empirical second moment of data with size n .

Original: Proposition 1 (Example of distribution with inversely bounded Second Moment) .
Revised: Proposition 1 (Example of distribution with inversely bounded Second Moment) .

Original: Let E x ∼D
Revised: Let E x ∼D

Original: [ xx ⊤ ] = Σ and ξ be the minimum non-zero eigenvalue of Σ .
Revised: [ xx ⊤ ] = Σ and ξ be the minimum non-zero eigenvalue of Σ .

Original: Suppose
Revised: Suppose

Original: S = { ( x 1 , y 1 ) , ..., ( x n , y n ) } is sampled from D with the x ∈ R d with ∥ x ∥ ≤ 1 .
Revised: S = { ( x 1 , y 1 ) , ..., ( x n , y n ) } is sampled from D with the x ∈ R d with ∥ x ∥ ≤ 1 .

Original: Then, there exists universal constant C, c
Revised: Then, there exists universal constant C, c

Original: so that if n ≥ max { 4 C 2 d ξ 2 , 512 cξ 2 log( 1 ξ ) } , D has a ( ξ 3 , n, µ ) -inversely bounded Second Moment if µ ≥ 1 n 4 .
Revised: so that if n ≥ max { 4 C 2 d ξ 2 , 512 cξ 2 log( 1 ξ ) } , D has a ( ξ 3 , n, µ ) -inversely bounded Second Moment if µ ≥ 1 n 4 .

Original: In our next theorem, we leverage the inversely bounded
Revised: In our next theorem, we leverage the inversely bounded

Original: Second Moment property to prove a non-accumulated onaverage stability bound for SGD on linear models with a regularized loss function.
Revised: Second Moment property to prove a non-accumulated onaverage stability bound for SGD on linear models with a regularized loss function.

Original: We characterize a linear model by rewriting the loss function f ( w ; z ) in terms of f y ( w ⊤ x ) where f y ( · ) is a scalar function depending only on the inner product of the model parameter w and the input feature x .
Revised: We characterize a linear model by rewriting the loss function f ( w ; z ) in terms of f y ( w ⊤ x ) where f y ( · ) is a scalar function depending only on the inner product of the model parameter w and the input feature x .

Original: Theorem 3 (Data-dependent stability of SGD with inversely bounded Second Moment) .
Revised: Theorem 3 (Data-dependent stability of SGD with inversely bounded Second Moment) .

Original: Suppose a loss function f ( w, z ) is of the form
Revised: Suppose a loss function f ( w, z ) is of the form

Original: [Equation]
Revised: [Equation]

Original: where f y ( w ⊤ x ) satisfies (1) | f ′ y ( · ) | ≤ L , (2) 0 < γ ≤ f ′′ y
Revised: where f y ( w ⊤ x ) satisfies (1) | f ′ y ( · ) | ≤ L , (2) 0 < γ ≤ f ′′ y

Original: ( · ) ≤ β , (3) S, S ′ are sampled from D with ξ be the minimum nonzero eigenvalue of E x ∼D
Revised: ( · ) ≤ β , (3) S, S ′ are sampled from D with ξ be the minimum nonzero eigenvalue of E x ∼D

Original: [ xx ⊤ ] and a uniformly bounded support X : ∥ x ∥ ≤ 1 , X ⊂ R d and 4) µ ≥ γn 4 .
Revised: [ xx ⊤ ] and a uniformly bounded support X : ∥ x ∥ ≤ 1 , X ⊂ R d and 4) µ ≥ γn 4 .

Original: Let W be a convex and compact set, w t and w ′ t be the outputs of SGD on S and S ′ after t steps, respectively.
Revised: Let W be a convex and compact set, w t and w ′ t be the outputs of SGD on S and S ′ after t steps, respectively.

Original: Let the divergence ∆ t :=
Revised: Let the divergence ∆ t :=

Original: w t − w ′ t and α ≤ µ 2 β 2 be the step size of SGD.
Revised: w t − w ′ t and α ≤ µ 2 β 2 be the step size of SGD.

Original: There exists universal constant C, c
Revised: There exists universal constant C, c

Original: so that if
Revised: so that if

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original:  , then
Revised: } , then

Original: [Equation]
Revised: [Equation]

Original: and 
Revised: and (cid:98) [Equation] D

Original: [Equation]
Revised: [Equation]

Original: Remark 3.
Revised: Remark 3.

Original: The inversely bounded Second Moment condition allows SGD to maintain an average stability guarantee for a family of widely used models with a negligible regularizer and large sample size.
Revised: The inversely bounded Second Moment condition allows SGD to maintain an average stability guarantee for a family of widely used models with a negligible regularizer and large sample size.

Original: The theorem suggests that if the dataset S is sampled from a ‘good’ distribution, one can obtain an advanced generalization property which mainly depends on the distribution.
Revised: The theorem suggests that if the dataset S is sampled from a ‘good’ distribution, one can obtain an advanced generalization property which mainly depends on the distribution.

Original: The theorem also justifies the common choice of small values for the weight in the L 2regularizer (also known as weight decay) when training ridge regression type models.
Revised: The theorem also justifies the common choice of small values for the weight in the L 2regularizer (also known as weight decay) when training ridge regression type models.

Original: Note that the term µ 2 w ⊤ w makes the loss function strongly convex, and a O (cid:0) 1 n (cid:1) is established with µ = O (1) in [Hardt et al., 2016].
Revised: Note that the term µ 2 w ⊤ w makes the loss function strongly convex, and a O (cid:0) 1 n (cid:1) is established with µ = O (1) in [Hardt et al., 2016].

Original: The major difference of Theorem 3 is that the weight of the ℓpenalty µ is O (cid:0) 1 n 4 (cid:1) for uniformly bounded x .
Revised: The major difference of Theorem 3 is that the weight of the ℓpenalty µ is O (cid:0) 1 n 4 (cid:1) for uniformly bounded x .

Original: A small value of µ will not bias the original loss function thus allow the
Revised: A small value of µ will not bias the original loss function thus allow the

Original: SGD to sufficiently minimize the empirical risk.
Revised: SGD to sufficiently minimize the empirical risk.

Original: In stead of leveraging the ℓ 2 penalty, the stability of SGD is obtained upon the ‘nice’ property of the distribution.
Revised: In stead of leveraging the ℓ 2 penalty, the stability of SGD is obtained upon the ‘nice’ property of the distribution.

Original: Example: Linear regression.
Revised: Example: Linear regression.

Original: Linear regression minimizes the quadratic loss on w : f ( w, S )
Revised: Linear regression minimizes the quadratic loss on w : f ( w, S )

Original: = 12 n (cid:80) x j ∈ S ( x ⊤ j w − y j ) 2 , w ∈ W , where W is a convex compact set that contains the origin and has bounded radius R .
Revised: = 12 n (cid:80) x j ∈ S ( x ⊤ j w − y j ) 2 , w ∈ W , where W is a convex compact set that contains the origin and has bounded radius R .

Original: The Hessian of an individual linear regression loss term is x j x ⊤ j which is not strongly-convex .
Revised: The Hessian of an individual linear regression loss term is x j x ⊤ j which is not strongly-convex .

Original: However, one can rewrite the loss function as f y ( w ⊤ x ) where f ′′ y ( · ) = 1 .
Revised: However, one can rewrite the loss function as f y ( w ⊤ x ) where f ′′ y ( · ) = 1 .

Original: Next we show certain conditions that are sufficient to make | f ′ ( · ) | ≤ L .
Revised: Next we present certain conditions that are sufficient to make | f ′ ( · ) | ≤ L .

Original: We assume ∥ x i ∥
Revised: We assume ∥ x i ∥

Original: = 1 , y i ∈ [ − 1 , 1] , ∀ i ∈ [ n ] .
Revised: = 1 , y i ∈ [ − 1 , 1] , ∀ i ∈ [ n ] .

Original: Let Π W ( v ) =
Revised: Let Π W ( v ) =

Original: argmin w ∈W ∥ w − v ∥ .
Revised: argmin w ∈W ∥ w − v ∥ .

Original: Note that SGD updates as w t
Revised: Note that SGD updates as w t

Original: +1 =
Revised: +1 =

Original: Π w ∈W (cid:18) w t − α t ( x ⊤ j w t − y j ) x j (cid:19) .
Revised: Π w ∈W (cid:18) w t − α t ( x ⊤ j w t − y j ) x j (cid:19) .

Original: One can show that sup w ∈W sup x,y ∈ S f ′ y ( w ⊤ x ) ≤ R + 1 .
Revised: One can show that sup w ∈W sup x,y ∈ S f ′ y ( w ⊤ x ) ≤ R + 1 .

Original: 4.2 NON-CONVEX CASE
Revised: 4.2 NON-CONVEX CASE

Original: In this section, we construct a non-convex loss function to analyze the tightness of the divergence bound in [Hardt et al., 2016].
Revised: In this section, we construct a non-convex loss function to analyze the tightness of the divergence bound in [Hardt et al., 2016].

Original: We first focus on the case where SGD applies a step size that decreases with t .
Revised: We first focus on the case where SGD applies a step size that decreases with t .

Original: Define a hitting time to be the time t that satisfies w t − 1 − w ′ t − 1 = 0 and w t − w ′ t ̸ = 0 .
Revised: Define a hitting time to be the time t that satisfies w t − 1 − w ′ t − 1 = 0 and w t − w ′ t ̸ = 0 .

Original: We first fix a hitting time t 0 and prove Lemma 3.
Revised: We first fix a hitting time t 0 and prove Lemma 3.

Original: Lemma 3 (Divergence of non-convex loss function) .
Revised: Lemma 3 (Divergence of non-convex loss function) .

Original: There exists a function f which is non-convex and β -smooth, twin datasets S, S ′ and constant a > 0 such that the following holds: if SGD is run using step size α t = a 0 .
Revised: There exists a function f which is non-convex and β -smooth, twin datasets S, S ′ and constant a > 0 such that the following holds: if SGD is run using step size α t = a 0 .

Original: 99 βt for 1 ≤ t < T , and w t , w ′ t are the outputs of SGD on S and
Revised: 99 βt for 1 ≤ t < T , and w t , w ′ t are the outputs of SGD on S and

Original: S ′ , respectively, and ∆ t = w t − w ′ t , then ∀ 1 ≤ t 0 ≤
Revised: S ′ , respectively, and ∆ t = w t − w ′ t , then ∀ 1 ≤ t 0 ≤

Original: T,
Revised: T,

Original: E

Original: [Equation]
Revised: [Equation]

Original: The following theorem follows from Lemma 3 by optimizing over t 0 .
Revised: The following theorem follows from Lemma 3 by optimizing over t 0 .

Original: The choice of hitting time t 0 plays an important role in the analysis, which is also illustrated in the “burn-in
Revised: The choice of hitting time t 0 plays an important role in the analysis, which is also illustrated in the “burn-in

Original: Lemma” 3.11 in [Hardt et al., 2016].
Revised: Lemma” 3.11 in [Hardt et al., 2016].

Original: Theorem 4 (Lower bound for non-convex loss functions) .
Revised: Theorem 4 (Lower bound for non-convex loss functions) .

Original: Let w t , w ′ t be the outputs of SGD on twin datasets S, S ′ , and ∆ t = w t − w ′ t .
Revised: Let w t , w ′ t be the outputs of SGD on twin datasets S, S ′ , and ∆ t = w t − w ′ t .

Original: There exists a function f which is non-convex and β -smooth, twin datasets S, S ′ and constants a < 0 .such that the divergence of SGD after T > n rounds using constant step size α t = a 0 .
Revised: There exists a function f which is non-convex and β -smooth, twin datasets S, S ′ and constants a < 0 .such that the divergence of SGD after T > n rounds using constant step size α t = a 0 .

Original: βt satisfies
Revised: βt satisfies

Original: [Equation]
Revised: [Equation]

Original: Remark 4.
Revised: Remark 4.

Original: In the above theorem, we require α
Revised: In the above theorem, we require α

Original: [Equation]
Revised: [Equation]

Original: with an extra constant factor 10 .
Revised: with an extra constant factor 10 .

Original: 99 to apply the inequality 1 + ax 0 .
Revised: 99 to apply the inequality 1 + ax 0 .

Original: 99 > e ax with sufficiently small a .
Revised: 99 > e ax with sufficiently small a .

Original: To remove the constant 1 / 0 . 99 in the learning rate one need to avoid using the inequality 1 + x < e x at the first place in deriving the upper bound.
Revised: To remove the constant 1 / 0 . 99 in the learning rate one need to avoid using the inequality 1 + x < e x at the first place in deriving the upper bound.

Original: This can be done by a refined analysis for upper bound via setting learning rate α
Revised: This can be done by a refined analysis for upper bound via setting learning rate α

Original: [Equation]
Revised: [Equation]

Original: Remark 5.
Revised: Remark 5.

Original: Note in [Hardt et al., 2016], an assumption is made on the non-convex loss function, namely that f ( u, z ) ∈ (0 , 1) .
Revised: Note in [Hardt et al., 2016], an assumption is made on the non-convex loss function, namely that f ( u, z ) ∈ (0 , 1) .

Original: In our lower bound construction, we do not have such an assumption thus our lower bound can not be directly compared with the upper bound in [Hardt et al., 2016].
Revised: In our lower bound construction, we do not have such an assumption thus our lower bound can not be directly compared with the upper bound in [Hardt et al., 2016].

Original: The bound in [Hardt et al., 2016] is of the form O (cid:16) T a 1+ a n (cid:17) , for
Revised: The bound in [Hardt et al., 2016] is of the form O (cid:16) T a 1+ a n (cid:17) , for

Original: T a 1+ a ≥ n , our lower bound will exceed the upper bound in [Hardt et al., 2016].
Revised: T a 1+ a ≥ n , our lower bound will exceed the upper bound in [Hardt et al., 2016].

Original: However, such a gap implies that even with additional assumptions in [Hardt et al., 2016], the upper bound still may not be tight.
Revised: However, such a gap implies that even with additional assumptions in [Hardt et al., 2016], the upper bound still may not be tight.

Original: The lower bound is derived by choosing the hitting time t 0 < n , i.e., the first time SGD picks the different entries z, z ′ in the twin dataset before round n , suggesting additional space for improvement on the analysis.
Revised: The lower bound is derived by choosing the hitting time t 0 < n , i.e., the first time SGD picks the different entries z, z ′ in the twin dataset before round n , suggesting additional space for improvement on the analysis.

Original: We investigate this gap and derive a tighter bound in the next theorem which improves on Theorem 3.12 in [Hardt et al., 2016].
Revised: We investigate this gap and derive a tighter bound in the next theorem which improves on Theorem 3.12 in [Hardt et al., 2016].

Original: To prove a better upper bound for non-convex losses, we need the following lemma, which gives us the expectation of divergence for a given hitting time t k + 1 , which is the timestamp of SGD first selecting the k -th different sample.
Revised: To prove a better upper bound for non-convex losses, we need the following lemma, which gives us the expectation of divergence for a given hitting time t k + 1 , which is the timestamp of SGD first selecting the k -th different sample.

Original: Lemma 4.
Revised: Lemma 4.

Original: [Hardt et al., 2016]
Revised: [Hardt et al., 2016]

Original: Assume f is β -smooth and
Revised: Assume f is β -smooth and

Original: L -Lipschitz.
Revised: L -Lipschitz.

Original: Let w t , w ′ t be outputs of SGD on twin datasets
Revised: Let w t , w ′ t be outputs of SGD on twin datasets

Original: S, S ′ respectively after t iterations and let ∆ t = [ w t − w ′ t ] and δ t = E ∥ ∆ t ∥ .
Revised: S, S ′ respectively after t iterations and let ∆ t = [ w t − w ′ t ] and δ t = E ∥ ∆ t ∥ .

Original: Running SGD on f ( w ; S ) with step size α t = a βt satisfies the following conditions:
Revised: Running SGD on f ( w ; S ) with step size α t = a βt satisfies the following conditions:

Original: • The SGD update rule is a (1 + α t β ) -expander and 2 α t L -bounded.
Revised: • The SGD update rule is a (1 + α t β ) -expander and 2 α t L -bounded.

Original: • E A [ ∥ ∆ t ∥| ∆ t − 1 ] ≤ (1 + α t β ) ∥
Revised: • E A [ ∥ ∆ t ∥| ∆ t − 1 ] ≤ (1 + α t β ) ∥

Original: ∆ t − 1 ∥ + 2 α t Ln .
Revised: ∆ t − 1 ∥ + 2 α t Ln .

Original: • E A [ ∥ ∆ T ∥| ∆ t k = 0] ≤ (cid:0)
Revised: • E A [ ∥ ∆ T ∥| ∆ t k = 0] ≤ (cid:0)

Original: Tt k (cid:1) a 

Revised: Tt k (cid:1) a

Original: Ln .
Revised: Ln .

Original: Lemma 4 bounds the case when the hitting time is equals to t k .
Revised: Lemma 4 bounds the case when the hitting time is equals to t k .

Original: In the proof of Hardt et al.
Revised: In the proof of Hardt et al.

Original: [2016] for non-convex stability upper bound, the t k is chose to be T a 1+ a .
Revised: [2016] for non-convex stability upper bound, the t k is chose to be T a 1+ a .

Original: However, we observe that a choice of t k with additional care on the analysis leads to an improved upper bound.
Revised: However, we observe that a choice of t k with additional care on the analysis leads to an improved upper bound.

Original: Therefore, we could obtain the upper bound for the stability of uniform sampling SGD as follows:
Revised: Therefore, we could obtain the upper bound for the stability of uniform sampling SGD as follows:

Original: Theorem 5 (Uniform sampling SGD) .
Revised: Theorem 5 (Uniform sampling SGD) .

Original: Assume f is β -smooth and L -Lipschitz.
Revised: Assume f is β -smooth and L -Lipschitz.

Original: Running T > n iterations of
Revised: Running T > n iterations of

Original: SGD on f ( w ; S ) with step size α t = aβt , the stability of
Revised: SGD on f ( w ; S ) with step size α t = aβt , the stability of

Original: SGD satisfies
Revised: SGD satisfies

Original: [Equation]
Revised: [Equation]

Original: We remark that the above analysis is for uniform sampling
Revised: We remark that the above analysis is for uniform sampling

Original: SGD, where the algorithm keeps sampling with replacement.
Revised: SGD, where the algorithm keeps sampling with replacement.

Original: We also derive a version of Theorem 5 which samples without replacement in the appendix, which also matches the lower bound.
Revised: We also derive a version of Theorem 5 which samples without replacement in the appendix, which also matches the lower bound.

Original: Dividing our bound by the bound in Theorem 3.12 of [Hardt et al., 2016], we obtain the ratio ˜Ω (cid:18) T a 2 1+ a n a (cid:19) .
Revised: Dividing our bound by the bound in Theorem 3.12 of [Hardt et al., 2016], we obtain the ratio ˜Ω (cid:18) T a 2 1+ a n a (cid:19) .

Original: This factor is less than 1 (and so we improve the upper bound) exactly when T a 1+ a ≤ n .
Revised: This factor is less than 1 (and so we improve the upper bound) exactly when T a 1+ a ≤ n .

Original: Note that this is potentially a large range as a is a small and positive constant.
Revised: Note that this is potentially a large range as a is a small and positive constant.

Original: In [Kuzborskij and Lampert, 2018], the data-dependent stability of SGD is analyzed, incorporating the dependence on the variance of SGD curvature and the loss of the initial parameter w 0 in analyzing the divergence of SGD.
Revised: In [Kuzborskij and Lampert, 2018], the data-dependent stability of SGD is analyzed, incorporating the dependence on the variance of SGD curvature and the loss of the initial parameter w 0 in analyzing the divergence of SGD.

Original: This framework has applications in transfer learning, as well as implications including optimistic generalization error.
Revised: This framework has applications in transfer learning, as well as implications including optimistic generalization error.

Original: We observe that our analysis in Theorems 5 can be combined with the data-dependent framework, and we now report our data-dependent versions of Theorems 5.
Revised: We observe that our analysis in Theorems 5 can be combined with the data-dependent framework, and we now report our data-dependent versions of Theorems 5.

Original: The analysis requires the additional bounded variance assumption for SGD such that
Revised: The analysis requires the additional bounded variance assumption for SGD such that

Original: [Equation]
Revised: [Equation]

Original: In the rest of this section we assume the variance of SGD satisfies this property.
Revised: In the rest of this section we assume the variance of SGD satisfies this property.

Original: We borrow the following lemma from [Kuzborskij and Lampert, 2018] which is a data-dependent version of Lemma 4.
Revised: We borrow the following lemma from [Kuzborskij and Lampert, 2018] which is a data-dependent version of Lemma 4.

Original: Lemma 5.
Revised: Lemma 5.

Original: [Kuzborskij and Lampert, 2018]
Revised: [Kuzborskij and Lampert, 2018]

Original: Assume f is β -smooth, L -Lipschitz, and has a ρ -Lipschitz Hessian.
Revised: Assume f is β -smooth, L -Lipschitz, and has a ρ -Lipschitz Hessian.

Original: With w 0 the initial weight and w t , w t ′ the outputs of SGD on twin datasets S, S ′ respectively after t iterations, let ∆ t =
Revised: With w 0 the initial weight and w t , w t ′ the outputs of SGD on twin datasets S, S ′ respectively after t iterations, let ∆ t =

Original: [ w t − w t ′ ] .
Revised: [ w t − w t ′ ] .

Original: Running SGD on f ( w ; S ) with step size α t = bt where b ≤ min { 2 β ,
Revised: Running SGD on f ( w ; S ) with step size α t = bt where b ≤ min { 2 β ,

Original: [Equation]
Revised: [Equation]

Original: has the following properties:
Revised: has the following properties:

Original: The SGD update rule is a (1 + α t ψ t ) -expander
Revised: The SGD update rule is a (1 + α t ψ t ) -expander

Original: and α t L -bounded.
Revised: and α t L -bounded.

Original: Here ψ t = min { β, κ t } where
Revised: Here ψ t = min { β, κ t } where

Original: [Equation]
Revised: [Equation]

Original: E A [ ∥ ∆ t +1 ∥| ∆ t 0 = 0] ≤ { E A [ ∥ ∆ t ∥| ∆ t 0 = 0][

Revised: E A [ ∥ ∆ t +1 ∥| ∆ t 0 = 0] ≤ { E A [ ∥ ∆ t ∥| ∆ t 0 = 0][

Original: + (1 − 1 n ) α t ψ t ] } + 2 α t
Revised: + (1 − 1 n ) α t ψ t ] } + 2 α t

Original: Ln .
Revised: Ln .

Original: E S,S ′ { E A [ ∥ ∆ T ∥| ∆ t 0 = 0] } ≤ Ln (cid:16)
Revised: E S,S ′ { E A [ ∥ ∆ T ∥| ∆ t 0 = 0] } ≤ Ln (cid:16)

Original: Tt 0 (cid:17) ζb ,
Revised: Tt 0 (cid:17) ζb ,

Original: where ζ = ˜ O (min { β, E z [ ∥∇ 2 f ( w 0 ; z ) ∥ 2 ] + ∆ ∗ 1 ,σ 2 } ) , ∆ ∗ 1 ,σ 2 = ρ ( bσ + (cid:112) bE z
Revised: where ζ = ˜ O (min { β, E z [ ∥∇ 2 f ( w 0 ; z ) ∥ 2 ] + ∆ ∗ 1 ,σ 2 } ) , ∆ ∗ 1 ,σ 2 = ρ ( bσ + (cid:112) bE z

Original: [ f ( w 0 ; z )] − k ∗ and k ∗ = inf w E z
Revised: [ f ( w 0 ; z )] − k ∗ and k ∗ = inf w E z

Original: [ f ( w ; z )] .
Revised: [ f ( w ; z )] .

Original: Based on the above lemma, we can prove an upper bound of on-average stability with uniform sampling SGD using the same technique as for Theorem 5.
Revised: Based on the above lemma, we can prove an upper bound of on-average stability with uniform sampling SGD using the same technique as for Theorem 5.

Original: Theorem 6.
Revised: Theorem 6.

Original: (Data-dependent version of Theorem 5) Assume f is β -smooth, L -Lipschitz, and has a ρ -Lipschitz
Revised: (Data-dependent version of Theorem 5) Assume f is β -smooth, L -Lipschitz, and has a ρ -Lipschitz

Original: Hessian.
Revised: Hessian.

Original: Let w t , w t ′ be the outputs of SGD on twin datasets
Revised: Let w t , w t ′ be the outputs of SGD on twin datasets

Original: S, S ′ respectively after t iterations and let ∆ t = [ w t − w t ′ ] and δ t = E A ∥
Revised: S, S ′ respectively after t iterations and let ∆ t = [ w t − w t ′ ] and δ t = E A ∥

Original: ∆ t ∥ .
Revised: ∆ t ∥ .

Original: And let ζ follow the same definition as in Lemma 5.
Revised: And let ζ follow the same definition as in Lemma 5.

Original: Running SGD on f ( w ; S ) with step size α t = bt where b < 1 satisfies
Revised: Running SGD on f ( w ; S ) with step size α t = bt where b < 1 satisfies

Original: [Equation]
Revised: [Equation]

Original: We conclude this section with the following lower bound on the uniform stability of SGD with constant stepsize for non-convex loss functions.
Revised: We conclude this section with the following lower bound on the uniform stability of SGD with constant stepsize for non-convex loss functions.

Original: We show that for non-convex functions satisfying classical conditions β -smooth, we cannot avoid a pessimistic bound.
Revised: We show that for non-convex functions satisfying classical conditions β -smooth, we cannot avoid a pessimistic bound.

Original: Thus, in order to analyze the generalization power of SGD for deep learning loss functions from an optimization perspective, different conditions are necessary.
Revised: Thus, in order to analyze the generalization power of SGD for deep learning loss functions from an optimization perspective, different conditions are necessary.

Original: Theorem 7.
Revised: Theorem 7.

Original: Let w t , w ′ t be the outputs of SGD on twin datasets S, S ′ , and let ∆ t = w t − w ′ t .
Revised: Let w t , w ′ t be the outputs of SGD on twin datasets S, S ′ , and let ∆ t = w t − w ′ t .

Original: There exists a nonconvex, β -smooth function f , twin sets S, S ′ and constants a, γ such that the divergence of SGD after T > n rounds using constant step size α = a 0 .
Revised: There exists a nonconvex, β -smooth function f , twin sets S, S ′ and constants a, γ such that the divergence of SGD after T > n rounds using constant step size α = a 0 .

Original: 99 γ satisfies ε stab ≥ e aT/ 2 /n 2 .
Revised: 99 γ satisfies ε stab ≥ e aT/ 2 /n 2 .

Original:  CONCLUSION AND FUTURE WORK
Revised:  CONCLUSION AND FUTURE WORK

Original: We first provided matching upper and lower dataindependent bounds on the stability of SGD for three kinds of loss functions: convex, strongly-convex, and non-convex, essentially closing the gap in all cases.
Revised: We first provided matching upper and lower dataindependent bounds on the stability of SGD for three kinds of loss functions: convex, strongly-convex, and non-convex, essentially closing the gap in all cases.

Original: We then provided stronger data-dependent generalization bounds for both convex and non-convex loss functions by analyzing averagestability, showing that nice properties of data can both improve generalization and also reduce the need for regularization.
Revised: We then provided stronger data-dependent generalization bounds for both convex and non-convex loss functions by analyzing averagestability, showing that nice properties of data can both improve generalization and also reduce the need for regularization.

Original: At least two interesting open questions arise from our work: a) Can one obtain data-dependent lower bounds on average-stability that show the tightness of existing analysis?
Revised: At least two interesting open questions arise from our work: a) Can one obtain data-dependent lower bounds on average-stability that show the tightness of existing analysis?

Original: b) Can one devise properties of data-distributions or loss functions (perhaps motivated by deep learning) that imply better data-dependent stability bounds?
Revised: b) Can one devise properties of data-distributions or loss functions (perhaps motivated by deep learning) that imply better data-dependent stability bounds?

Original: Acknowledgements
Revised: Acknowledgements

Original: We thank anonymous reviewers for their constructive feedback.
Revised: We thank anonymous reviewers for their constructive feedback.

Original: Mayank Goswami would like to acknowledge support from NSF awards CRII-1755791 and CCF-1910873.
Revised: Mayank Goswami would like to acknowledge support from NSF awards CRII-1755791 and CCF-1910873.

Original: Chao
Revised: Chao

Original: Chen was partially supported by grants NSF IISand
Revised: Chen was partially supported by grants NSF IISand

Original: CCF-1855760.
Revised: CCF-1855760.