Original: From Gradient Flow on Population Loss to Learning with Stochastic Gradient Descent
Revised: From Gradient Flow on Population Loss to Learning with Stochastic Gradient Descent

Original: Abstract
Revised: Abstract

Original: Stochastic Gradient Descent (SGD) has been the method of choice for learninglarge-scale non-convex models.
Revised: Stochastic Gradient Descent (SGD) has been the method of choice for learninglarge-scale non-convex models.

Original: While a general analysis of when SGD workshas been elusive, there has been a lot of recent progress in understanding theconvergence of Gradient Flow (GF) on the population loss, partly due to thesimplicity that a continuous-time analysis buys us.
Revised: While a general analysis of when SGD workshas been elusive, there has been a lot of recent progress in understanding theconvergence of Gradient Flow (GF) on the population loss, partly due to thesimplicity that a continuous-time analysis buys us.

Original: An overarching theme of ourpaper is providing general conditions under which SGD converges, assuming that
Revised: An overarching theme of ourpaper is providing general conditions under which SGD converges, assuming that

Original: GF on the population loss converges.
Revised: GF on the population loss converges.

Original: Our main tool to establish this connectionis a general converse Lyapunov like theorem, which implies the existence of a
Revised: Our main tool to establish this connectionis a general converse Lyapunov like theorem, which implies the existence of a

Original: Lyapunov potential under mild assumptions on the rates of convergence of GF.
Revised: Lyapunov potential under mild assumptions on the rates of convergence of GF.

Original: Infact, using these potentials, we show a one-to-one correspondence between rates ofconvergence of GF and geometrical properties of the underlying objective.
Revised: Infact, using these potentials, we show a one-to-one correspondence between rates ofconvergence of GF and geometrical properties of the underlying objective.

Original: Whenthese potentials further satisfy certain self-bounding properties, we show that theycan be used to provide a convergence guarantee for Gradient Descent (GD) and
Revised: Whenthese potentials further satisfy certain self-bounding properties, we show that theycan be used to provide a convergence guarantee for Gradient Descent (GD) and

Original: SGD (even when the paths of GF and GD/SGD are quite far apart).
Revised: SGD (even when the paths of GF and GD/SGD are quite far apart).

Original: It turns outthat these self-bounding assumptions are in a sense also necessary for GD/SGD towork.
Revised: It turns outthat these self-bounding assumptions are in a sense also necessary for GD/SGD towork.

Original: Using our framework, we provide a uniﬁed analysis for GD/SGD not onlyfor classical settings like convex losses, or objectives that satisfy PL/ KL properties,but also for more complex problems including Phase Retrieval and Matrix sq-root,and extending the results in the recent work of Chatterjee [8].
Revised: Using our framework, we provide a uniﬁed analysis for GD/SGD not onlyfor classical settings like convex losses, or objectives that satisfy PL/ KL properties,but also for more complex problems including Phase Retrieval and Matrix sq-root,and extending the results in the recent work of Chatterjee [8].

Original:  Introduction
Revised:  Introduction

Original: Stochastic Gradient Descent (SGD) has been a method of choice to train complex, large scale machinelearning models.
Revised: Stochastic Gradient Descent (SGD) has been a method of choice to train complex, large scale machinelearning models.

Original: While understanding of SGD for convex objectives is comprehensive, a generalunderstanding of when SGD works for non-convex models has been somewhat elusive.
Revised: While understanding of SGD for convex objectives is comprehensive, a generalunderstanding of when SGD works for non-convex models has been somewhat elusive.

Original: A largeslew of properties like, convexity [36], one-point-convexity [27], linearizability [24], K Ł [3, 31]and P Ł [25, 37, 32] properties, and more problem speciﬁc, tailored analysis of SGD and Gradient
Revised: A largeslew of properties like, convexity [35], one-point-convexity [26], linearizability [23], KL [3, 30]and PL [24, 36, 31] properties, and more problem speciﬁc, tailored analysis of SGD and Gradient

Original: Descent (GD) for speciﬁc problem instances like matrix square-root problem, matrix completion [23],phase retrieval [7, 10, 41] and Dictionary learning [2] have been proposed.
Revised: Descent (GD) for speciﬁc problem instances like matrix square-root problem, matrix completion [22],phase retrieval [7, 10] and Dictionary learning [2] have been proposed.

Original: Recent success of SGD inover-parameterized deep learning models have lead to the idea that SGD perhaps optimizes trainingobjective with an implicit bias given by some implicit regularizer [16, 38, 21, 17, 18].
Revised: Recent success of SGD inover-parameterized deep learning models have lead to the idea that SGD perhaps optimizes trainingobjective with an implicit bias given by some implicit regularizer [15, 37, 20, 16, 17].

Original: However,in [24] it is argued that there are over-parameterized models for which SGD works but no methodthat minimizes an implicit regularized training objective can learn successfully, thus showing that ingeneral, the success of SGD cannot be explained by implicit regularization.
Revised: However,in [23] it is argued that there are over-parameterized models for which SGD works but no methodthat minimizes an implicit regularized training objective can learn successfully, thus showing that ingeneral, the success of SGD cannot be explained by implicit regularization.

Original: The goal of our paper is to provide a unifying analysis for when SGD/GD works.
Revised: The goal of our paper is to provide a unifying analysis for when SGD/GD works.

Original: More speciﬁcally,we do this via ﬁrst showing that Gradient Flow (GF) works and then extending this analysis to SGDand GD.
Revised: More speciﬁcally,we do this via ﬁrst showing that Gradient Flow (GF) works and then extending this analysis to SGDand GD.

Original: Gradient Flow (GF) can be seen as a continuous time analogue of GD.
Revised: Gradient Flow (GF) can be seen as a continuous time analogue of GD.

Original: In an idealizedworld, if one had access to the population loss, it turns out that convergence analysis for running gradient ﬂow on population loss is somewhat simpler due to tools from continuous time analysis and
Revised: In an idealizedworld, if one had access to the population loss, it turns out that convergence analysis for running gradient ﬂow on population loss is somewhat simpler due to tools from continuous time analysis and

Original: PDEs.
Revised: PDEs.

Original: There has been several recent works [4, 9, 11] that have provided convergence analysis for
Revised: There has been several recent works [4, 9, 11] that have provided convergence analysis for

Original: GF even on non-convex objectives.
Revised: GF even on non-convex objectives.

Original: The high level theme of this paper is to show that, under somemild/appropriate assumptions of population loss/objective and on the noise of gradient estimates, “if,
Revised: The high level theme of this paper is to show that, under somemild/appropriate assumptions of population loss/objective and on the noise of gradient estimates, “if,

Original: GF converges on population loss, then SGD that uses one fresh example per iteration is successfulat learning”.
Revised: GF converges on population loss, then SGD that uses one fresh example per iteration is successfulat learning”.

Original: Notice, that GF converging on population loss is a purely deterministic optimizationproblem.
Revised: Notice, that GF converging on population loss is a purely deterministic optimizationproblem.

Original: However, the fact that SGD works is a learning result that implies a sample complexitybound.
Revised: However, the fact that SGD works is a learning result that implies a sample complexitybound.

Original: There have been past works that have aimed at providing convergence analysis for Gradient Descent(GD) starting from Gradient Flow (GF).
Revised: There have been past works that have aimed at providing convergence analysis for Gradient Descent(GD) starting from Gradient Flow (GF).

Original: Typical route to obtain a convergence analysis of GD startingfrom GF tries to think of GD updates as approximating GF path.
Revised: Typical route to obtain a convergence analysis of GD startingfrom GF tries to think of GD updates as approximating GF path.

Original: Even with more sophisticateddiscretization schemes like Euler discretization, obtaining convergence for GD, starting from GF canbe quite complex.
Revised: Even with more sophisticateddiscretization schemes like Euler discretization, obtaining convergence for GD, starting from GF canbe quite complex.

Original: In this paper, to show that when GF converges, SGD/GD also converges, we take adifferent approach.
Revised: In this paper, to show that when GF converges, SGD/GD also converges, we take adifferent approach.

Original: A key tool for proving convergence results for GF is by constructing so called
Revised: A key tool for proving convergence results for GF is by constructing so called

Original: Lyapunov potentials.
Revised: Lyapunov potentials.

Original: In the literature of Ordinary Differential Equations (ODEs), when ODEs haveregular enough convergence rates, one can show, so called converse
Revised: In the literature of Ordinary Differential Equations (ODEs), when ODEs haveregular enough convergence rates, one can show, so called converse

Original: Lyapunov theorems (see [26] fora nice survey of classic results) that state that when an ODE converges to stable solutions, there hasto exist a corresponding Lyapunov potential.
Revised: Lyapunov theorems (see [25] fora nice survey of classic results) that state that when an ODE converges to stable solutions, there hasto exist a corresponding Lyapunov potential.

Original: While convergence of GF in terms of sub-optimality isquite different from convergence in the ODE sense, in this paper, we ﬁrst prove a converse Lyapunovstyle theorem for GF.
Revised: While convergence of GF in terms of sub-optimality isquite different from convergence in the ODE sense, in this paper, we ﬁrst prove a converse Lyapunovstyle theorem for GF.

Original: Speciﬁcally, we show that when GF converges in terms of sub-optimalityto a global minimum, then there has to exist a corresponding Lyapunov potential and using suchpotential, the rates can be recovered.
Revised: Speciﬁcally, we show that when GF converges in terms of sub-optimalityto a global minimum, then there has to exist a corresponding Lyapunov potential and using suchpotential, the rates can be recovered.

Original: This result becomes a starting point for our analysis.
Revised: This result becomes a starting point for our analysis.

Original: We showthat if this Lyapunov potential (obtained from the converse Lyapunov style theorem) satisﬁes certainextra self-bounding regularity conditions, then one can show that GD and SGD algorithms convergein terms of sub-optimality when appropriate step sizes are used.
Revised: We showthat if this Lyapunov potential (obtained from the converse Lyapunov style theorem) satisﬁes certainextra self-bounding regularity conditions, then one can show that GD and SGD algorithms convergein terms of sub-optimality when appropriate step sizes are used.

Original: Such convergence for SGD/GDhappens even when the GF path and GD/SGD paths can be quite different.
Revised: Such convergence for SGD/GDhappens even when the GF path and GD/SGD paths can be quite different.

Original: We now summarize our main contributions below:
Revised: We now summarize our main contributions below:

Original: • We prove a converse Lyapunov style theorem that shows that if gradient ﬂow convergeswith rate speciﬁed by with an appropriate rate function, then there exists a correspondingLyapunov potential that recovers this rate.•
Revised: We prove a converse Lyapunov style theorem that shows that if gradient ﬂow converges withrate speciﬁed by with an appropriate rate function, then there exists a corresponding Lyapunovpotential that recovers this rate.

Original: We provide a geometric characterization for a given rate of convergence of gradient ﬂow (ie.
Revised: We provide a geometric characterization for a given rate of convergence of gradient ﬂow (ie.

Original: GF converges at a particular rate if and only if a speciﬁc geometric condition on objectiveholds.)
Revised: GFconverges at a particular rate if and only if a speciﬁc geometric condition on objective holds.)

Original: • There are problems for which GF converges at a speciﬁc rate but GD can be arbitrarily slowto converge.•
Revised: There are problems for which GF converges at a speciﬁc rate but GD can be arbitrarily slow to converge.4.

Original: This motivates the necessity of additional conditions to ensure GD/SGD converges evenwhen GF converges.
Revised: This motivates the necessity of additional conditions to ensure GD/SGD converges evenwhen GF converges.

Original: We provide certain self-bounding regularity conditions on the Lyapunovpotential, under which we show that GD converges.
Revised: We provide certain self-bounding regularity conditions on the Lyapunovpotential, under which we show that GD converges.

Original: We also provide conditions on gradientestimate noise under which we show that SGD using these gradient estimates also converges.• We instantiate our results for problems such as matrix square-root and phase retrieval,amongst other applications.
Revised: We also provide conditions on gradientestimate noise under which we show that SGD using these gradient estimates also converges.5. We instantiate our results for problems such as matrix square-root and phase retrieval, amongstother applications.

Original:  Setup
Revised:  Setup

Original: Given a continuously differentiable function and non-negative function F ∶
Revised: Given a continuously differentiable function and non-negative function F ∶

Original: R d ￿
Revised: R d ￿

Original: R , our goal isto minimize F ( w ) .
Revised: R , our goal isto minimize F ( w ) .

Original: Without any loss of generality, we assume that min w F ( w ) = 0 .
Revised: Without any loss of generality, we assume that min w F ( w ) = 0 .

Original: First-orderalgorithms are popular for such optimization tasks.
Revised: First-orderalgorithms are popular for such optimization tasks.

Original: In the following, we formally describe the
Revised: In the following, we formally describe the

Original: Gradient Descent and Stochastic Gradient Descent algorithm, and their continuous time counterpartcalled gradient ﬂow.
Revised: Gradient Descent and Stochastic Gradient Descent algorithm, and their continuous time counterpartcalled gradient ﬂow.

Original: Gradient Descent (GD).
Revised: Gradient Descent (GD).

Original: Gradient descent is the most popular iterative algorithm to minimizedifferentiable functions.
Revised: Gradient descent is the most popular iterative algorithm to minimizedifferentiable functions.

Original: Starting from an initial point w 0 , GD on the function F ( w ) performs thefollowing update on every iteration:
Revised: Starting from an initial point w 0 , GD on the function F ( w ) performs thefollowing update on every iteration:

Original: [Equation]
Revised: [Equation]

Original: where ⌘ denotes the step size.
Revised: where ⌘ denotes the step size.

Original: After T rounds, GD algorithm returns the point ̂ w T ∶ = argmins ≤
Revised: After T rounds, GD algorithm returns the point [Equation] argmin ≤

Original: T F ( w s ) .
Revised: T F ( w s ) .

Original: Stochastic Gradient Descent (SGD).
Revised: Stochastic Gradient Descent (SGD).

Original: Stochastic gradient descent (SGD) has been the methodof choice for optimizing complex convex and non-convex learning problems in practice.
Revised: Stochastic gradient descent (SGD) has been the methodof choice for optimizing complex convex and non-convex learning problems in practice.

Original: In thelearning setting, F ( w ) corresponds to the unknown population loss and can be written as F ( w ) =
Revised: In thelearning setting, F ( w ) corresponds to the unknown population loss and can be written as F ( w ) =

Original: E z ∼ D
Revised: E z ∼ D

Original: [ f ( w ; z )] where the expectation is taken with respect to samples z drawn from an unknowndistribution
Revised: [ f ( w ; z )] where the expectation is taken with respect to samples z drawn from an unknowndistribution

Original: D .
Revised: D .

Original: SGD algorithm (mini-batch size 1) is an iterative algorithm that at every round t ≥ 0 ,draws a fresh sample z t from
Revised: SGD algorithm (mini-batch size 1) is an iterative algorithm that at every round t ≥ 0 ,draws a fresh sample z t from

Original: D to compute a stochastic unbiased estimate ∇ f ( w i ; z i ) of the gradient∇
Revised: D to compute a stochastic unbiased estimate ∇ f ( w i ; z i ) of the gradient∇

Original: F ( w i ) , and performs the updatew t + ← w t − ⌘ ∇ f ( w
Revised: F ( w i ) , and performs the update

Original: [Equation]
Revised: [Equation]

Original: z

Original: [Equation]
Revised: [Equation]

Original: ) (2) where ⌘ is the step size and w 0 denotes the initial point.
Revised: where ⌘ is the step size and w 0 denotes the initial point.

Original: After T rounds, SGD algorithm returns ̂
Revised: After T rounds, SGD algorithm returns ̂

Original: w Tby sampling a point uniformly at random from the set { w 1 ,...,w T } .
Revised: w Tby sampling a point uniformly at random from the set { w 1 ,...,w T } .

Original: Gradient Flow (GF).
Revised: Gradient Flow (GF).

Original: Gradient ﬂow from a point w 0 is continuous time process ( w ( t )) t ≥ 0 that98 starts at w ( ) =
Revised: Gradient ﬂow from a point w 0 is continuous time process ( w ) that starts atw ( ) =

Original: w 0 and evolves asd w ( t
Revised: w 0 and evolves as [Equation]

Original: [Equation]
Revised: [Equation]

Original: F ( w ( t )) . GF has been thought of as a continuous time analogue of GD and is popularly used to understandbehavior of gradient based optimization algorithms in the limit, primarily due to its simplicity andlack of step size. 
Revised: GF has been thought of as a continuous time analogue of GD and is popularly used to understandbehavior of gradient based optimization algorithms in the limit, primarily due to its simplicity andlack of step size. Additional notation. For a vector w ∈

Original: Gradient Flow, Potentials and Geometry
Revised: R d , w [ j ] denotes its j -th coordinate and ￿ w ￿ denotes its Euclidean norm. For any w 1 ,w∈

Revised: R d , ￿ w 1 ,w￿ denotes their inner product.

Revised: For a matrix W , min (

Revised: W ) and ￿

Revised: M ￿ denotes its minimum singular value and spectral norm respectively.

Revised: We use 1 d todenote a d -dimensional vector of all 1 s, and I d to denote the identity matrix in d -dimensions.

Revised: N ( 0 , ) denotes Gaussian distribution with variance  2 , and Ber ( p ) denotes the Bernoulli distribution withmean p .

Revised: We deﬁne the set R + to contain all non-negative real numbers.

Original: Lyapunov potentials are a popular tool for understanding convergence of GF [43, 45, 29].
Revised: For a function f ∶ R d × R , we denote the p -th derivate at the point w by ∇ f ( w ) ∈ R d .

Original: At an intuitive level, a Lyapunov potential is any non-negative function  that satisﬁes
Revised: The function fis said to be L -Lipschitz if f ( w) − f ( w) ≤ L ￿ w− w￿ for all w 1 ,w 2 .

Original: [Equation]
Revised: For a set

Original: ∇  [Equation] w 
Revised: W , we denoteclo ( W ) as its closure under GF, i.e. clo ( W ) = { w ′ ∶ w ′ is in GF path of some w∈ W } .

Original:  [Equation]
Revised:  Gradient Flow, Potentials and Geometry

Original:  −∇F [Equation]
Revised: Lyapunov potentials are a popular tool for in understanding convergence of GF [40, 42, 28]. At an intuitive level, a Lyapunov potential is any non-negative function  that satisﬁes ￿ ∇  ( w ) , −∇

Original:  w ) [Equation] 0 ,i.e.  decreases along the GF paths of F .
Revised: F ( w ) ￿ ≤ 0 ,i.e.  decreases along the GF paths of F .

Original: This monotonicity property helps to show asymptoticconvergence of GF to stable points of the underlying objective.
Revised: This monotonicity property helps to show asymptoticconvergence of GF to stable points of the underlying objective.

Original: In our work, we consider potentialfunctions for which the rate of change (decrease) of the potential along the GF path is related to thesuboptimality of the objective at that point.
Revised: In our work, we consider potentialfunctions for which the rate of change (decrease) of the potential along the GF path is related to thesuboptimality of the objective at that point.

Original: Deﬁnition 1 (Admissible potentials) .
Revised: Deﬁnition 1 (Admissible potentials) .

Original: A differentiable potential function  g ∶
Revised: A differentiable potential function  g ∶

Original: R d ￿
Revised: R d ￿

Original: R + is admissiblew.r.t.
Revised: R + is admissiblew.r.t.

Original: F on a set
Revised: F on a set

Original: W if there exists a monotonically increasing function g∶
Revised: W if there exists a non-decreasing function g

Original: R + ￿ R + with g ( ) = 0such
Revised: [Equation]

Original:  that for any w ∈ W ,[-
-]
Revised: such that for any w ∈ W ,[-
-]

Original: [Equation]
Revised: [Equation]

Original: Existence of potential functions of the above form can be used to provide rates of convergence for GFas show in the following theorem.
Revised: Existence of potential functions of the above form can be used to provide rates of convergence for GFas show in the following theorem.

Original: Theorem 1 (From potentials to gradient ﬂow) .
Revised: Theorem 1 (From potentials to gradient ﬂow) .

Original: Let
Revised: Let

Original: W be a set of initial points that we want to consider,and let  g be an admissible potential w.r.t.
Revised: W be a set of initial points that we want to consider,and let  g be an admissible potential w.r.t.

Original: F on the set clo ( W ) .
Revised: F on the set clo ( W ) .

Original: Then, for any initialization w∈ W ,the point w ( t ) on the GF path with w ( ) = w 0 satisﬁes for any t ≥
Revised: Then, for any initialization w∈ W ,the point w ( t ) on the GF path with w ( [Equation] = w 0 satisﬁes for any t ≥

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The idea that admissible potential functions imply convergence rates for GF has appeared in variousforms in the prior literature [5, 28, 43, 45].
Revised: The idea that admissible potential functions imply convergence rates for GF has appeared in variousforms in the prior literature [5, 27, 40, 42].

Original: As an example, consider the potential function  ( w ) =
Revised: As an example, consider the potential function  ( w ) =

Original: ￿ w − w ∗ ￿￿ 2 .
Revised: ￿ w − w ∗ ￿￿ 2 .

Original: Notice that  is an admissible potential for any F that is convex with g ( z )
Revised: Notice that  is an admissible potential for any F that is convex with g ( z )

Original: =
Revised: =

Original: z .
Revised: z .

Original: This isbecause convexity implies that (4) is true for any w .
Revised: This isbecause convexity implies that (4) is true for any w .

Original: Hence, using Theorem 1 we get a ￿ w − w ∗ ￿ ￿ 2 trate of convergence for GF on any convex objective.
Revised: Hence, using Theorem 1 we get a ￿ w − w ∗ ￿ ￿ 2 trate of convergence for GF on any convex objective.

Original: Our main result in this section is to establish a converse Lyapunov style theorem − that given a rate,ﬁnds a potential function corresponding to that rate.
Revised: Our main result in this section is to establish a converse Lyapunov style theorem − that given a rate,ﬁnds a potential function corresponding to that rate.

Original: We start by deﬁning admissible rate functions.
Revised: We start by deﬁning admissible rate functions.

Original: Deﬁnition 2 (Admissible rate functions) .
Revised: Deﬁnition 2 (Admissible rate functions) .

Original: A function R ∶
Revised: A function R ∶

Original: R d ×
Revised: R d ×

Original: R ￿
Revised: R ￿

Original: R + is an admissible rate functionw.r.t.
Revised: R + is an admissible rate functionw.r.t.

Original: F if for any w ∈
Revised: a path w ∶

Original: R  d , [Equation] ( a )
Revised: R + ￿ R d if,( a )

Revised: For any w ′ ∈

Original: R  ( w,t ) is a non-increasing function of t such that lim t → ∞
Revised: R d , R ( w ′ ,t ) is a non-increasing function of t such that lim t → ∞

Original: R ( w,t ) = 0 .
Revised: R ( w ′ ,t ) = 0 .

Original: ( b )
Revised: ( b )

Original: R satisﬁes the relation: ∫ ∞ t = ￿ @

Original: R ( w,t ) @ t + ￿ ∇ R (w,t ) , ∇ F ( w  ) ￿￿ d t ≥ 0 .
Revised: R satisﬁes R ( w ( s ) ,t ) ≤

Original: Remark 1.

Original: In order to simplify the task of checking whether a given rate is admissible, note that

Original: Deﬁnition 2-(b) is satisﬁed whenever the condition @

Original: R ( w,t ) @ t + ￿ ∇ R (w,t ) , ∇ F ( w )￿ ≥ 0 holds for everyw,t ≥ 0 .
Revised: R ( w ( ) ,s + t ) for any s,t ≥ 0 .

Original: Many rate functions, e.g. R ( w,t ) =

Original: F ( w ) e − t and F being K Ł , in fact satisfy this conditionfor every w,t ≥ 0 .

Original: Furthermore, also note that Deﬁnition 2-(b) is satisﬁed whenever the rate function is such that

Original: R ( w ( s ) ,t ) ≤

Original: R ( w,s + t ) for all s,t ≥ 0 and w ∈

Original: R d , which may be an easier to check condition, e.g.when R ( w,t ) =

Original: F ( w ) e − t .

Original: We utilize admissible rate functions to characterize behavior of GF on F .
Revised: We utilize admissible rate functions to characterize behavior of GF on F .

Original: Before we proceed, letus motivate the two properties above.
Revised: Before we proceed, letus motivate the two properties above.

Original: Property ( a ) is natural for any rate function and captures thefact that running GF for more time leads to better guarantees.
Revised: Property ( a ) is natural for any rate function and captures thefact that running GF for more time leads to better guarantees.

Original: The property ( b ) , while seeming a bitmysterious, captures the fact that having additional information about the GF path should only improvethe rate.
Revised: The property ( b ) , while seeming a bitmysterious, captures the fact that having additional information about the GF path should only improvethe rate.

Original: Note that R ( w ( ) ,s + t ) corresponds to an upper bound on the sub-optimality at w ( s + t ) and
Revised: Note that R ( w ( ) ,s + t ) corresponds to an upper bound on the sub-optimality at w ( s + t ) and

Original: R ( w ( s ) ,t ) corresponds to an upper bound on the same quantity but with the additional informationthat w ( s ) is a point on the GF path.
Revised: R ( w ( s ) ,t ) corresponds to an upper bound on the same quantity but with the additional informationthat w ( s ) is a point on the GF path.

Original: Furthermore, given any rate function R , it is easy to constructa new rate function ¯ R that always satisﬁes property ( b ) by deﬁning ¯ R ( w,t )
Revised: Furthermore, given any rate function R , it is easy to constructa new rate function ¯ R that always satisﬁes property ( b ) by deﬁning ¯ R ( w,t )

Original: =
Revised: =

Original: min s ≥ 0 R ( w ′ ,t + s ) where w ′ is any point suchthat the point w lies on the GF path from w ′ at time s .
Revised: min s ≥ 0 R ( w ′ ,t − s ) where w ′ is any point such the point w lies on the GF path from w ′ at time s .

Original: We also remarkthat the function R ( [Equation] ) =
Revised: We also remark thatthe function R ( w,t ) =

Original: [Equation] t )) is always an admissible rate function.
Revised: F ( w ( t )) is always an admissible rate function.

Original: Furthermore, all the ratefunctions appearing in this paper satisfy the above properties.
Revised: Furthermore, all the ratefunctions appearing in this paper satisfy the above properties.

Original: Our next result shows how existence of rate functions for GF that satisfy the above properties can beused to construction admissible potentials w.r.t.
Revised: Our next result shows how existence of rate functions for GF that satisfy the above properties can beused to construction admissible potentials w.r.t.

Original: F .
Revised: F .

Original: Theorem 2 (From gradient ﬂow to potentials) .
Revised: Theorem 2 (From gradient ﬂow to potentials) .

Original: Let
Revised: Let

Original: W ⊆
Revised: W ⊆

Original: R d be any set of initial points that we wantto consider, and R be an admissible rate function w.r.t.
Revised: R d be any set of initial points that we wantto consider, and R be an admissible rate function w.r.t.

Original: all GF paths originating from any point in
Revised: all GF paths originating from any point in

Original: W .
Revised: W .

Original: Further, suppose that for any w∈ W , the point w ( t ) on the GF path satisﬁes F ( w ( t )) ≤
Revised: Further, suppose that for any w∈ W , the point w ( t ) on the GF path satisﬁes F ( w ( t )) ≤

Original: R ( w 0 ,t ) ,then the function  g deﬁned as
Revised: R ( w 0 ,t ) ,then the function  g deﬁned as g ( w ) =

Original: [Equation]{+￿ ∞ t = 
 g ( R ( w,t ))+}
Revised: {+￿ ∞ t = 
 g ( R ( w,t ))+}

Original:  (5) is an admissible potential w.r.t.
Revised: d t (5) is an admissible potential w.r.t.

Original: F on the set clo ( W ) , for any differentiable and monotonically increasing function g 
Revised: F on the set clo ( W ) , for any differentiable and non-decreasingfunction g ∶

Original: [Equation]
Revised: R + ￿

Original:  + that satisﬁes ∫ ∞ t = 

Revised: R + that satisﬁes ∫ ∞ t =

Original: g (
Revised: g (

Original: R ( w,t ))
Revised: R ( w,t ))

Original: d t < ∞ and ∫ ∞ t = 

Revised: d t < ∞ and ∫ ∞ t =

Original: g ′ (
Revised: g ′ (

Original: R ( w,t ))
Revised: R ( w,t ))

Original: ￿ ∇
Revised: ￿ ∇

Original: R ( w,t ) ￿ d t < ∞  for every w ∈ clo ( W ) .
Revised: R ( w,t ) ￿ d t < ∞ 154 for every w ∈ clo ( W ) .

Original: As an illustration on how to apply Theorem 2, assume that for F the rate for GF is R ( w,t ) =
Revised: As an illustration on how to apply Theorem 2, assume that for F the rate for GF is R ( w,t ) =

Original: ￿ w − w ∗ ￿ ￿ 2 t .
Revised: ￿ w − w ∗ ￿ ￿ 2 t .

Original: For instance, we already know that such a rate holds when F is convex.
Revised: For instance, we already know that such a rate holds when F is convex.

Original: For this rate,by choosing g ( z ) =
Revised: For this rate,by choosing g ( z ) =

Original: ( + ￿ z ) − 2 , we get that the function  g ( w ) = ∫ ∞ t =
Revised: ( + ￿ z ) − 2 , we get that the function  g ( w ) = ∫ ∞ t =

Original: ￿ + 2 t ￿￿ w − w ∗
Revised: ￿ + 2 t ￿￿ w − w ∗

Original: ￿ ￿ − 2 d t = ￿ w − w ∗
Revised: ￿ ￿ − 2 d t = ￿ w − w ∗

Original: ￿ ￿ 2 is an admissible potential w.r.t.
Revised: ￿ ￿ 2 is an admissible potential w.r.t.

Original: F .
Revised: F .

Original: We provide more examples in Section 5.
Revised: We provide more examples in Section 5.

Original: Theorem 1 and Theorem 2 are, in a sense, converse of each other.
Revised: Theorem 1 and Theorem 2 are, in a sense, converse of each other.

Original: Theorem 1 shows that the existenceof an admissible potential function implies a rate of convergence for GF.
Revised: Theorem 1 shows that the existenceof an admissible potential function implies a rate of convergence for GF.

Original: On the other hand, Theorem 2shows how to construct admissible potentials starting from the fact that GF has a rate.
Revised: On the other hand, Theorem 2shows how to construct admissible potentials starting from the fact that GF has a rate.

Original: One mightwonder whether there always exist a Lyapunov function, more speciﬁcally a g function above, suchthat the rate implied by the constructed potential in Theorem 1 matches the rate that we started withfor Theorem 2, i.e. R ( w,t )
Revised: One mightwonder whether there always exist a Lyapunov function, more speciﬁcally a g function above, suchthat the rate implied by the constructed potential in Theorem 1 matches the rate that we started withfor Theorem 2, i.e. R ( w,t )

Original: ≈ g − (  g ( w ) ￿ t ) .
Revised: ≈ g − (  g ( w ) ￿ t ) .

Original: We answer this in the positive for rate functions thatare of the product form.
Revised: We answer this in the positive for rate functions thatare of the product form.

Original: Corollary 1.
Revised: Corollary 1.

Original: Let
Revised: Let

Original: W ⊆
Revised: W ⊆

Original: R d be any set of initial points that we want to consider, and R be an admissiblerate function w.r.t.
Revised: R d be any set of initial points that we want to consider, and R be an admissiblerate function w.r.t.

Original: all GF paths originating from points in
Revised: all GF paths originating from points in

Original: W .
Revised: W .

Original: Additionally, suppose R has the productform R ( w,t )
Revised: Additionally, suppose R has theproduct form R ( w,t )

Original: = h ( w ) r ( t ) where h is differentiable and r is a non-increasing function 
Revised: = h ( w ) r ( t ) where h is differentiable and r is a non-increasing function thatsatisﬁes r ( s ) ≤  ￿ r

Original: that satisﬁesr ( t ) ≤  ￿ r ′ ( t ) ￿ max { 1 ,t } for any t ∈
Revised: [Equation] s [Equation] ￿ max { 1 ,s } for any s ∈

Original: R (where  is a universal constant).
Revised: R (where  is a universal constant).

Original: Furthermore, supposethat for any w∈ W , the point w ( t ) on the GF path satisﬁes F ( w ( t )) ≤
Revised: Furthermore,suppose that for any w∈ W , the point w ( t ) on the GF path satisﬁes F ( w ( t )) ≤

Original: R ( w 0 ,t ) .
Revised: R ( w 0 ,t ) .

Original: Then, thereexists a monotonically increasing function g ∶
Revised: Then,there exists a non-decreasing function g ∶

Original: R + ￿
Revised: R + ￿

Original: R + such that the potential  ( w ) constructed in
Revised: R + such that the potential  ( w ) constructed in

Original: Theorem 2 using g , when plugged in Theorem 1, implies that GF has the rate
Revised: Theorem 2 using g , when plugged in Theorem 1, implies that GF has the rate

Original: [Equation]
Revised: [Equation]

Original: ) for anyinitialization w ( ) ∈ W .
Revised: ) for any w [Equation] [Equation] W .

Original: 3.1 Geometric Interpretation
Revised: 3.1 Geometric Interpretation

Original: [Figure] The deﬁnition of an admissible potential comes with a geometric condition on the function F given in(4).
Revised: The deﬁnition of an admissible potential comes with a geometric condition on the function F given in(4).

Original: Since Theorem 2 constructs admissible potentials, when a rate R ( w,t ) holds for GF it suggeststhat the geometric property in (4) holds for the objective function F .
Revised: Since Theorem 2 constructs admissible potentials, when a rate R ( w,t ) holds for GF it suggests that the geometric property in (4) holds for the objective function F .

Original: As an example, say GF on Fsatisﬁes the rate R ( w,t ) =
Revised: As an example, say GF on Fsatisﬁes the rate R ( w,t ) =

Original: F ( w) e −  t . 
Revised: ￿ w − w ∗ ￿ ￿ 2 t . Using Theorem 2 with g ( z ) = ( ✏ − ￿ ✏ + ￿ z ) − − ✏ for any✏ > 0 , the function  ( w ) =

Original: From Theorem 2, we note that  g ( w ) = F ( w ) is an admissiblepotential w.r.t F with g ( z )
Revised: ￿ w − w ∗ ￿ ￿ 2 is an admissible potential w.r.t. F (for that g) and impliesthe following geometric property

Original: = z .
Revised: [Equation]

Original: This implies the geometric property￿ ∇
Revised: [Equation] ✏ ⋅ [Equation] − ￿ ✏ F

Original: F ( w) , ∇
Revised: [Equation] w

Original: F ( w ) ￿ ≥
Revised: [Equation] ) − − ✏ . Since the above holds for any ✏ > 0 , taking limit of ✏ → 0 implies that the geometric property [Equation]

Original: F ( w ) (6) holds for F whenever GF has rate R ( w,t ) =
Revised: [Equation] holds for F whenever GF has rate R ( w,t ) =

Original: F ( w) e −  t .
Revised: ￿ w − w ∗ ￿ ￿ 2 t .

Original: On the other, we know that whenever(6) holds the function ( w ) =
Revised: On the other, we know that whenever(6) holds the function

Original: F ( w ) satisﬁes (4) and is thus an admissible potential for F (withg ( z ) = z ), and hence Theorem 1 implies the rate ofF ( w ) ￿t , which is equivalent to the rate F ( w) e − t(c.f.
Revised: [Equation] 2 satisﬁes (4) and is thus an admissible potential for F (withg ( z ) = z ), and hence Theorem 1 implies the rate of ￿ w − w [Equation]

Revised: ￿

Original: Lemma 7).
Revised: [Equation]

Revised: ￿ 2 t .

Original: This implies an equivalence between the rates R( w,t ) =
Revised: This implies that there is anequivalence between the rates R

Original: F ( w ) e − t and the geometricproperty (6).
Revised: [Equation] 2 t and the geometric property (6).

Revised: Notice thatthe condition (6) is weaker than convexity but is sufﬁcient for the corresponding rate of convergencefor GF.

Original: We formalize this in the following.
Revised: We formalize this in the following lemma.

Original: Proposition 1.
Revised: Proposition 1.

Original: The following two properties are equivalent:
Revised: The following two properties are equivalent:(a)

Original: (a) For any w ( ) ∈R d and t ≥ 0 , GF has the rate F ( w ( t )) ≤ F ( w ( )) e − 
Revised: For any w ( ) ∈

Original: [Equation] (b) F ( w  )  satisﬁes the Polyak- Ł ojasiewicz (PL) property i.e.   F ( w [Equation]
Revised: R d and t ≥ 0 , the point w ( t ) on its GF path satisﬁes F ( w ( t )) ≤  ￿ w ( ) − w ∗ ￿ 2 t ,(b) F ( w ) is linearizable w.r.t.

Original: ￿

Original: [Equation] F [Equation] w 
Revised: w ∗ i.e. F ( w ) ≤  ￿ ∇

Original: [Equation] 2 ,for any  ≥ 0 .
Revised: F ( w ) ,w − w ∗ ￿ ,for any  ≥ 0 .

Original: Theorem 2 implies ( b ) and yields the potential function  [Equation]
Revised: Theorem 2 implies ( b ) and yields the potential function  ( w ) =

Original:  w [Equation] w [Equation]
Revised: ￿ w − w ∗ ￿ ￿ 2 .

Original: In the following, we  show a correspondence between the rate R ( w,t ) =
Revised: Similarly, we also show a correspondence between the rate R ( w,t ) =

Original: ￿ w () − w∗ ￿ − ￿ w ( t )  −w ∗ ￿ 2 t , andlinearizability—a condition that is weaker than convexity but is sufﬁcient for  the corresponding rateof convergence for GF.
Revised: F ( w ) e − t and PolyakLojasiewicz (PL) property for F in the following lemma.

Original: Proposition 2.
Revised: Proposition 2.

Original: The following two properties are equivalent:
Revised: The following two properties are equivalent:(a)

Original: For any  [Equation]
Revised: For any w ( ) ∈

Original: a) For any w ( ) ∈ R d and t ≥ 0 ,GF has the admissible rate F 
Revised: R d and t ≥ 0 , the point w ( t ) on its GF path satisﬁes F ( w ( t )) ≤

Original: [Equation]
Revised: F ( w ( ))

Original: ￿ w ( ) − w ∗ ￿ − ￿ w ( t  ) − w ∗ ￿ 2 t , [Equation]
Revised: e −  t ,(b) F ( w ) satisﬁes the Polyak-Lojasiewicz (PL) property i.e. F ( w ) ≤ 1  ￿ ∇

Original: More generally, the equivalence between GF rates and  the corresponding geometry on F can becharacterized as follows.
Revised: F ( w ) ￿ 2 ,for any  ≥ 0 . Theorem 2 implies ( b ) and yields the potential function  ( w ) = F ( w ) .

Original: Remark 2.
Revised: Remark 1.

Original: GF on F enjoys theadmissible rate R ( 
Revised: GF on F enjoys the rate R ( w,t ) =

Original: [Equation] ￿ t ) if and only if F has thegeometric property
Revised: g − (  g ( w ) ￿ t ) if and only if F has the geometricproperty ￿ ∇  g ( w ) , ∇ F ( w ) ￿ ≥ g (

Original: [Equation]
Revised: F ( w )) .

Original:  Stochastic Gradient Descent and Gradient Descent
Revised:  Stochastic Gradient Descent and Gradient Descent

Original: GD can be thought of as an approximate discretization of gradient ﬂow.
Revised: GD can be thought of as an approximate discretization of gradient ﬂow.

Original: Thus, for problems where GFconverges with a given rate R , one may try to get convergence guarantees for GD from an initial pointw 0 by bounding the distance between the GD and GF trajectories starting from w 0 .
Revised: Thus, for problems where GFconverges with a given rate R , one may try to get convergence guarantees for GD from an initial pointw 0 by bounding the distance between the GD and GF trajectories starting from w 0 .

Original: This is exactly theapproach taken in prior works [19, 30, 44, 40, 48, 14].
Revised: This is exactly theapproach taken in prior works [18, 29, 41, 38, 45, 14].

Original: However, coming up with non-vacuous boundson the distance between corresponding GF and GD iterated is often quite challenging and requiresmuch stronger assumptions on the underlying objective.
Revised: However, coming up with non-vacuous boundson the distance between corresponding GF and GD iterated is often quite challenging and requiresmuch stronger assumptions on the underlying objective.

Original: In fact there are cases where both GF and
Revised: In fact there are cases where both GF and

Original: GD converge to the same global minimum but their paths can be quite far away from each other.
Revised: GD converge to the same global minimum but their paths can be quite far away from each other.

Original: Wetake a different approach for proving convergence of GD/SGD which directly relies on the propertiesof corresponding potential for F .
Revised: Wetake a different approach for proving convergence of GD/SGD which directly relies on the propertiesof corresponding potential for F .

Original: In the following theorem, we note that further assumption on top ofthe premise that GF has a rate are required, to even hope that GD succeeds.
Revised: In the following theorem, we note that further assumption on top ofthe premise that GF has a rate are required, to even hope that GD succeeds.

Original: Theorem 3.
Revised: Theorem 3.

Original: For any integer T> 0 , there exists a continuously differentiable convex function F forwhich min w F 
Revised: There exists a continuously differentiable convex function F for which min w F ( w ) = 0and w ∗ = 0 is the unique minimizer, such that:

Original: [Equation]
Revised: [Equation]

Original: and w
Revised: a)

Original: [Equation]
Revised: [Equation]

Original: 0 is the unique minimizer, such that:

Original: (a)  ( w ) =
Revised: 2 is an

Original: ￿ w ￿ ￿ 2 is an admissible potentialfor F . Thus, Theorem implies
Revised: admissible potential

Original: that
Revised: for

Original:  for any initial point w0 , the point w( t ) on its GF path satisﬁes F (
Revised: F . Thus, Theorem 1 implies that for any initialpoint w the point w

Original: [Equation]
Revised: [Equation]

Revised: on its GF

Revised: path

Revised: satisﬁes F

Original: [Equation]
Revised: [Equation]

Original: 2 t . (b) There exists an initial point w 0 with ￿ w￿ ≤ 1 and F ( w) ≤ 2 such that GD fails to ﬁnd
Revised: (b) There exists an initial point w 0 with ￿ w￿ ≤ 1 and F ( w) ≤ 2 such that GD fails to ﬁnd

Original: an
Revised: a

Original: ￿ 10 -suboptimal solution for any step size ⌘ within t ≤ T 0 steps.
Revised: ￿ 10 -suboptimal solution for any step size ⌘ .

Original: Below we present two regularity conditions that are sufﬁcient to show convergence of GD, given aprescribed rate for GF.
Revised: Below we present two regularity conditions that are sufﬁcient to show a

Original: Assumption 1.
Revised: Assumption 1.

Original: There exists a monotonically increasing function ∶
Revised: There exists a non-decreasing function ∶

Original: R + ￿
Revised: R + ￿

Original: R + such that ￿ ∇
Revised: R + such that ￿ ∇

Original: F ( w ) ￿ ≤ (
Revised: F ( w ) ￿ ≤ (

Original: F ( w ))
Revised: F ( w ))

Original: for any point w ∈ W .
Revised: for any point w ∈ W .

Original: Assumption 2.
Revised: Assumption 2.

Original: The potential function  is second-order differentiable, and there exists a monotonically increasing function ⇢ ∶
Revised: The potential function  is second-order differentiable, and there exists a nondecreasing function ⇢ ∶

Original: R + ￿
Revised: R + ￿

Original: R + such that ￿ ∇ 2  ( w ) ￿ ≤ ⇢ (  ( w )) at any point w ∈ W .
Revised: R + such that ￿ ∇ 2  ( w ) ￿ ≤ ⇢ (  ( w )) at any point w ∈ W .

Original: We will refer to the above conditions on F and  as self-bounding regularity conditions.
Revised: We will refer to the above conditions on F and  as self-bounding regularity conditions.

Original: Thefollowing theorem provides convergence guarantees for GD when an admissible potential exists andthe above assumptions are satisﬁes.
Revised: Thefollowing theorem provides convergence guarantees for GD when an admissible potential exists andthe above assumptions are satisﬁes.

Original: Theorem 4 (GD convergence guarantee) .
Revised: Theorem 4 (GD convergence guarantee) .

Original: Let  g be an admissible potential w.r.t.
Revised: Let  g be an admissible potential w.r.t.

Original: F . Assume that Fsatisﬁes Assumption 1 and  g satisﬁes Assumption 2.
Revised: F . Assume that Fsatisﬁes Assumption 1 and  g satisﬁes Assumption 2.

Original: Then, for any T ≥ 0 and setting ⌘ appropriately,the point ̂ w T returned by GD algorithm has the convergence guarantee 

Revised: Then, for any T ≥ 0 and setting ⌘ appropriately,the point ̂ w T returned by GD algorithm has the convergence guarantee

Original: [Equation]
Revised: [Equation]

Original: Furthermore, if the function g is monotonically increasing, then for a different appropriate choice of⌘ ,g ( 
Revised: Furthermore, if the function g is non-decreasing, then for a different appropriate choice of ⌘ ,g ( F ( ̂

Original: [Equation]
Revised: [Equation]

Original: ( ̂
Revised: =

Revised: O ￿

Original: [Equation]
Revised: [Equation]

Original: Let us consider an example.
Revised: Let us consider an example.

Original: Suppose GF on F achieves the rate R ( w,t )
Revised: Suppose GF on F achieves the rate R ( w,t )

Original: =
Revised: =

Original: ￿ w − w ∗ ￿ ￿ 2 t .
Revised: ￿ w − w ∗ ￿ ￿ 2 t .

Original: Then, g ( w ) =
Revised: Then, g ( w ) =

Original: ￿ w − w ∗ ￿ ￿ 2 is an admissible potential for F from Theorem 2 with g ( z ) =
Revised: ￿ w − w ∗ ￿ ￿ 2 is an admissible potential for F from Theorem 2 with g ( z ) =

Original: ( ✏ − ￿ ✏ + ￿ z ) − − ✏for any ✏ > 0 .
Revised: ( ✏ − ￿ ✏ + ￿ z ) − − ✏for any ✏ > 0 .

Original: However, as we saw in Theorem 3 just existence of such a rate function does notimply the GD will succeed and we need to make further assumptions.
Revised: However, as we saw in Theorem 3 just existence of such a rate function does notimply the GD will succeed and we need to make further assumptions.

Original: Notice that in this case  g ( w ) satisﬁes Assumption 2 with ⇢ ( z ) = 1 .
Revised: Notice that in this case  g ( w ) satisﬁes Assumption 2 with ⇢ ( z ) = 1 .

Original: If we further assume that F is L -Lipschitz, then Assumption 1is satisﬁed with ( z ) =
Revised: If we further assume that F is L -Lipschitz, then Assumption 1is satisﬁed with ( z ) =

Original: L 2 .
Revised: L 2 .

Original: Hence, applying Theorem 4 and taking the limit of ✏ → 0 , we get that
Revised: Hence, applying Theorem 4 and taking the limit of ✏ → 0 , we get that

Original: GD has convergence rate F ( ̂ w T ) =
Revised: GD has convergence rate F ( ̂ w T ) =

Original: O ( ￿ w − w ∗ ￿
Revised: O ( ￿ w − w ∗ ￿

Original: L ￿√
Revised: L ￿√

Original: T ) .
Revised: T ) .

Original: Instead if F was H -smooth, Assumption 1is satisﬁed with ( z ) = 4 Hz and ( z ) ￿ g
Revised: Instead if F was H -smooth, Assumption 1is satisﬁed with ( z ) = 4 Hz and ( z ) ￿ g

Original: ( z ) =
Revised: ( z ) =

Original: 4 H is a monotonically increasing function and thususing (8), we get that GD has the convergence rate F ( ̂ w T ) =
Revised: 4 H is a non-decreasing function and thus using(8), we get that GD has the convergence rate F ( ̂ w T ) =

Original: O (
Revised: O (

Original: H ￿ w − w ∗
Revised: H ￿ w − w ∗

Original: ￿ ￿
Revised: ￿ ￿

Original: T ) .
Revised: T ) .

Original: Notice that bothof these rates are optimal for GD under the Lipschitz/Smoothness assumptions on F and given thefact that GF converges at the rate R starting from w [36].
Revised: Notice that both ofthese rates are optimal for GD under the Lipschitz/Smoothness assumptions on F and given thefact that GF converges at the rate R starting from w [35].

Original: On similar lines, using the rates for GFconvergence on PL/K Ł functions, we can also recover optimal convergence rates for GD underappropriate smoothness assumptions on F .
Revised: On similar lines, using the rates for GFconvergence on PL/KL functions, we can also recover optimal convergence rates for GD underappropriate smoothness assumptions on F .

Original: Details are deferred to the appendix.
Revised: Details are deferred to the appendix.

Original: We next consider the convergence of SGD algorithm.
Revised: We next consider the convergence of SGD algorithm.

Original: Recall that at the iterate w t , SGD performsthe updated using ∇ f ( w t ,z t ) , a stochastic and unbiased estimate of ∇
Revised: Recall that at the iterate w t , SGD performsthe updated using ∇ f ( w t ,z t ) , a stochastic and unbiased estimate of ∇

Original: F ( w t ) .
Revised: F ( w t ) .

Original: Of course, unlessone has some form of control over the distribution of ∇ f ( w t ,z t ) , one cannot hope to prove anyconvergence guarantees of SGD.
Revised: Of course, unlessone has some form of control over the distribution of ∇ f ( w t ,z t ) , one cannot hope to prove anyconvergence guarantees of SGD.

Original: To this end, we make the following regularity assumption on thenoise in ∇ f ( w,z t ) while estimating ∇
Revised: To this end, we make the following regularity assumption on thenoise in ∇ f ( w,z t ) while estimating ∇

Original: F ( w ) .
Revised: F ( w ) .

Original: Assumption 3 (Noise regularity) .
Revised: Assumption 3 (Noise regularity) .

Original: There exists a monotonically increasing function  ∶
Revised: There exists a non-decreasing function  ∶

Original: R + ￿
Revised: R + ￿

Original: R +249 such that for any point w , the gradient estimate ∇
Revised: R + such that forany point w , the gradient estimate ∇

Original: [Equation]
Revised: [Equation]

Original: satisﬁes
Revised: satisﬁes

Original: Pr

Original: [Equation]
Revised: [Equation]

Original: F

Original: [Equation]

Original: w

Original: [Equation] e [Equation] Assumption 3 is quite general, and can be specialized by appropriately setting the function  to modelvarious stochastic optimization problem settings observed in practice.
Revised: Assumption 3 is quite general, and can be specialized by appropriately setting the function  to modelvarious stochastic optimization problem settings observed in practice.

Original: For example, the classicalstochastic optimization setting in which
Revised: For example, the classicalstochastic optimization setting in which

Original: [Equation]
Revised: [Equation]

Original: " t where " t is a sub-Gaussian randomvariable with mean 0 and variance  2 is captured by the above assumption when  (z ) =
Revised: t where " t is a sub-Gaussian randomvariable with mean 0 and variance  2 is captured by the above assumption when  (

Original:  2 [35].
Revised: [Equation]

Original: However, it turns out that for many interesting ML problems, the noise typically scales with thefunction value [46, 47].
Revised: However, it turns out that for many interesting ML problems, the noise typically scales with thefunction value [43, 44].

Original: Theorem 5 (SGD convergence guarantee) .
Revised: Theorem 5 (SGD convergence guarantee) .

Original: Let  g be an admissible potential w.r.t.
Revised: Let  g be an admissible potential w.r.t.

Original: F . Assume that
Revised: F . Assume that

Original: F satisﬁes Assumption 1,  g satisﬁes Assumption 2 and the stochastic gradient estimates ∇ f ( w ; z ) 258 satisfy Assumption 3.
Revised: F satisﬁes Assumption 1,  g satisﬁes Assumption 2 and the stochastic gradient estimates ∇ f ( w [Equation] ) 258 satisfy Assumption 3.

Original: Then, for any T ≥ 0 and setting ⌘ appropriately, the point ̂ w T returned by SGDalgorithm has the convergence guarantee 1g ({+
+}
Revised: Then, for any T ≥ 0 and setting ⌘ appropriately, the point ̂ w T returned by SGDalgorithm has the convergence guarantee {+
+}

Original: F ( ̂ w T )) = ̃ O ￿ √ T ￿ .
Revised: [Equation]

Original: with probability at least 0 .
Revised: with probability at least 0 .

Original: 7 over the randomization of the algorithm and stochastic gradients.
Revised: 7 over the randomization of the algorithm and stochastic gradients.

Original: Remark 3.
Revised: Remark 2.

Original: In most classic settings, one expects a￿√
Revised: In most classic settings, one expects a￿√

Original: T rate for SGD [6].
Revised: T rate for SGD [6].

Original: However, in cases where g is an admissible potential and g ( z )
Revised: However, in cases where g is an admissible potential and g ( z )

Original: =
Revised: =

Original: o ( z ) , Theorem 5 seems to suggest a g − ( ￿√
Revised: o ( z ) , Theorem 5 seems to suggest a g − ( ￿√

Original: T ) rate ofconvergence which is faster than￿√
Revised: T ) rate ofconvergence which is faster than￿√

Original: T .
Revised: T .

Original: This is where the self-bounding regularity conditions playan important role.
Revised: This is where the self-bounding regularity conditions playan important role.

Original: As an example for P Ł style rates, one can show that F ( w ) p is an admissiblepotential with g ( z ) =
Revised: As an example for PL style rates, one can show that F ( w ) p is an admissiblepotential with g ( z ) =

Original: z p for any p .
Revised: z p for any p .

Original: However, the self-regularity conditions are not satisﬁed unlessp ≥ 1 .
Revised: However, the self-regularity conditions are not satisﬁed unlessp ≥ 1 .

Original: Setting p = 1 recovers the￿√
Revised: Setting p = 1 recovers the￿√

Original: T rate of SGD for P Ł functions which is optimal [1].
Revised: T rate of SGD for PL functions which is optimal [1].

Original:  Examples: From Gradient Flow to Gradient Descent
Revised:  Examples: From Gradient Flow to Gradient Descent

Original: So far, we have already discussed classical examples like P Ł functions, convex functions, etc.
Revised: So far, we have already discussed classical examples like PL functions, convex functions, etc where

Revised: GF converges from any starting point.

Revised: However, in many applications, GF is known to only convergefrom nice enough initial points that satisfy certain properties.

Original: In thissection, we show the versatility of our framework by discussing more complex non-convex losseslike phase retrieval and matrix square root.
Revised: In this section, we show how to use ourtools to show convergence of GD/SGD for such problems, and consider phase retrieval and matrixsquare root as examples. We ﬁrst provide the following general utility lemma that shows how toconstruct admissible potentials when the rate for GF from w 0 holds only when w 0 satisﬁes a certainproperty characterized by h ( w ) ≥ 0 . Lemma 1. Let h ∶

Revised: R d ￿ [ 0 ,] be a continuously differentiable function, and suppose that for anypoint w such that h ( w ) > 0 , GF has rate F ( w ( t )) ≤

Revised: R ( w,t ) where w ( ) =

Original: 5.1 Phase retrieval
Revised: w and R is an admissiblerate function. Furthermore, suppose that F satisﬁes Assumption 1 and( a ) The function [Equation]

Revised: ∞ t = 0 R ( w,t ) d t is continuously differentiable andmax { ￿ ∇  ( w ) ￿ , ￿ ∇ 2  ( w ) ￿ } ≤  (  ( w )) and  is a non-decreasing, positive function.

Revised: ( b )

Revised: For any w , max {( h ( w ) − h ( w ∗ )) 2 , ￿ ∇ h ( w ) ￿ , ￿ ∇ 2 h ( w ) ￿ } ≤ ⇡ (  ( w )) where ⇡ is a nondecreasing and positive valued function with the property that k ⇡ ( z ) ≤ ⇡ ( kz ) for any k ≥ 1 .

Revised: Then, the function  g ( w ) =

Revised:  ( w ) h ( w ) is an admissible potential w.r.t.

Revised: F with the function g

Revised: ( z ) =

Revised: z andsatisﬁes the self-bounding regularity condition in Assumption 2.

Revised: 5.1 Phase retrieval

Original: In the phase retrieval problem [7, 10, 41], we wish to reconstruct a hidden vector w ∗ ∈
Revised: In the phase retrieval problem [7, 10], we wish to reconstruct a hidden vector w ∗ ∈

Original: R d with ￿ w ∗ ￿ = 273 1 using phaseless observations
Revised: R d with ￿ w ∗ ￿ = 1using phaseless observations

Original: S = {( a j ,y j )} j ≤
Revised: S = {( a j ,y j )} j ≤

Original: T of the form y j = ￿ a j ,w ∗ ￿ 2 where a j ∼ N ( 0 , Id ) .
Revised: T of the form y j = ￿ a j ,w ∗ ￿ 2 where a j ∼ N ( 0 , I ) .

Original: The classical approach to recover w ∗ is by using the per-sample loss function
Revised: The classical approach to recover w ∗ is by using the per-sample loss function

Original: f pr ( w ; ( a j ,y j ))
Revised: f pr ( w ; ( a j ,y j ))

Original: =
Revised: =

Original: (( a ￿ j w ) − y j ) 2 for which the corresponding population loss is given by
Revised: (( a ￿ j w ) − y j ) 2 for which the corresponding population loss is given by

Original: F pr ( w ) =
Revised: F pr ( w ) =

Original: E [ f pr ( w ; ( a,y ))] =
Revised: E [ f pr ( w ; ( a,y ))] =

Original: E a ∼ N ( 0 , I d) ￿￿ ( a ￿ w [Equation]
Revised: E a ∼ N ( 0 , I d

Original: [Equation]
Revised: [Equation]

Original: F pr is non-convex, and has stationary points (and local minima) that do not correspond to the globalminima.
Revised: F pr is non-convex, and has stationary points (and local minima) that do not correspond to the globalminima.

Original: In the following, we provide convergence guarantees for GD algorithm on F pr , and SGDalgorithms that computes stochastic gradient estimates using
Revised: In the following, we provide convergence guarantees for GD algorithm on F pr , and SGDalgorithms that computes stochastic gradient estimates using

Original: S .
Revised: S .

Original: We ﬁrst note that F pr satisﬁesself-bounding regularity conditions, and GF on F pr converges to the global minimizer for any initialpoint w 0 .
Revised: We ﬁrst note that F pr satisﬁesself-bounding regularity conditions, and GF on F pr converges to the global minimizer when theinitial point w 0 satisﬁes additional assumptions.

Original: Lemma 1.
Revised: Lemma 2.

Original: F pr satisﬁes Assumption 1.
Revised: F pr satisﬁes Assumption 1.

Original: Furthermore, for anyinitial point w 0  , the point w ( t ) on itsgradient ﬂow path  satisﬁes
Revised: Furthermore, for any point w 0 for which ￿￿ w 0 ,w ∗ ￿￿ ≥  ￿￿ d log ( d ) and ￿￿ w￿ − ￿ ≤ ￿ log ( d ) , the point w ( t ) on the GF path with w ( ) = w 0 satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Furthermore, the function R pr above is an admissible rate of convergence w.r.t.
Revised: where c 1 ,c 2 and c 3 are universal constants.

Original: F pr . The rate above follows from independently analysis the terms parallel and perpendicular components￿ w,w ∗ ￿ and ￿ w ￿ − ￿ w,w ∗ ￿ respectively.
Revised: The rate above is a consequence of the analysis in [10]. Lemma 2 provides conditions on w 0 underwhich the GF path converges with the rate function R pr .

Original: Our main tool for getting the convergence guarantee for GD / SGD is to utilize Theorem 2 to get an admissible potential w.r.t.
Revised: Our main tool for getting the convergenceguarantee for GD / SGD is to utilize Lemma 1 to get an admissible potential w.r.t.

Original: F pr , which can be plugged in 
Revised: F pr , which can beplugged in Theorem 4 and 5 to get a rate.

Revised: However, note that the conditions in Lemma 2 under which

Original: Theorem 4 and 5 to get the corresponding rates.
Revised: GF has a rate are neither continuously differentiable w.r.t. w 0 , nor do not satisfy the self-boundingregularity conditions required in Lemma 1. We circumvent this issue by relaxing the conditions on w 0 and by combining them into a single condition using a smooth approximation to min function. Deﬁneh pr

Original: Theorem 6.
Revised: [Equation] w [Equation]

Revised: ￿￿￿￿￿

Revised: [Equation]

Revised: (10) where  ( z

Revised: [Equation]

Revised: = {

Revised: [Equation]

Revised: if

Revised: [Equation]

Revised: Note that for any w 0 for which h ( w) ≥ 0 , we must have that ￿￿ w

Revised: [Equation]

Revised: w

Revised: [Equation]

Revised: ￿￿ d log ( d ) and￿￿

Revised: w￿ − ￿ ≤ ￿ log ( d ) , and thus GF from w 0 has rate R pr .

Revised: Furthermore, h pr is continuously differentiable, and both h pr and R pr satisfy all the conditions in the premise of Lemma 1.

Revised: Thus, Lemma 1provides an admissible potential w.r.t.

Revised: F pr that can be used to get the following rates.

Revised: Theorem 6.

Original: Consider the phase retrieval objective F pr given in (9) .
Revised: Consider the phase retrieval objective F pr given in (9) .

Original: For any initial point w 0 and T ≥ 1 , setting ⌘ appropriately,
Revised: For any initial point w 0 forwhich h pr ( w) > 0 , T ≥ 0 and setting ⌘ appropriately,

Original: (a) The point ￿ W T returned by GD starting from w 0 satisﬁes F pr
Revised: (a) The point ̂ w T returned by GD starting from w 0 satisﬁes F pr

Original: [Equation]
Revised: [Equation]

Original: for all T ≥ t 0 , where t 0 is a w 0 dependent constant.(b)
Revised: (b) The point ̂ w T returned by SGD

Original: The point ̂ w T returned by SGD starting from w 0 and using stochastic gradient estimates forwhich Assumption 3 holds, satisﬁes F pr( ̂
Revised: starting from w 0 and using stochastic gradient estimates forwhich Assumption 3 holds, satisﬁes F pr

Original: [Equation]
Revised: [Equation]

Original: ̃
Revised: ) =

Original: O ￿

Original: [Equation]
Revised: [Equation]

Original: √ T ￿ with probability at least 0 .

Original: The O( ⋅ ) notation above hides w 0 dependent constants which we specify in the Appendix. Our ratefor GD above matches the best known result in the literature in terms of the dependence on T [10]. To the best of our knowledge, ours is also the ﬁrst convergence analysis of SGD under arbitrary noiseconditions satisfying Assumption 3. While this rate is optimal under certain noise conditions, e.g.when  ( z ) =
Revised: O

Original:  2 , further improvements are possible when  is favorable.

Original: For example, supposethe stochastic gradient estimates were computed using samples from

Original: S by taking a fresh samplefor each estimate, i.e.

Original: [Equation]
Revised: [Equation]

Original: w . In this case, the stochastic gradientsatisfy Assumption 3 with 
Revised: ￿ with probability at least 0 .

Original: [Equation] where c is a universal constant (c.f. Candes et al. Lemma 7.4, 7.7]).
Revised: 7 ,where h pr is deﬁned in (10) , and C 1 and C 2 are constant that depends on w 0 ,w ∗ and d .

Original: While, our framework implies that this SGD algorithm (computing estimates usingsamples) converges at the rate of￿√ T , this rate can be improved further [10], and  we defer thereﬁned analysis for future research.
Revised: The O ( ⋅ ) notation above hides w 0 and problem dependent constants which we fully specify inthe Appendix.

Original: In many applications, GF is only known to converge from nice enough initial points that satisfycertain properties. In this section, we show how to use our tools to show convergence of GD/SGD forsuch problems, and consider matrix square root as an example. We ﬁrst provide the following generalutility lemma that shows how to construct admissible potentials when the rate for GF from w 0 holdsonly when w 0 satisﬁes a certain property characterized by h ( w) ≥ 0 .
Revised: Note that stochastic gradient estimates computed by using samples from

Original: Lemma 2. Let h ∶ R d ￿ [ 0 ,] be a continuously differentiable function, and suppose that for anypoint w for which h ( w ) > 0 , GF with w ( ) = w has rate F ( w ( t )) ≤ R ( w,t ) where R ( w, ⋅ ) is amonotonically decreasing function in t .
Revised: S , i.e.∇ f

Original: Furthermore, suppose that F ( w ) ≤ R ( w,) , F satisﬁes Assumption 1, R ( w,h ( w ) t ) is an admissible rate function w.r.t. F , and for any w,( a ) the function  ( w ) ∶ = ∫ ∞ t = 0 R ( w,t ) d t is continuously differentiable, andmax { ￿ ∇  ( w ) ￿ , ￿ ∇ 2  ( w ) ￿ } ≤  (  ( w )) where  is a positive, non-decreasing.
Revised: [Equation]

Original: ( b ) max { ￿ ∇ h ( w ) ￿ , ￿ ∇ 2 h ( w ) ￿ } ≤ ⇡ (  ( w )) where ⇡ is a positive, non-decreasing. ( c ) ( h ( w ) − h ( w ∗ )) ≤ µ (  ( w )) where µ is a positive, monotonically increasing function withthe property that kµ ( z ) ≤ µ ( kz ) for any k ≥ 1 . Then, the function  g ( w ) =  ( w ) ￿ h ( w ) is an admissible potential w.r.t.
Revised: w satisfy Assumption 3.

Original: F with g ( z ) = z , andsatisﬁes the self-bounding regularity condition in Assumption 2.
Revised: As a result SGD algorithm (usingsamples) converges for phase retrieval at the rate of￿√ t .

Original: 5.2.1 Matrix square root
Revised: 5.2 Matrix square root

Original: In the matrix square root problem [12, 20], we are given a positive deﬁnite and symmetric matrix
Revised: In the matrix square root problem [12, 19], we are given a positive deﬁnite and symmetric matrix

Original: M ∈
Revised: M [Equation]

Original: R d × d with  d (
Revised: d with  min (

Original: M ) > 0 , and wish to ﬁnd a symmetric W
Revised: M ) > 0 , and wish to ﬁnd a symmetric W

Original: [Equation]
Revised: [Equation]

Original: d that minimizes the objective
Revised: d that minimizes the objective

Original: [Equation]
Revised: [Equation]

Original: F ms is non-convex in W , and has spurious stationary points.
Revised: F ms is non-convex in W , and has spurious stationary points.

Original: In the following, we provide convergenceguarantees for GD/SGD algorithm on F ms .
Revised: In the following, we provide convergenceguarantees for GD / SGD algorithm on F ms .

Original: We ﬁrst note that F ms satisﬁes self-bounding regularityconditions, and GF on F ms converges to the global minimizer when the initial point w 0 satisﬁesadditional assumptions.We capture these initial conditions using the function h ms deﬁned as [Equation] where the function 
Revised: We ﬁrst note that F ms satisﬁes self-bounding regularityconditions, and GF on F ms converges to the global minimizer when the initial point w 0 satisﬁesadditional assumptions.

Original: [Equation]

Original: ↵ , and denotes a smoothened version of the indicator function given by

Original: [Equation]

Original: ≤ z ≤ ↵ 2 , − 2 ↵ 2 z+ 4 ↵ z − 1 if ↵≤ z ≤ ↵ , and 1 if ↵ ≤ z } .

Original: Lemma 3.
Revised: Lemma 3.

Original: F ms satisﬁes Assumption 1.
Revised: F ms satisﬁes Assumption 1.

Original: Furthermore, for any initial point W 0 for which h ms (
Revised: Furthermore, for any  > 0 and any point W 0 for which min (

Original: W) > 0 ,the point W ( t ) on its GF path  satisﬁesF
Revised: W) >  , the point W ( t ) on the GF path from W ( ) = W 0 satisﬁes

Original: [Equation]
Revised: [Equation]

Original: where ↵ =  min [Equation] ) ￿ ↵ and the function h ms is deﬁned in (11) . Theabove rate follows from directly solving the PDE associated with the gradient ﬂow on theunderlying objective.
Revised: The rate above is a consequence of the analysis in [19].

Original: Lemma 3 provides conditions on W 0 under which the GF path converges withthe rate function R ms .
Revised: Lemma 3 provides conditions on W 0 underwhich the GF path converges with the rate function R ms .

Original: Our main tool  forshowing the convergence of GD / SGD is  by using Lemma 2to get admissible potentials.
Revised: Our main tool to getting convergencerate for GD / SGD is again by using Lemma 1 to get admissible potentials.

Original: Note that the function h ms takes values in [ 0 ,] , is continuouslydifferentiable, and as we show in the appendix satisﬁes all therequired self-bounding regularityconditions in Lemma 2.
Revised: However, note that theconditions in Lemma 2 under which GF has a rate are neither continuously differentiable w.r.t. W 0 ,nor do not satisfy the self-bounding regularity conditions required in Lemma 1. We circumvent thisissue by relaxing the conditions on W 0 . Deﬁne the function

Original: Thus, Lemma 2 provides an admissible potential w.r.t.
Revised: [Equation] where  is deﬁned in Section 5.1. Note that for any W 0 for which h  ( W) > 0 , we must also havethat  min (

Revised: [Equation]

Revised:  > 0 and thus GF from W 0 has the rate R ms .

Revised: Furthermore, h pr is continuouslydifferentiable, and both h pr and R pr satisfy all the conditions in the premise of Lemma 1.

Revised: Thus,

Revised: Lemma 1 provides an admissible potential w.r.t.

Original: F pr which can beused to get the following rates.
Revised: F pr which can be used to get the following rates.

Original: Theorem 7.
Revised: Theorem 7.

Original: Consider the matrix square root objective F ms given in (10) .
Revised: Consider the matrix square root objective F ms given in (11) .

Original: For any  > 0 , initial point
Revised: For any  > 0 , initial point

Original: W 0 for which h ms (
Revised: W 0 for which h ms (

Original: W) > 0 and setting ⌘ appropriately, 
Revised: W) > 0 and setting ⌘ appropriately, (a) The point

Revised: ￿

Original: (a) The point ￿ W T returned by GD starting from w0 satisﬁes F ms 
Revised: W T returned by GD starting from w satisﬁes F ms ( ￿

Revised: W

Original: [Equation]
Revised: [Equation]

Original: for all T ≥
Revised: O ￿ e

Original: t 0 ,where t 0 is a w 0 dependent constant.
Revised: [Equation] C [Equation] ￿ ,

Original: b) The point
Revised: (b) The point

Original: ￿
Revised: ￿

Revised: W T returned by SGD starting from

Original: W
Revised: w

Original: T returned by SGD starting from w 0 and using stochastic gradient estimates for
Revised: and using stochastic gradient estimates forwhich

Original: which Assumption3 holds, satisﬁes F ms 
Revised: Assumption holds, satisﬁes F ms ( ￿ W

Revised: [Equation]

Revised: ̃

Revised: O ￿

Revised: C

Original: [Equation]
Revised: [Equation]

Original:  with probability at least 0 .
Revised: T ￿ with probability at least 0 .

Original: The O ( ⋅ ) notation above hides W 0 dependent constants which we specify in the Appendix. Ourrate for GD above matches the best known result in the literature in terms of the dependence on T[20]. Ours is also the ﬁrst convergence analysis of SGD under arbitrary noise conditions satisfying Assumption 3.
Revised: 7 ,where C 1 and C 2 are constant that depends on W 0 ,M and d .

Original: Note that the classical stochastic optimization setting in which∇ f ms ( w ; z ) =
Revised: Note that the classical stochastic optimization setting in which

Original: W− 352 M )
Revised: [Equation]

Original: W + 2 W ( W−
Revised: [Equation]

Original: M ) + " t where " t is a sub-Gaussian random variable with mean 0 and variance 2 satisﬁes Assumption 3 with  ( z ) =
Revised: " t where " t is a sub-Gaussian random variable with mean 0 and variance  2 satisﬁes Assumption 3.

Original:  2 , and as a resultof Theorem 7, we get that SGD convergesat the rate of￿√ T . To the best of our knowledge, convergence of SGD in the stochastic optimizationsetting for matrix square root problem was not known before.
Revised: As a result we have convergence guarantees for SGD algorithm for this case.

Original: 5.3 Extending Chatterjee 2022 [8]
Revised: 5.3 Extending Chatterjee 2022 [8]

Original: If the objective F is such that some potential  g satisﬁes the geometric condition in (4) for everyw , then we have a rate of convergence for GF (Theorem 1).
Revised: If the objective F is such that some potential  g satisﬁes the geometric condition in (4) for everyw , then we have a rate of convergence for GF (Theorem 1).

Original: As we saw earlier, for instance, usingthis machinery one can obtain rates for GF/GD/SGD when F has P Ł property everywhere.
Revised: As we saw earlier, for instance, usingthis machinery one can obtain rates for GF/GD/SGD when F has PL property everywhere.

Original: However,such global properties, that (4) holds for every w are often too stringent to hold in practice.
Revised: However,such global properties, that (4) holds for every w are often too stringent to hold in practice.

Original: Inorder to go beyond global assumption, in Lemma 2 we showed how to extend our tools (by deﬁningcorresponding admissible potentials) when such properties (and thus rates for GF) only hold in someregion.
Revised: Inorder to go beyond global assumption, in Lemma 1 we showed how to extend our tools (by deﬁningcorresponding admissible potentials) when such properties (and thus rates for GF) only hold in someregion.

Original: Convergence under such local properties has also been considered before in other works[10, 13, 22, 34, 42, 20, 33].
Revised: Convergence under such local properties has also been considered before in other works[10, 13, 21, 33, 39, 19, 32].

Original: However, all of these results usually rely on being able to choose aninitialization w 0 in the good region, where the corresponding local property holds, and is close enoughto the global minima that we wish to converge to.
Revised: However, all of these results usually rely on being able to choose aninitialization w 0 in the good region, where the corresponding local property holds, and is close enoughto the global minima that we wish to converge to.

Original: This is not always practical, and to circumvent thisissue in a recent work of Chatterjee [8], an assumption that is “local” w.r.t.
Revised: This is not always practical, and to circumvent thisissue in a recent work of Chatterjee [8], an assumption that is “local” w.r.t.

Original: initial point is providedunder which one can show that GF and GD starting from this initialization is guaranteed to converge(at an exponential rate).
Revised: initial point is providedunder which one can show that GF and GD starting from this initialization is guaranteed to converge(at an exponential rate).

Original: The interesting property of this condition is that it is local to initial point w 0considered and does not make any global assumption on the objective.
Revised: The interesting property of this condition is that it is local to initial point w 0considered and does not make any global assumption on the objective.

Original: Using the tools in this paper, this type of local property can be easily extended to more generalproperties than what was considered in Chatterjee [8].
Revised: Using the tools in this paper, this type of local property can be easily extended to more generalproperties than what was considered in Chatterjee [8].

Original: For ease of presentation, we present below theresult for H -smooth objective F and for GF convergence, the corresponding techniques can be easilyextended show GD/SGD convergence when Assumption 1 holds.
Revised: For ease of presentation, we present below theresult for H -smooth objective F and for GF convergence, the corresponding techniques can be easilyextended show GD/SGD convergence when Assumption 1 holds.

Original: Given a function r ∶
Revised: Given a function r ∶

Original: R d ￿
Revised: R d ￿

Original: R + anda monotonically increasing positive function g , deﬁne
Revised: R + anda non-decreasing positive function g , deﬁne

Original: [Equation]
Revised: [Equation]

Original: Our main assumption on the initial point w 0 is that for some  > 0 and some functions R and g ,
Revised: Our main assumption on the initial point w 0 is that for some  > 0 and some functions R and g ,￿

Original: [Equation]
Revised: [Equation]

Original:  The next lemma shows that for any initial point w 0 that satisﬁes the local condition above, one has arate of convergence for GF starting from w 0 .
Revised: [Equation] The next lemma shows that for any initial point w 0 that satisﬁes the local condition above, one has arate of convergence for GF starting from w 0 .

Original: Lemma 4.
Revised: Lemma 4.

Original: Suppose w 0 satisﬁes (13) for some functions R and g , and radius  = > 0 .
Revised: Suppose w 0 satisﬁes (14) for some functions R and g , and radius  = > 0 .

Original: Then, gradient ﬂow starting from w ( ) = w 0 satisﬁes for any t ≥ 0 ,
Revised: Then, gradient ﬂow starting from w ( ) = w 0 satisﬁes for any t ≥ 0 ,

Original: F ( w ( t
Revised: F ( w ( t

Original: [Equation]
Revised: [Equation]

Original: ) ≤ g
Revised: ≤ g

Original: [Equation]
Revised: [Equation]

Original: ￿ r ( w) T ↵ ( w 0 , ) ￿ .

Original: To obtain nearly matching rates for the type of condition in Chatterjee [8], one can choose r ( w ) =
Revised: To obtain nearly matching rates for the type of condition in Chatterjee [8], one can choose r ( w ) =

Original: p ⋅
Revised: p ⋅

Original: F ( w ) ￿ p and g ( z ) =
Revised: F ( w ) ￿ p and g ( z ) =

Original: z￿ p .
Revised: z￿ p .

Original: Since p is arbitrary, setting p =
Revised: Since p is arbitrary, setting p =

Original: T ↵ ( w 0 , ) ￿ e we obtain nearly thesame rate and the local condition as Chatterjee [8] (upto constants).
Revised: T ↵ ( w 0 , ) ￿ e we obtain nearly thesame rate and the local condition as Chatterjee [8] (upto constants).

Original: The interesting part though, isthat this is for only one choice of g and r , whereas we can get the convergence for GF when thecondition holds for any g,r .
Revised: The interesting part though, isthat this is for only one choice of g and r , whereas we can get the convergence for GF when thecondition holds for any g,r .

Original: In Chatterjee [8], examples of overparmeterized deep neural nets areshown to satisfy the assumption (for the speciﬁc r and g above).
Revised: In Chatterjee [8], examples of overparmeterized deep neural nets areshown to satisfy the assumption (for the speciﬁc r and g above).

Original: With a wider choice of g and rwe can extend these to more general models (eg. neural networks with milder assumptions on theactivation function). 
Revised: With a wider choice of g and rwe can extend these to more general models (eg. neural networks with milder assumptions on theactivation function). Checklist

Revised: The checklist follows the references.

Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Revised: You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Revised: For example:

Revised: • Did you include the license to the code and datasets?

Revised: [N/A]

Revised: • Did you include the license to the code and datasets?

Revised: [N/A]

Revised: The code and the data areproprietary.

Revised: • Did you include the license to the code and datasets?

Revised: [N/A]

Revised: Please do not modify the questions and only use the provided macros for your answers.

Revised: Note that the

Revised: Checklist section does not count towards the page limit.

Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Revised: For all authors...

Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Revised: contributions and scope?

Revised: [Yes]

Revised: (b) Did you describe the limitations of your work?

Revised: [Yes]

Revised: (c) Did you discuss any potential negative societal impacts of your work?

Revised: [N/A] We donot foresee any negative societal impacts of our work.(d)

Revised: Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Revised: [Yes]2.

Revised: If you are including theoretical results...

Revised: (a) Did you state the full set of assumptions of all theoretical results?

Revised: [Yes](b) Did you include complete proofs of all theoretical results?

Revised: [Yes]3.

Revised: If you ran experiments...

Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Revised: [N/A]

Revised: (b) Did you specify all the training details (e

Revised: g., data splits, hyperparameters,

Revised: how they

Revised: were chosen)?

Revised: [N/A]

Revised: (c) Did you report error bars (e

Revised: g., with respect to the random seed after running experi-

Revised: ments multiple times)?

Revised: [N/A]

Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Revised: [N/A]

Revised: If you are using existing assets (e

Revised: g., code, data, models) or curating/releasing new assets...

Revised: (a) If your work uses existing assets, did you cite the creators?

Revised: [N/A]

Revised: (b) Did you mention the license of the assets?

Revised: [N/A]

Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [N/A]

Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Revised: using/curating?

Revised: [N/A]

Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Revised: information or offensive content?

Revised: [N/A]

Revised: If you used crowdsourcing or conducted research with human subjects...

Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Revised: applicable?

Revised: [N/A]

Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Revised: Board (IRB) approvals, if applicable?

Revised: [N/A]

Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Revised: spent on participant compensation?

Revised: [N/A]