Original: A Uniﬁed Convergence Theorem for Stochastic Optimization Methods
Revised: A Uniﬁed Convergence Theorem for Stochastic Optimization Methods

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: In this work, we provide a fundamental uniﬁed convergence theorem used forderiving expected and almost sure convergence results for a series of stochasticoptimization methods.
Revised: In this work, we provide a fundamental uniﬁed convergence theorem used forderiving expected and almost sure convergence results for a series of stochasticoptimization methods.

Original: Our uniﬁed theorem only requires to verify several representative conditions and is not tailored to any speciﬁc algorithm.
Revised: Our uniﬁed theorem only requires to verify several representative conditions and is not tailored to any speciﬁc algorithm.

Original: As a direct application, we recover expected and almost sure convergence results of the stochasticgradient method ( SGD ) and random reshufﬂing ( RR ) under more general settings.
Revised: As a direct application, we recover expected and almost sure convergence results of the stochasticgradient method ( SGD ) and random reshufﬂing ( RR ) under more general settings.

Original: Moreover, we establish new expected and almost sure convergence results forthe stochastic proximal gradient method ( prox - SGD ) and stochastic model-basedmethods for nonsmooth nonconvex optimization problems.
Revised: Moreover, we establish new expected and almost sure convergence results forthe stochastic proximal gradient method ( prox - SGD ) and stochastic model-basedmethods for nonsmooth nonconvex optimization problems.

Original: These applicationsreveal that our uniﬁed theorem provides a plugin-type convergence analysis andstrong convergence guarantees for a wide class of stochastic optimization methods
Revised: These applicationsreveal that our uniﬁed theorem provides a plugin-type convergence analysis andstrong convergence guarantees for a wide class of stochastic optimization methods

Original:  Introduction
Revised:  Introduction

Original: Stochastic optimization methods are widely used to solve stochastic optimization problems andempirical risk minimization, serving as one of the foundations of machine learning.
Revised: Stochastic optimization methods are widely used to solve stochastic optimization problems andempirical risk minimization, serving as one of the foundations of machine learning.

Original: Among themany different stochastic methods, the most classic one is the stochastic gradient method ( SGD ),which dates back to Robbins and Monro [32].
Revised: Among themany different stochastic methods, the most classic one is the stochastic gradient method ( SGD ),which dates back to Robbins and Monro [34].

Original: If the problem at hand has a ﬁnite-sum structure, thenanother popular stochastic method is random reshufﬂing ( RR ) [18].
Revised: If the problem at hand has a ﬁnite-sum structure, thenanother popular stochastic method is random reshufﬂing ( RR ) [19].

Original: When the objective function hasa composite form or is weakly convex (nonsmooth and nonconvex), then the stochastic proximalgradient method ( prox - SGD ) and stochastic model-based algorithms are the most typical approaches[16, 9].
Revised: When the objective function hasa composite form or is weakly convex (nonsmooth and nonconvex), then the stochastic proximalgradient method ( prox - SGD ) and stochastic model-based algorithms are the most typical approaches[17, 10].

Original: Apart from the mentioned stochastic methods, there are many others like SGD with momentum, Adam, stochastic higher order methods, etc.
Revised: Apart from the mentioned stochastic methods, there are many others like SGD withmomentum, Adam, stochastic higher order methods, etc.

Original: In this work, our goal is to establish and understandfundamental convergence properties of these stochastic optimization methods via a novel uniﬁedconvergence framework.
Revised: In this work, our goal is to establish andunderstand fundamental convergence properties of these stochastic optimization methods via a noveluniﬁed convergence framework.

Original: Motivations.
Revised: Motivations.

Original: Suppose we apply SGD to minimize a smooth nonconvex function f .
Revised: Suppose we apply SGD to minimize a smooth nonconvex function f .

Original: SGD generates asequence of iterates { x k } k  0 , which is a stochastic process due to the randomness of the algorithmand the utilized stochastic oracles.
Revised: SGD generates asequence of iterates { x k } k  0 , which is a stochastic process due to the randomness of the algorithmand the utilized stochastic oracles.

Original: The most commonly seen ‘convergence result’ for SGD is theexpected iteration complexity , which typically takes the form [15]
Revised: The most commonly seen ‘convergence result’ for SGD is theexpected iteration complexity , which typically takes the form [16]

Original: [Equation]
Revised: [Equation]

Original: where T denotes the total number of iterations and ¯ k is an index sampled uniformly at randomfrom { 0 , . .
Revised: where T denotes the total number of iterations and ¯ k is an index sampled uniformly at randomfrom { 0 , . .

Original: , T } .
Revised: , T } .

Original: Note that we ignored some higher-order convergence terms and constants to easethe presentation.
Revised: Note that we ignored some higher-order convergence terms and constants to easethe presentation.

Original: Complexity results are integral to understand core properties and progress of thealgorithm during the ﬁrst T iterations, while the asymptotic convergence behavior plays an equallyimportant role as it characterizes whether an algorithm can eventually approach an exact stationary point or not.
Revised: Complexity results are integral to understand core properties and progress of thealgorithm during the ﬁrst T iterations, while the asymptotic convergence behavior plays an equallyimportant role as it characterizes whether an algorithm can eventually approach an exact stationarypoint or not.

Revised: We refer to Appendix H for additional motivational background for studying asymptoticconvergence properties of stochastic optimization methods.

Original: Here, an expected convergence result  has the form
Revised: Here, an expected convergence result , associated with the nonconvex minimization problem min x f ( x ) , has the form

Original: [Equation]
Revised: [Equation]

Original: Intuitively, it should be possible to derive expected convergence from the expected iteration complexity(1) by letting T ! 1 .
Revised: Intuitively, it should be possible to derive expected convergence from the expected iteration complexity(1) by letting T ! 1 .

Original: However, this is not the case as the ‘ min ’ operator and the sampled ¯ k are notwell deﬁned or become meaningless when T goes to 1 .
Revised: However, this is not the case as the ‘ min ’ operator and the sampled ¯ k are notwell deﬁned or become meaningless when T goes to 1 .

Original: The above results are stated in expectation and describe the behavior of the algorithm by averaginginﬁnitely many runs.
Revised: The above results are stated in expectation and describe the behavior of the algorithm by averaginginﬁnitely many runs.

Original: Though this is an important convergence measure, in practical situations thealgorithm is often only run once and the last iterate is returned as a solution.
Revised: Though this is an important convergence measure, in practical situations thealgorithm is often only run once and the last iterate is returned as a solution.

Original: This observation motivates and necessitates almost sure convergence results , which establish convergence with probability1 for a single run of the stochastic method:
Revised: This observation motivates and necessitates almost sure convergence results , which establish convergence with probability1 for a single run of the stochastic method:

Original: [Equation]
Revised: [Equation]

Original: 42 Backgrounds.
Revised: 44 Backgrounds.

Original: Expected and almost sure convergence results have been extensively studied forconvex optimization; see, e.g., [8, 30, 38, 42, 4, 37].
Revised: Expected and almost sure convergence results have been extensively studied forconvex optimization; see, e.g., [9, 32, 40, 44, 5, 39].

Original: Almost sure convergence of SGD for minimizinga smooth nonconvex function f was provided in the seminal work [2] using very standard assumptions,i.e., Lipschitz continuous r f and bounded variance.
Revised: Almost sure convergence of SGD for minimizinga smooth nonconvex function f was provided in the seminal work [3] using very standard assumptions,i.e., Lipschitz continuous r f and bounded variance.

Original: Under the same conditions, the same almostsure convergence of SGD was established in [29] based on a much simpler argument than that of [2].
Revised: Under the same conditions, the same almostsure convergence of SGD was established in [31] based on a much simpler argument than that of [3].

Original: A weaker ‘ lim inf ’-type almost sure convergence result for SGD with AdaGrad step sizes was shownin [23].
Revised: A weaker ‘ lim inf ’-type almost sure convergence result for SGD with AdaGrad step sizes was shownin [24].

Original: Recently, the work [25] derives almost sure convergence of SGD under the assumptions thatf and r f are Lipschitz continuous, f and r f have bounded sub-level set, and the second momentof the stochastic error is bounded.
Revised: Recently, the work [26] derives almost sure convergence of SGD under the assumptions thatf and r f are Lipschitz continuous, f and r f have bounded sub-level set, and the second momentof the stochastic error is bounded.

Original: This result relies on stronger assumptions than the base resultsin [2].
Revised: This result relies on stronger assumptions than the base resultsin [3].

Original: In terms of expected convergence, the work [5] showed lim k !
Revised: In terms of expected convergence, the work [6] showed lim k !

Original: 1 E [ kr f ( x k ) k ] = 

Revised: 1 E [ kr f ( x k ) k ] =

Original: underthe additional assumptions that f is twice continuously differentiable and the multiplication of the
Revised: underthe additional assumptions that f is twice continuously differentiable and the multiplication of the

Original: Hessian and gradient r 2 f ( x ) r f ( x ) is Lipschitz continuous.
Revised: Hessian and gradient r 2 f ( x ) r f ( x ) is Lipschitz continuous.

Original: Though the convergence of SGD is well-understood and a classical topic, asymptotic convergenceresults of the type (2) and (3) often require a careful and separate analysis for other stochasticoptimization methods — especially when the objective function is simultaneously nonsmooth andnonconvex.
Revised: Though the convergence of SGD is well-understood and a classical topic, asymptotic convergenceresults of the type (2) and (3) often require a careful and separate analysis for other stochasticoptimization methods — especially when the objective function is simultaneously nonsmooth andnonconvex.

Original: In fact and as outlined, a direct transition from the more common complexity results (1)to the full convergence results (2) and (3) is often not possible without further investigation.
Revised: In fact and as outlined, a direct transition from the more common complexity results (1)to the full convergence results (2) and (3) is often not possible without further investigation.

Original: Main contributions.
Revised: Main contributions.

Original: We provide a fundamental uniﬁed convergence theorem (see Theorem 2.1) forderiving both expected and almost sure convergence of stochastic optimization methods.
Revised: We provide a fundamental uniﬁed convergence theorem (see Theorem 2.1) forderiving both expected and almost sure convergence of stochastic optimization methods.

Original: Our theoremis not tailored to any speciﬁc algorithm, instead it incorporates several abstract conditions that suit avast and general class of problem structures and algorithms.
Revised: Our theoremis not tailored to any speciﬁc algorithm, instead it incorporates several abstract conditions that suit avast and general class of problem structures and algorithms.

Original: The proof of this theorem is elementary.
Revised: The proof of this theorem is elementary.

Original: We then apply our novel theoretical framework to several classical stochastic optimization methodsto recover existing and to establish new convergence results.
Revised: We then apply our novel theoretical framework to several classical stochastic optimization methodsto recover existing and to establish new convergence results.

Original: Speciﬁcally, we recover expected andalmost sure convergence results for SGD and RR .
Revised: Speciﬁcally, we recover expected andalmost sure convergence results for SGD and RR .

Original: Though these results are largely known in theliterature, we derive uniﬁed and slightly stronger results under a general ABC condition [21, 20]rather than the standard bounded variance assumption.
Revised: Though these results are largely known in theliterature, we derive uniﬁed and slightly stronger results under a general ABC condition [22, 21]rather than the standard bounded variance assumption.

Original: We also remove the stringent assumption usedin [5] to show (2) for SGD .
Revised: We also remove the stringent assumption usedin [6] to show (2) for SGD .

Original: As a core application of our framework, we derive expected and almostsure convergence results for prox - SGD in the nonconvex setting and under the more general ABCcondition and for stochastic model-based methods under very standard assumptions.
Revised: As a core application of our framework, we derive expected and almostsure convergence results for prox - SGD in the nonconvex setting and under the more general ABCcondition and for stochastic model-based methods under very standard assumptions.

Original: In particular, weshow that the iterates { x k } k  0 generated by prox - SGD and other stochastic model-based methodswill approach the set of stationary points almost surely and in an expectation sense.
Revised: In particular, weshow that the iterates { x k } k  0 generated by prox - SGD and other stochastic model-based methodswill approach the set of stationary points almost surely and in an expectation sense.

Original: These results arenew to our knowledge (see also Subsection 3.5 for further discussion).
Revised: These results arenew to our knowledge (see also Subsection 3.5 for further discussion).

Original: The above applications illustrate the general plugin-type purpose of our uniﬁed convergence analysisframework.
Revised: The above applications illustrate the general plugin-type purpose of our uniﬁed convergence analysisframework.

Original: Based on the given recursion and certain properties of the algorithmic update, we canderive broad convergence results by utilizing our theorem, which can signiﬁcantly simplify theconvergence analysis of stochastic optimization methods; see Subsection 2.1 for a summary.
Revised: Based on the given recursion and certain properties of the algorithmic update, we canderive broad convergence results by utilizing our theorem, which can signiﬁcantly simplify theconvergence analysis of stochastic optimization methods; see Subsection 2.1 for a summary.

Original:  A uniﬁed convergence theorem
Revised:  A uniﬁed convergence theorem

Original: Throughout this work, let ( ⌦ , F , {F k } k  0 , P ) be a ﬁltered probability space and let us assume thatthe sequence of iterates { x k } k  0 is adapted to the ﬁltration {F k } k  0 , i.e., each of the random vectorsx k : ⌦ !
Revised: Throughout this work, let ( ⌦ , F , {F k } k  0 , P ) be a ﬁltered probability space and let us assume thatthe sequence of iterates { x k } k  0 is adapted to the ﬁltration {F k } k  0 , i.e., each of the random vectorsx k : ⌦ !

Original: R n is F k -measurable.
Revised: R n is F k -measurable.

Original: In this section, we present a uniﬁed convergence theorem for the sequence { x k } k  0 based on anabstract convergence measure  .
Revised: In this section, we present a uniﬁed convergence theorem for the sequence { x k } k  0 based on anabstract convergence measure  .

Original: We then present the main steps for showing the convergence ofa stochastic optimization method by following a step-by-step veriﬁcation of the conditions in ouruniﬁed convergence theorem.
Revised: We then present the main steps for showing the convergence ofa stochastic optimization method by following a step-by-step veriﬁcation of the conditions in ouruniﬁed convergence theorem.

Original: Theorem 2.1.
Revised: Theorem 2.1.

Original: Let the mapping  : R n !
Revised: Let the mapping  : R n !

Original: R m and the sequences { x k } k  0 ✓ R n and { µ k } k  

Revised: R m and the sequences { x k } k  0 ✓ R n and { µ k } k

Original: ✓
Revised: ✓

Original: R ++ be given.
Revised: R ++ be given.

Original: Consider the following conditions:
Revised: Consider the following conditions:

Original: (P.1)
Revised: (P.1)

Original: The function  is L  -Lipschitz continuous for some L  > 0 , i.e., we have k  ( x )   ( y ) k
Revised: The function  is L  -Lipschitz continuous for some L  > 0 , i.e., we have k  ( x )   ( y ) k

Original: L
Revised: L

Original:  k x  y k for all x , y 2 R n .(P.2)
Revised:  k x  y k for all x , y 2 R n .(P.2)

Original: There exists a constant a > 0 such that
Revised: There exists a constant a > 0 such that

Original: [Equation]
Revised: [Equation]

Original: The following statements are valid:
Revised: The following statements are valid:

Original: (i) Let the conditions (P.1) – (P.2) be satisﬁed and suppose further that(P.3)
Revised: (i) Let the conditions (P.1) – (P.2) be satisﬁed and suppose further that(P.3)

Original: There exist constants A , B , b  0 and p 1 , p 2 , q > 0 such that
Revised: There exist constants A , B , b  0 and p 1 , p 2 , q > 0 such that

Original: [Equation]
Revised: [Equation]

Original: (P.4)
Revised: (P.4)

Original: The sequence { µ k } k  0 and the parameters a, b, q, p 1 , p 2 satisfyµ k ! 0  ,
Revised: The sequence { µ k } k  0 and the parameters a, b, q, p 1 , p 2 satisfy{ µ k } k  0 is bounded ,

Original: [Equation]
Revised: [Equation]

Revised: and a, q  1 , a  b, p

Original: [Equation]
Revised: [Equation]

Original: and a, q  1 , a  b, p 1 , p 2  q.
Revised: p 2  q.

Original: Then, it holds that lim k !
Revised: Then, it holds that lim k !

Original: 1 E [ k  ( x k ) k ] = 0 .(ii)
Revised: 1 E [ k  ( x k ) k ] = 0 .(ii)

Original: Let the properties (P.1) – (P.2) hold and assume further that(P.3 0 )
Revised: Let the properties (P.1) – (P.2) hold and assume further that(P.3 0 )

Original: There exist constants A , b  0 , p 1 , p 2 , q > 0 and random vectors
Revised: There exist constants A , b  0 , p 1 , p 2 , q > 0 and random vectors

Original: A k , B k : ⌦ !
Revised: A k , B k : ⌦ !

Original: R nsuch that
Revised: R n such that

Original: [Equation]
Revised: [Equation]

Original: A k + µ p 2 k B k and for all k , it holds that E
Revised: and for all k , A k , B k are F k +1 -measurable and we have E

Original: [ A k | F k ] = 0 almost surely, E [ k A k k q ]  A , and lim 
Revised: [ A k | F k ] = 0 almostsurely, E [ k A k k q ]  A , and lim sup k !

Original: sup k !1 k B k k q / (1 + k  ( x k ) k b ) <  almost surely. (P.4 0 )
Revised: 1 k B k k q / (1 + k  ( x k ) k b ) < 1 almost surely.(P.4 0 )

Original: The sequence { µ k } k  0 and the parameters a, b, q, p 1 , p 2 satisfy µ k ! 0 ,
Revised: The sequence { µ k } k  0 and the parameters a, b, q, p 1 , p 2 satisfy µ k ! 0 ,

Original: [-X 1 k =
-]
Revised: [-X 1 k =
-][Equation]

Original: µ k = 1 , X 1 k =0 µ 2 p 1 k < 1 , and q  2 , qa  b, p 1 > 1 2 , p 2  1 .

Original: Then, it holds that lim k !
Revised: Then, it holds that lim k !

Original:  k  ( x k ) k = 0 almost surely.
Revised: 1 k  ( x k ) k = 0 almost surely.

Original: The proof of Theorem 2.1 is elementary.
Revised: The proof of Theorem 2.1 is elementary.

Original: We provide the core ideas here and defer its proof to
Revised: We provide the core ideas here and defer its proof to

Original: Appendix A. Item (i) is proved by contradiction.
Revised: Appendix A. Item (i) is proved by contradiction.

Original: An easy ﬁrst result is lim inf k !
Revised: An easy ﬁrst result is lim inf k !

Original: 1 E [ k  ( x k ) k a ] = 0 .
Revised: 1 E [ k  ( x k ) k a ] = 0 .

Original: We proceed and assume that { E [ k  ( x k ) k ] } k  0 does not converge to zero.
Revised: We proceed and assume that { E [ k  ( x k ) k ] } k  0 does not converge to zero.

Original: Then, for some  > 0 ,we can construct two subsequences { ` t } t  0 and { u t } t  0 such that ` t < u t and E [ k  ( x ` t ) k ]  2  ,
Revised: Then, for some  > 0 ,we can construct two subsequences { ` t } t  0 and { u t } t  0 such that ` t < u t and E [ k  ( x ` t ) k ]  2  ,

Original: E [ k  ( x u t )
Revised: E [ k  ( x u t )

Original: k a ]   a , and E [ k  ( x k ) k a ] >  a for all ` t < k < u t .
Revised: k a ]   a , and E [ k  ( x k ) k a ] >  a for all ` t < k < u t .

Original: Based on this construction, theconditions in the theorem, and a set of inequalities, we will eventually reach a contradiction.
Revised: Based on this construction, theconditions in the theorem, and a set of inequalities, we will eventually reach a contradiction.

Original: Wenotice that the Lipschitz continuity of  plays a prominent role when establishing this contradiction.
Revised: Wenotice that the Lipschitz continuity of  plays a prominent role when establishing this contradiction.

Original: Our overall proof strategy is inspired by the analysis of classical trust region-type methods, see, e.g.,[7, Theorem 6.4.6].
Revised: Our overall proof strategy is inspired by the analysis of classical trust region-type methods, see, e.g.,[8, Theorem 6.4.6].

Original: For item (ii), we ﬁrst control the stochastic behavior of the error terms A k bymartingale convergence theory.
Revised: For item (ii), we ﬁrst control the stochastic behavior of the error terms A k bymartingale convergence theory.

Original: We can then conduct sample-based arguments to derive the ﬁnalresult, which is essentially deterministic and hence, follows similar arguments to that of item (i).
Revised: We can then conduct sample-based arguments to derive the ﬁnalresult, which is essentially deterministic and hence, follows similar arguments to that of item (i).

Original: In the next subsection, we state the main steps for showing convergence of stochastic optimizationmethods.
Revised: In the next subsection, we state the main steps for showing convergence of stochastic optimizationmethods.

Original: This also clariﬁes the abstract conditions listed in the theorem.
Revised: This also clariﬁes the abstract conditions listed in the theorem.

Original: 2.1 The steps for showing convergence of stochastic optimization methods
Revised: 2.1 The steps for showing convergence of stochastic optimization methods

Original: In order to apply the uniﬁed convergence theorem, we have to verify the conditions stated in thetheorem, resulting in three main phases below.
Revised: In order to apply the uniﬁed convergence theorem, we have to verify the conditions stated in thetheorem, resulting in three main phases below.

Original: Phase I: Verifying (P.1)–(P.2).
Revised: Phase I: Verifying (P.1)–(P.2).

Original: Conditions (P.1)–(P.2) are used for both the expected and the almostsure convergence results.
Revised: Conditions (P.1)–(P.2) are used for both the expected and the almostsure convergence results.

Original: Condition (P.1) is a problem property and is very standard.
Revised: Condition (P.1) is a problem property and is very standard.

Original: We present theﬁnal convergence results in terms of the abstract measure  .
Revised: We present theﬁnal convergence results in terms of the abstract measure  .

Original: This measure can be regarded as f  f ⇤in convex optimization, kr f k in smooth nonconvex optimization, the magnitude of the gradient ofthe Moreau envelope in weakly convex optimization, etc.
Revised: This measure can be regarded as f  f ⇤in convex optimization, kr f k in smooth nonconvex optimization, the magnitude of the gradient ofthe Moreau envelope in weakly convex optimization, etc.

Original: In all the situations, assuming Lipschitz continuity of the convergence measure  is standard and is arguably a minimal assumption in orderto obtain iteration complexity and/or convergence results.
Revised: In all the situations, assuming Lipschitz continuity of the convergence measure  is standard and is arguably a minimal assumption in orderto obtain iteration complexity and/or convergence results.

Original: Condition (P.2) is typically a result of the algorithmic property or complexity analysis.
Revised: Condition (P.2) is typically a result of the algorithmic property or complexity analysis.

Original: To verify thiscondition, one ﬁrst establishes the recursion of the stochastic method, which almost always has theform
Revised: To verify thiscondition, one ﬁrst establishes the recursion of the stochastic method, which almost always has theform

Original: [Equation]
Revised: [Equation]

Original: Here, y k is a suitable Lyapunov function measuring the (approximate) descent property of thestochastic method, ⇣ k represents the error term satisfying
Revised: Here, y k is a suitable Lyapunov function measuring the (approximate) descent property of thestochastic method, ⇣ k represents the error term satisfying

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: ⇣ k < 1 ,  k is often related to thestep size and satisﬁes 
Revised: ⇣ k < 1 ,  k is often related to thestep sizes and satisﬁes P

Original: [Equation]
Revised: [Equation]

Original:  k < 1 .
Revised:  k < 1 .

Original: Then, applying the supermartingale convergence theorem (see Theorem B.1), we obtain
Revised: Then, applying the supermartingale convergence theorem(see Theorem B.1), we obtain

Original: [Equation]
Revised: [Equation]

Original: Phase II: Verifying (P.3)–(P.4) for showing expected convergence.
Revised: Phase II: Verifying (P.3)–(P.4) for showing expected convergence.

Original: Condition (P.3) requires anupper bound on the step length of the update in terms of expectation, including upper bounds forthe search direction and the stochastic error of the algorithm.
Revised: Condition (P.3) requires anupper bound on the step length of the update in terms of expectation, including upper bounds forthe search direction and the stochastic error of the algorithm.

Original: It is often related to certain boundedvariance-type assumptions for analyzing stochastic methods.
Revised: It is often related to certain boundedvariance-type assumptions for analyzing stochastic methods.

Original: For instance, (P.3) is satisﬁed under thestandard bounded variance assumption for SGD , the more general ABC assumption for SGD , thebounded stochastic subgradients assumption, etc.
Revised: For instance, (P.3) is satisﬁed under thestandard bounded variance assumption for SGD , the more general ABC assumption for SGD , thebounded stochastic subgradients assumption, etc.

Original: Condition (P.4) is a standard diminishing step sizescondition used in stochastic optimization.
Revised: Condition (P.4) is a standard diminishing step sizescondition used in stochastic optimization.

Original: Then, one can apply item (i) of Theorem 2.1 to obtain E [ k  ( x k ) k ] ! 0 .
Revised: Then, one can apply item (i) of Theorem 2.1 to obtain E [ k  ( x k ) k ] ! 0 .

Original: Phase III: Verifying (P.3 0 )–(P.4 0 ) for showing almost sure convergence.
Revised: Phase III: Verifying (P.3 0 )–(P.4 0 ) for showing almost sure convergence.

Original: Condition (P.3 0 ) is parallelto (P.3).
Revised: Condition (P.3 0 ) is parallelto (P.3).

Original: It decomposes the update into a martingale term A k and a bounded error term B k .
Revised: It decomposes the update into a martingale term A k and a bounded error term B k .

Original: We willsee later that this condition holds true for many stochastic methods.
Revised: We willsee later that this condition holds true for many stochastic methods.

Original: Though this condition requiresthe update to have a certain decomposable form, it indeed can be veriﬁed by bounding the step lengthof the update in conditional expectation, which is similar to (P.3).
Revised: Though this condition requiresthe update to have a certain decomposable form, it indeed can be veriﬁed by bounding the step lengthof the update in conditional expectation, which is similar to (P.3).

Original: Hence, (P.3 0 ) can be interpreted asa conditional version of (P.3).
Revised: Hence, (P.3 0 ) can be interpreted asa conditional version of (P.3).

Original: To see this, we can construct
Revised: To see this, we can construct

Original: [Equation]
Revised: [Equation]

Original: By Jensen’s inequality, we then have E [ A k| F k ]
Revised: By Jensen’s inequality, we then have E [ A k

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Thus, once it is possible to derive E [ k x k +1  x k k q | F k ]
Revised: Thus, once it is possible to derive E [ k x k +1  x k k q | F k ]

Original: = O ( µ qk ) in an almost sure sense, condition(P.3 0 ) is veriﬁed with p 1 = p 2 = 1 .
Revised: = O ( µ qk ) in an almost sure sense, condition(P.3 0 ) is veriﬁed with p 1 = p 2 = 1 .

Original: Condition (P.4 0 ) is parallel to (P.4) and is standard in stochasticoptimization.
Revised: Condition (P.4 0 ) is parallel to (P.4) and is standard in stochasticoptimization.

Original: Application of item (ii) of Theorem 2.1 then yields k  ( x k ) k ! 

Revised: Application of item (ii) of Theorem 2.1 then yields k  ( x k ) k !

Original: almost surely.
Revised: almost surely.

Original: In the next section, we will illustrate how to show convergence for a set of classic stochastic methodsby following the above three steps.
Revised: In the next section, we will illustrate how to show convergence for a set of classic stochastic methodsby following the above three steps.

Original:  Applications to stochastic optimization methods
Revised:  Applications to stochastic optimization methods

Original: 3.1 Convergence results of SGD
Revised: 3.1 Convergence results of SGD

Original: We consider the standard SGD method for solving the smooth optimization problem min x 2 R n f ( x ) ,where the iteration of SGD is given by
Revised: We consider the standard SGD method for solving the smooth optimization problem min x 2 R n f ( x ) ,where the iteration of SGD is given by

Original: [Equation]
Revised: [Equation]

Original: Here, g k denotes a stochastic approximation of the gradient r f ( x k ) .
Revised: Here, g k denotes a stochastic approximation of the gradient r f ( x k ) .

Original: We assume that each stochasticgradient g k is F k +1 -measurable and that the generated stochastic process { x k } k  0 is adapted to the 159 ﬁltration {F k } k  0 .
Revised: We assume that each stochasticgradient g k is F k +1 -measurable and that the generated stochastic process { x k } k  0 is adapted to the 161 ﬁltration {F k } k  0 .

Original: We consider the following standard assumptions:
Revised: We consider the following standard assumptions:

Original: (A.1)
Revised: (A.1)

Original: The mapping r f : R n !
Revised: The mapping r f : R n !

Original: R n is Lipschitz continuous on R n with modulus L > 0 .(A.2)
Revised: R n is Lipschitz continuous on R n with modulus L > 0 .(A.2)

Original: The objective function f is bounded from below on R n ,
Revised: The objective function f is bounded from below on R n ,

Original: i
Revised: i

Original: e., there is ¯ f such that f ( x )  ¯ ffor all x 2 R n .
Revised: e., there is ¯ f such that f ( x )  ¯ ffor all x 2 R n .

Original: (A.3)
Revised: (A.3)

Original: Each oracle g k deﬁnes an unbiased estimator of r f ( x k ) ,
Revised: Each oracle g k deﬁnes an unbiased estimator of r f ( x k ) ,

Original: i
Revised: i

Original: e., it holds that E [ g k | F k ] =
Revised: e., it holds that E [ g k | F k ] =

Original: r f ( x k ) almost surely, and there exist C , D  0 such that
Revised: r f ( x k ) almost surely, and there exist C , D  0 such that

Original: [Equation]
Revised: [Equation]

Original: almost surely 8 k 2 N 
Revised: almost surely 8 k 2 N .

Original: [Equation]

Original: (A.4)
Revised: (A.4)

Original: The step sizes { ↵ k } k  0 satisfy
Revised: The step sizes { ↵ k } k  0 satisfy

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original:  1 and
Revised: = 1 and

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: We now derive the convergence of SGD below by setting  ⌘ r f and µ k ⌘ ↵ k .
Revised: We now derive the convergence of SGD below by setting  ⌘ r f and µ k ⌘ ↵ k .

Original: Phase I: Verifying (P.1)–(P.2).
Revised: Phase I: Verifying (P.1)–(P.2).

Original: (A.1) veriﬁes condition (P.1) with L  ⌘ L .
Revised: (A.1) veriﬁes condition (P.1) with L  ⌘ L .

Original: We now check (P.2).
Revised: We now check (P.2).

Original: Using (A.2), (A.3), and a standard analysis for SGD gives the following recursion (see Appendix C.1for the full derivation):
Revised: Using (A.2), (A.3), and a standard analysis for SGD gives the following recursion (see Appendix C.1for the full derivation):

Original: [Equation]
Revised: [Equation]

Original: Taking total expectation, using (A.4), and applying the supermartingale convergence theorem (Theorem B.1) gives
Revised: Taking total expectation, using (A.4), and applying the supermartingale convergence theorem (Theorem B.1) gives

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: E [ kr f
Revised: E [ kr f

Original: [Equation]
Revised: [Equation]

Original: x k )
Revised: x k )

Original: k 2 ] < 1 .
Revised: k 2 ] < 1 .

Original: Furthermore, the sequence { E [ f ( x k )] } k  0 converges 172 to some ﬁnite value.
Revised: Furthermore, the sequence { E [ f ( x k )] } k  0 converges 174 to some ﬁnite value.

Original: This veriﬁes (P.2) with a = 2 .
Revised: This veriﬁes (P.2) with a = 2 .

Original: Phase II: Verifying (P.3)–(P.4) for showing expected convergence.
Revised: Phase II: Verifying (P.3)–(P.4) for showing expected convergence.

Original: For (P.3), we have by (5) and(A.3) that
Revised: For (P.3), we have by (5) and(A.3) that

Original: [Equation]
Revised: [Equation]

Original: Due to the convergence of { E [ f ( x k )] } k  0 , there exists F such that E [ f ( x k )  ¯ f ]  F for all k .
Revised: Due to the convergence of { E [ f ( x k )] } k  0 , there exists F such that E [ f ( x k )  ¯ f ]  F for all k .

Original: Thus,condition (P.3) holds with q = 2 , A = CF + D , p 1 = 2 , B = 1 , p 2 = 2 , and b = 2 .
Revised: Thus,condition (P.3) holds with q = 2 , A = CF + D , p 1 = 2 , B = 1 , p 2 = 2 , and b = 2 .

Original: Condition (P.4)is veriﬁed by (A.4) and the previous parameters choices.
Revised: Condition (P.4)is veriﬁed by (A.4) and the previous parameters choices.

Original: Therefore, we can apply Theorem 2.1 todeduce E [ kr f ( x k ) k ] ! 0 .
Revised: Therefore, we can apply Theorem 2.1 todeduce E [ kr f ( x k ) k ] ! 0 .

Original: Phase III: Verifying (P.3 0 )–(P.4 0 ) for showing almost sure convergence.
Revised: Phase III: Verifying (P.3 0 )–(P.4 0 ) for showing almost sure convergence.

Original: For (P.3 0 ), it follows fromthe update (5) that
Revised: For (P.3 0 ), it follows fromthe update (5) that

Original: [Equation]
Revised: [Equation]

Original: We have p 1 = 1 , A k = g k  r f ( x k ) , p 2 = 1 , and B k = r f ( x k ) .
Revised: We have p 1 = 1 , A k = g k  r f ( x k ) , p 2 = 1 , and B k = r f ( x k ) .

Original: Using (A.2), (A.3), E [ f ( x k ) ¯ f ]  F , and choosing any q = b > 0 establishes (P.3 0 ).
Revised: Using (A.2), (A.3), E [ f ( x k ) ¯ f ]  F , and choosing any q = b > 0 establishes (P.3 0 ).

Original: As before, condition (P.4 0 ) follows from (A.4)and the previous parameters choices.
Revised: As before, condition (P.4 0 ) follows from (A.4)and the previous parameters choices.

Original: Applying Theorem 2.1 yields kr f ( x k ) k ! 

Revised: Applying Theorem 2.1 yields kr f ( x k ) k !

Original: almost surely.
Revised: almost surely.

Original: Finally, we summarize the above results in the following corollary.
Revised: Finally, we summarize the above results in the following corollary.

Original: Corollary 3.1.
Revised: Corollary 3.1.

Original: Let us consider SGD (5) for smooth nonconvex optimization problems under (A.1) –(A.4) .
Revised: Let us consider SGD (5) for smooth nonconvex optimization problems under (A.1) –(A.4) .

Original: Then, we have lim k !
Revised: Then, we have lim k !

Original: 1 E [ kr f ( x k ) k ]
Revised: 1 E [ kr f ( x k ) k ]

Original: = 

Revised: =

Original: and lim k !
Revised: and lim k !

Original: 1 kr f ( x k ) k
Revised: 1 kr f ( x k ) k

Original: = 0 almost surely.
Revised: = 0 almost surely.

Original: 3.2 Convergence results of random reshufﬂing
Revised: 3.2 Convergence results of random reshufﬂing

Original: We now consider random reshufﬂing ( RR ) applied to problems with a ﬁnite sum structure
Revised: We now consider random reshufﬂing ( RR ) applied to problems with a ﬁnite sum structure

Original: [Equation]
Revised: [Equation]

Original: where each component function f ( · , i ) : R n !
Revised: where each component function f ( · , i ) : R n !

Original: R is supposed to be smooth.
Revised: R is supposed to be smooth.

Original: At iteration k , RRﬁrst generates a random permutation
Revised: At iteration k , RRﬁrst generates a random permutation

Original:  k +1 of the index set { 1 , . . .
Revised:  k +1 of the index set { 1 , . . .

Original: , N } .
Revised: , N } .

Original: It then updates x k to x k +1through N consecutive gradient descent-type steps by accessing and using the component gradients{ r f ( · ,  k + ) , . . .
Revised: It then updates x k to x k +1through N consecutive gradient descent-type steps by accessing and using the component gradients{ r f ( · ,  k + ) , . . .

Original: , r f ( · ,  k +1 N ) } sequentially.
Revised: , r f ( · ,  k +1 N ) } sequentially.

Original: Speciﬁcally, one update-loop (epoch) of RR is givenby
Revised: Speciﬁcally, one update-loop (epoch) of RR is givenby

Original: [Equation]
Revised: [Equation]

Original: After one such loop, the step size ↵ k and the permutation  k +1 is updated accordingly; cf.
Revised: After one such loop, the step size ↵ k and the permutation  k +1 is updated accordingly; cf.

Original: We make the following standard assumptions:
Revised: We make the following standard assumptions:

Original: (B.1)
Revised: (B.1)

Original: For all i 2 { 1 , . . . , N } , f ( · , i ) is bounded from below by some ¯ f and the gradient r f ( · , i )is Lipschitz continuous on R n with modulus L > 0 .
Revised: For all i 2 { 1 , . . . , N } , f ( · , i ) is bounded from below by some ¯ f and the gradient r f ( · , i )is Lipschitz continuous on R n with modulus L > 0 .

Original: (B.2) The step sizes { ↵ k } k  0 satisfy
Revised: (B.2) The step sizes { ↵ k } k  0 satisfy

Original: [Equation]
Revised: [Equation]

Original: A detailed derivation of the steps shown in Subsection 2.1 for RR is deferred to Appendix D.2.
Revised: A detailed derivation of the steps shown in Subsection 2.1 for RR is deferred to Appendix D.2.

Original: Basedon the discussion in Appendix D.2 and on Theorem 2.1, we obtain the following results for RR .
Revised: Basedon the discussion in Appendix D.2 and on Theorem 2.1, we obtain the following results for RR .

Original: Corollary 3.2.
Revised: Corollary 3.2.

Original: We consider RR (7) for smooth nonconvex optimization problems under (B.1) – (B.2) .
Revised: We consider RR (7) for smooth nonconvex optimization problems under (B.1) – (B.2) .

Original: Then it holds that lim k !
Revised: Then it holds that lim k !

Original: 1 E [ kr f ( x k ) k ]
Revised: 1 E [ kr f ( x k ) k ]

Original: = 

Revised: =

Original: and lim k !
Revised: and lim k !

Original: 1 kr f ( x k ) k
Revised: 1 kr f ( x k ) k

Original: = 0 almost surely.
Revised: = 0 almost surely.

Original: 3.3 Convergence of the proximal stochastic gradient method
Revised: 3.3 Convergence of the proximal stochastic gradient method

Original: We consider the composite-type optimization problem
Revised: We consider the composite-type optimization problem

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where f : R n !
Revised: where f : R n !

Original: R is a continuously differentiable function and ' : R n !
Revised: R is a continuously differentiable function and ' : R n !

Original: ( 1 , 1 ] is ⌧ -weaklyconvex (see Appendix E.1), proper, and lower semicontinuous.
Revised: ( 1 , 1 ] is ⌧ -weaklyconvex (see Appendix E.1), proper, and lower semicontinuous.

Original: In this section, we want to apply ouruniﬁed framework to study the convergence behavior of the well-known proximal stochastic gradientmethod ( prox - SGD ):
Revised: In this section, we want to apply ouruniﬁed framework to study the convergence behavior of the well-known proximal stochastic gradientmethod ( prox - SGD ):

Original: [Equation]
Revised: [Equation]

Original: where g k ⇡ r f ( x k ) is a stochastic approximation of r f ( x k ) , { ↵ k } k  0 ✓ R + is a suitable stepsize sequence, and prox ↵ k ' :
Revised: where g k ⇡ r f ( x k ) is a stochastic approximation of r f ( x k ) , { ↵ k } k  0 ✓ R + is a suitable stepsize sequence, and prox ↵ k ' :

Original: R n !
Revised: R n !

Original: R n , prox ↵ k ' ( x ) :
Revised: R n , prox ↵ k ' ( x ) :

Original: = argmin y 2 R n ' ( y ) + 12 ↵ k k x  y k 2 is thewell-known proximity operator of ' .
Revised: = argmin y 2 R n ' ( y ) + 12 ↵ k k x  y k 2 is thewell-known proximity operator of ' .

Original: 3.3.1 Assumptions and preparations
Revised: 3.3.1 Assumptions and preparations

Original: We ﬁrst recall several useful concepts from nonsmooth and variational analysis.
Revised: We ﬁrst recall several useful concepts from nonsmooth and variational analysis.

Original: For a functionh : R n !
Revised: For a functionh : R n !

Original: ( 1 , 1 ] , the Fréchet (or regular) subdifferential of h at the point x is given by
Revised: ( 1 , 1 ] , the Fréchet (or regular) subdifferential of h at the point x is given by

Original: [Equation]
Revised: [Equation]

Original: see, e.g., [35, Chapter 8].
Revised: see, e.g., [37, Chapter 8].

Original: If h is convex, then the Fréchet subdifferential coincides with the standard(convex) subdifferential.
Revised: If h is convex, then the Fréchet subdifferential coincides with the standard(convex) subdifferential.

Original: It is well known that the associated ﬁrst-order optimality condition for thecomposite problem (8) — 0 2 @ ( x ) =
Revised: It is well known that the associated ﬁrst-order optimality condition for thecomposite problem (8) — 0 2 @ ( x ) =

Original: r f ( x ) + @' ( x ) — can be represented as a nonsmoothequation, [35, 19],
Revised: r f ( x ) + @' ( x ) — can be represented as a nonsmoothequation, [37, 20],

Original: [Equation]
Revised: [Equation]

Original: where F ↵ nat denotes the so-called natural residual .
Revised: where F ↵ nat denotes the so-called natural residual .

Original: The natural residual F ↵ nat is a common stationaritymeasure for the nonsmooth problem (8) and widely used in the analysis of proximal methods.
Revised: The natural residual F ↵ nat is a common stationaritymeasure for the nonsmooth problem (8) and widely used in the analysis of proximal methods.

Original: We will make the following assumptions on f , ' , and the stochastic oracles { g k } k  0 :
Revised: We will make the following assumptions on f , ' , and the stochastic oracles { g k } k  0 :

Original: (C.1)
Revised: (C.1)

Original: The function f is bounded from below on R n , i
Revised: The function f is bounded from below on R n , i

Original: e., there is ¯ f such that f ( x )  ¯ f for allx 2 R n , and the gradient mapping r f is Lipschitz continuous (on R n ) with modulus L > 0 .
Revised: e., there is ¯ f such that f ( x )  ¯ f for all x 2 R n , and the gradient mapping r f is Lipschitz continuous (on R n ) with modulus L > 0 .

Original: (C.2)
Revised: (C.2)

Original: The function ' is ⌧ -weakly convex, proper, lower semicontinuous, and bounded from belowon dom '
Revised: The function ' is ⌧ -weakly convex, proper, lower semicontinuous, and bounded from belowon dom '

Original: and we have ' ( x )  ¯ ' for all x 2 dom ' .
Revised: and we have ' ( x )  ¯ ' for all x 2 dom ' .

Original: (C.3)
Revised: (C.3)

Original: There exists L ' > 0 such that ' ( x )  ' ( y )  L ' k x  y k for all x , y 2 dom ' .(C.4)
Revised: There exists L ' > 0 such that ' ( x )  ' ( y )  L ' k x  y k for all x , y 2 dom ' .(C.4)

Original: Each g k deﬁnes an unbiased estimator of r f ( x k ) , i.e., we have E [ g k | F k ] =
Revised: Each g k deﬁnes an unbiased estimator of r f ( x k ) , i.e., we have E [ g k | F k ] =

Original: r f ( x k )
Revised: r f ( x k )

Original: almost surely, and there exist C , D  0 such that
Revised: almost surely, and there exist C , D  0 such that

Original: [Equation]
Revised: [Equation]

Original: almost surely 8 k 2 N .

Original: (C.5)
Revised: (C.5)

Original: The step sizes { ↵ k } k  0 satisfyP
Revised: The step sizes { ↵ k } k  0 satisfy

Original: [Equation]
Revised: [Equation]

Original: and [Equation] Here, we again assume that the generated stochastic processes { x k } k  0 is adapted to the ﬁltration{F k } k  0 .
Revised: Here, we again assume that the generated stochastic processes { x k } k  0 is adapted to the ﬁltration{F k } k  0 .

Original: The assumptions (C.1), (C.2), (C.4), and (C.5) are fairly standard and broadly applicable.
Revised: The assumptions (C.1), (C.2), (C.4), and (C.5) are fairly standard and broadly applicable.

Original: In particular, (C.1), (C.4), and (C.5) coincide with the conditions (A.1)–(A.4) used in the analysis of
Revised: In particular, (C.1), (C.4), and (C.5) coincide with the conditions (A.1)–(A.4) used in the analysis of

Original: SGD .
Revised: SGD .

Original: We continue with several remarks concerning condition (C.3).
Revised: We continue with several remarks concerning condition (C.3).

Original: Remark 3.3.
Revised: Remark 3.3.

Original: Assumption (C.3) requires the mapping ' to be Lipschitz continuous on its effectivedomain dom ' .
Revised: Assumption (C.3) requires the mapping ' to be Lipschitz continuous on its effectivedomain dom ' .

Original: This condition holds in many important applications, e.g., when ' is chosen as anorm or indicator function.
Revised: This condition holds in many important applications, e.g., when ' is chosen as anorm or indicator function.

Original: Nonconvex examples satisfying (C.2) and (C.3) include, e.g., the minimaxconcave penalty (MCP) function [41], the smoothly clipped absolute deviation (SCAD) [13], or thestudent-t loss function.
Revised: Nonconvex examples satisfying (C.2) and (C.3) include, e.g., the minimaxconcave penalty (MCP) function [43], the smoothly clipped absolute deviation (SCAD) [14], or thestudent-t loss function.

Original: We refer to [3] and Appendix E.2 for further discussion.
Revised: We refer to [4] and Appendix E.2 for further discussion.

Original: 3.3.2 Convergence results of prox-SGD
Revised: 3.3.2 Convergence results of prox-SGD

Original: We now analyze the convergence of the random process { x k } k  0 generated by the stochastic 241 algorithmic scheme (9).
Revised: We now analyze the convergence of the random process { x k } k  0 generated by the stochastic 243 algorithmic scheme (9).

Original: As pioneered in [9], we will use the Moreau envelope env ✓ ,
Revised: As pioneered in [10], we will use the Moreau envelope env ✓ ,

Original: [Equation]
Revised: [Equation]

Original:  as a smooth Lyapunov function to study the descent properties and convergence of prox - SGD .
Revised: 10) as a smooth Lyapunov function to study the descent properties and convergence of prox - SGD .

Original: We ﬁrst note that the conditions (C.1) and (C.2) imply ✓  1 -weak convexity of for every ✓ 2 (0 , ( L +⌧ )  1 ] .
Revised: We ﬁrst note that the conditions (C.1) and (C.2) imply ✓  1 -weak convexity of for every ✓ 2 (0 , ( L +⌧ )  1 ] .

Original: In this case, the Moreau envelope env ✓ is a well-deﬁned and continuously differentiablefunction with gradient r env ✓ ( x ) = 1 ✓ ( x  prox ✓ ( x )) ; see, e.g., [34, Theorem 31.5].
Revised: In this case, the Moreau envelope env ✓ is a well-deﬁned and continuously differentiablefunction with gradient r env ✓ ( x ) = 1 ✓ ( x  prox ✓ ( x )) ; see, e.g., [36, Theorem 31.5].

Original: As shown in [11, 9], the norm of the Moreau envelope — kr env ✓ ( x ) k — deﬁnes an alternativestationarity measure for problem (8) that is equivalent to the natural residual if ✓ is chosen sufﬁcientlysmall.
Revised: As shown in [12, 10], the norm of the Moreau envelope — kr env ✓ ( x ) k — deﬁnes an alternativestationarity measure for problem (8) that is equivalent to the natural residual if ✓ is chosen sufﬁcientlysmall.

Original: A more explicit derivation of this connection is provided in Lemma E.1.
Revised: A more explicit derivation of this connection is provided in Lemma E.1.

Original: Next, we establish convergence of prox - SGD by setting  ⌘ r env ✓ and µ k ⌘ ↵ k .
Revised: Next, we establish convergence of prox - SGD by setting  ⌘ r env ✓ and µ k ⌘ ↵ k .

Original: Our analysis is 250 based on the following two estimates which are veriﬁed in Appendix E.4 and Appendix E.5.
Revised: Our analysis is 252 based on the following two estimates which are veriﬁed in Appendix E.4 and Appendix E.5.

Original: Lemma 3.4.
Revised: Lemma 3.4.

Original: Let { x k } k  0 be generated by prox - SGD and let the assumptions (C.1) – (C.4) be satisﬁed.
Revised: Let { x k } k  0 be generated by prox - SGD and let the assumptions (C.1) – (C.4) be satisﬁed.

Original: Then, for ✓ 2 (0 , [3 L + ⌧ ]  1 ) and all k with ↵ k  min { 12 ⌧ , [-
 [Equation]-]  [ L + ⌧ ]) } , it holds that
Revised: Then, for ✓ 2 (0 , [3 L + ⌧ ]  1 ) and all k with ↵ k  min { 12 ⌧ , [-
 [Equation]-]1 2( ✓  1  [ L + ⌧ ]) } , it holds that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: almost surely, where ¯ :=
Revised: almost surely, where ¯ :=

Original: ¯ f + ¯ ' .
Revised: ¯ f + ¯ ' .

Original: Lemma 3.5.
Revised: Lemma 3.5.

Original: Let { x k } k  0 be generated by prox - SGD and suppose that the assumptions (C.1) – (C.4)hold.
Revised: Let { x k } k  0 be generated by prox - SGD and suppose that the assumptions (C.1) – (C.4)hold.

Original: Then, for ✓ 2 (0 , [ 43 L + ⌧ ]  1 ) and all k with ↵ k  12 ⌧ , we have almost surely 
Revised: Then, for ✓ 2 (0 , [ 4 3 L + ⌧ ]  1 ) and all k with ↵ k  1 2 ⌧ , we have almost surely [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Phase I: Verifying (P.1)–(P.2).
Revised: Phase I: Verifying (P.1)–(P.2).

Original: In [19, Corollary 3.4], it is shown that the gradient of the Moreauenvelope is Lipschitz continuous with modulus L e := max { ✓  1 , (1  [ L + ⌧ ] ✓ )  1 [ L + ⌧ ] } for all✓ 2 (0 , [ L + ⌧ ]  1 ) .
Revised: In [20, Corollary 3.4], it is shown that the gradient of the Moreauenvelope is Lipschitz continuous with modulus L e := max { ✓  1 , (1  [ L + ⌧ ] ✓ )  1 [ L + ⌧ ] } for all✓ 2 (0 , [ L + ⌧ ]  1 ) .

Original: Thus, condition (P.1) is satisﬁed.
Revised: Thus, condition (P.1) is satisﬁed.

Original: Furthermore, due to ↵ k ! 0 and choosing ✓ 2 (0 , [3 L + ⌧ ]  1 ) , the estimate (11) in Lemma 3.4holds for all k sufﬁciently large.
Revised: Furthermore, due to ↵ k ! 0 and choosing ✓ 2 (0 , [3 L + ⌧ ]  1 ) , the estimate (11) in Lemma 3.4holds for all k sufﬁciently large.

Original: Consequently, due to env ✓ ( x )  (prox ✓ ( x ))  ¯ and (C.5),
Revised: Consequently, due to env ✓ ( x )  (prox ✓ ( x ))  ¯ and (C.5),

Original: Theorem B.1 is applicable and upon taking total expectation, { E [env ✓ ( x k )] } k  0 converges to some
Revised: Theorem B.1 is applicable and upon taking total expectation, { E [env ✓ ( x k )] } k  0 converges to some

Original: E 2 R .
Revised: E 2 R .

Original: In addition, the sequence { env ✓ ( x k ) } k  0 converges almost surely to some random variablee ?
Revised: In addition, the sequence { env ✓ ( x k ) } k  0 converges almost surely to some random variablee ?

Original: and we have
Revised: and we have

Revised: P

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This veriﬁes condition (P.2) with a = 2 .
Revised: This veriﬁes condition (P.2) with a = 2 .

Original: Phase II: Verifying (P.3)–(P.4) for showing convergence in expectation.
Revised: Phase II: Verifying (P.3)–(P.4) for showing convergence in expectation.

Original: Assumptions (C.1)–(C.5)and Lemma 3.5 allow us to establish the required bound stated in (P.3).
Revised: Assumptions (C.1)–(C.5)and Lemma 3.5 allow us to establish the required bound stated in (P.3).

Original: Speciﬁcally, taking totalexpectation in (12), we have
Revised: Speciﬁcally, taking totalexpectation in (12), we have

Original: [Equation]
Revised: [Equation]

Original: + 4(((2 L + C ) ✓ + 1) L 2 ' + D ) ↵ 2 k for all k sufﬁciently large.
Revised: 2 k for all k sufﬁciently large.

Original: Due to E [env ✓ ( x k )] !
Revised: Due to E [env ✓ ( x k )] !

Original: E , there exists F such that E [env ✓ ( x k )  ¯ ]  Ffor all k .
Revised: E , there exists F such that E [env ✓ ( x k )  ¯ ]  Ffor all k .

Original: Hence, (P.3) holds with q = 2 ,
Revised: Hence, (P.3) holds with q = 2 ,

Original: A = 8(

Revised: A = 8(

Original: L + C ) F + 4(((2 L + C ) ✓ + 1) L 2 ' + D ) , p 1 = 2 , and
Revised: L + C ) F + 4(((2 L + C ) ✓ + 1) L 2 ' + D ) , p 1 = 2 , and

Original: B = 0 .
Revised: B = 0 .

Original: The property (P.4) easily follows from (C.5) and the parameter choices.
Revised: The property (P.4) easily follows from (C.5) and the parameter choices.

Original: Consequently, using
Revised: Consequently, using

Original: Theorem 2.1, we can infer E [ kr env ✓ ( x k ) k ] ! 0 .
Revised: Theorem 2.1, we can infer E [ kr env ✓ ( x k ) k ] ! 0 .

Original: Phase III: Verifying (P.3 0 )–(P.4 0 ) for showing almost sure convergence.
Revised: Phase III: Verifying (P.3 0 )–(P.4 0 ) for showing almost sure convergence.

Original: We follow the constructionin (4) and set A k = ↵  1 k ( x k +1  x k  E [ x k +1  x k | F k ]) , B k = ↵  1 k E [ x k +1  x k | F k ] , andp 1 , p 2 = 1 .
Revised: We follow the construction in (4) and set A k = ↵  1 k ( x k +1  x k  E [ x k +1  x k | F k ]) , B k = ↵  1 k E [ x k +1  x k | F k ] ,and p 1 , p 2 = 1 .

Original: Clearly, we have E [ A k | F k ] = 

Revised: Clearly, we have E [ A k | F k ] =

Original: and based on the previous results in Phase II , we canshow E [ k x k +1  x k k 2 ]
Revised: and based on the previous results in Phase II , wecan show E [ k x k +1  x k k 2 ]

Original: =
Revised: =

Original: O ( ↵ 2 k ) which establishes boundedness of { E [ k A k k 2 } k  0 .
Revised: O ( ↵ 2 k ) which establishes boundedness of { E [ k A k k 2 } k  0 .

Original: Similarly,  for275 B k and by Lemma 3.5 and Jensen’s inequality, we obtain
Revised: Similarly, 277 for B k and by Lemma 3.5 and Jensen’s inequality, we obtain

Original: [Equation]
Revised: [Equation]

Original: Due to env ✓ ( x k ) !
Revised: Due to env ✓ ( x k ) !

Original: e ?
Revised: e ?

Original: almost surely, this shows lim sup k !
Revised: almost surely, this shows lim sup k !

Original: k B k k 2 < 1 almost surely.
Revised: k B k k 2 < 1 almost surely.

Original: Hence, 277 all requirements in (P.3 0 ) are satisﬁed with q = 2 and b = 0 .
Revised: Hence, 279 all requirements in (P.3 0 ) are satisﬁed with q = 2 and b = 0 .

Original: Moreover, it is easy to see that property(P.4 0 ) also holds in this case.
Revised: Moreover, it is easy to see that property(P.4 0 ) also holds in this case.

Original: Overall, Theorem 2.1 implies kr env ✓ ( x k ) k ! 

Revised: Overall, Theorem 2.1 implies kr env ✓ ( x k ) k !

Original: almost surely.
Revised: almost surely.

Original: As mentioned, it is possible to express the obtained convergence results in terms of the natural residual
Revised: As mentioned, it is possible to express the obtained convergence results in terms of the natural residual

Original: F nat = F 1nat , see, e.g., Lemma E.1.
Revised: F nat = F 1nat , see, e.g., Lemma E.1.

Original: We summarize our observations in the following corollary.
Revised: We summarize our observations in the following corollary.

Original: Corollary 3.6.
Revised: Corollary 3.6.

Original: Let us consider prox - SGD (9) for the compositeoptimization problem (8) undertheassumptions (C.1) – (C.5) .
Revised: Let us consider prox - SGD (9) for the composite problem (8) under (C.1) – (C.5) .

Original: Then, we have lim k !
Revised: Then,we have lim k !

Original: 1 E [ k F nat ( x k ) k ] = 0 and lim k !
Revised: 1 E [ k F nat ( x k ) k ] = 0 and lim k !

Original: 1 k F nat ( x k ) k = 0283 almost surely.
Revised: 1 k F nat ( x k ) k = 0 almost surely.

Original: Remark 3.7.
Revised: Remark 3.7.

Original: As a byproduct, Lemma 3.4 also leads to an expected iteration complexity result ofprox - SGD by using the ABC condition (C.4) rather than the standard bounded variance assumption.
Revised: As a byproduct, Lemma 3.4 also leads to an expected iteration complexity result ofprox - SGD by using the ABC condition (C.4) rather than the standard bounded variance assumption.

Original: This is a nontrivial extension of [9, Corollary 3.6].
Revised: This is a nontrivial extension of [10, Corollary 3.6].

Original: We provide a full derivation in Appendix E.6.
Revised: We provide a full derivation in Appendix E.6.

Original: 3.4 Convergence of stochastic model-based methods
Revised: 3.4 Convergence of stochastic model-based methods

Original: In this section, we consider the convergence of stochastic model-based methods for nonsmoothweakly convex optimization problems
Revised: In this section, we consider the convergence of stochastic model-based methods for nonsmoothweakly convex optimization problems

Original: [Equation]
Revised: [Equation]

Original: where both f and ' are assumed to be (nonsmooth) weakly convex functions and is lower bounded,i.e., ( x )  ¯ for all x 2 dom( ') .
Revised: where both f and ' are assumed to be (nonsmooth) weakly convex functions and is lower bounded,i.e., ( x )  ¯ for all x 2 dom ' .

Original: Classical stochastic optimization methods — including proximalstochastic subgradient, stochastic proximal point, and stochastic prox-linear methods — for solving(13) are uniﬁed by the stochastic model-based methods ( SMM ) [Equation]
Revised: Classical stochastic optimization methods — including proximalstochastic subgradient, stochastic proximal point, and stochastic prox-linear methods — for solving(13) are uniﬁed by the stochastic model-based methods ( SMM ) [13, 10]:

Original: [Equation]
Revised: [Equation]

Original: where f x k ( x , ⇠ k ) is a stochastic approximation of f around x k using the sample ⇠ k ; see Appendix F.1for descriptions of three major types of SMM .
Revised: where f x k ( x , ⇠ k ) is a stochastic approximation of f around x k using the sample ⇠ k ; see Appendix F.1for descriptions of three major types of SMM .

Original: Setting F k :=
Revised: Setting F k :=

Original:  ( ⇠ 0 , . .
Revised:  ( ⇠ 0 , . .

Original: , ⇠ k  1 ) , it is easy to see that{ x k } k  0 is adapted to {F k } k  0 .
Revised: , ⇠ k  1 ) , it is easy to see that{ x k } k  0 is adapted to {F k } k  0 .

Original: We analyze convergence of SMM under the following assumptions.
Revised: We analyze convergence of SMM under the following assumptions.

Original: (D.1)
Revised: (D.1)

Original: The stochastic approximation function f x satisﬁes a one-sided accuracy property,
Revised: The stochastic approximation function f x satisﬁes a one-sided accuracy property,

Original: i
Revised: i

Original: e., we
Revised: e., we

Original: have E ⇠
Revised: have E ⇠

Original: [ f x ( x , ⇠ )]
Revised: [ f x ( x , ⇠ )]

Original: = f
Revised: = f

Original: ( x ) for all x 2 U and
Revised: ( x ) for all x 2 U and

Original: [Equation]
Revised: [Equation]

Original: where U is an open convex set containing dom ' .
Revised: where U is an open convex set containing dom ' .

Original: (D.2)
Revised: (D.2)

Original: The function y 7!
Revised: The function y 7!

Original: f x ( y , ⇠ ) + ' ( y ) is ⌘ -weakly convex for all x 2 U and almost every ⇠ .
Revised: f x ( y , ⇠ ) + ' ( y ) is ⌘ -weakly convex for all x 2 U and almost every ⇠ .

Original: (D.3)
Revised: (D.3)

Original: There exists L > 0 such that the stochastic approximation function f x satisﬁes
Revised: There exists L > 0 such that the stochastic approximation function f x satisﬁes

Original: [Equation]
Revised: [Equation]

Original: and almost every ⇠
Revised: and almost every ⇠

Original: [Equation]
Revised: [Equation]

Original: (D.4)
Revised: (D.4)

Original: The function ' is L ' -Lipschitz continuous. (D.5)
Revised: The function ' is L ' -Lipschitz continuous.(D.5)

Original: The step sizes { ↵ k } k  0 satisfy
Revised: The step sizes { ↵ k } k  0 satisfy

Original: [Equation]
Revised: [Equation]

Revised: k =0 ↵ k = 1 and

Revised: [Equation]

Original: Assumptions (D.1), (D.2), (D.3) are standard for analyzing SMM and identical to that of [9].
Revised: Assumptions (D.1), (D.2), (D.3) are standard for analyzing SMM and identical to that of [10].

Original: (D.5) isconvention for stochastic methods.
Revised: (D.5) isconvention for stochastic methods.

Original: Assumption (D.4) mimics (C.3); see Remark 3.3 for discussions.
Revised: Assumption (D.4) mimics (C.3); see Remark 3.3 for discussions.

Original: We now derive the convergence of SMM below by setting  ⌘ r env ✓ and µ k ⌘ ↵ k .
Revised: We now derive the convergence of SMM below by setting  ⌘ r env ✓ and µ k ⌘ ↵ k .

Original: Our derivation 307 is based on the following two estimates, in which the proof of Lemma 3.9 is given in Appendix F.2.
Revised: Our derivation 308 is based on the following two estimates, in which the proof of Lemma 3.9 is given in Appendix F.2.

Original: Lemma 3.8 (Theorem 4.3 of [9]) .
Revised: Lemma 3.8 (Theorem 4.3 of [10]) .

Original: Let ✓ 2 (0 , ( ⌧ + ⌘ )  1 ) and ↵ k < ✓ be given.
Revised: Let ✓ 2 (0 , ( ⌧ + ⌘ )  1 ) and ↵ k < ✓ be given.

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: Lemma 3.9.
Revised: Lemma 3.9.

Original: For all k with ↵ k  1 / (2 ⌘ ) , it holds that
Revised: For all k with ↵ k  1 / (2 ⌘ ) , it holds that

Original: [Equation]
Revised: [Equation]

Original: Phase I: Verifying (P.1)–(P.2).
Revised: Phase I: Verifying (P.1)–(P.2).

Original: As before, [19, Corollary 3.4] implies that the mapping r env ✓ is
Revised: As before, [20, Corollary 3.4] implies that the mapping r env ✓ is

Original: Lipschitz continuous for all ✓ 2 (0 , ( ⌧ + ⌘ )  1 ) Hence, condition (P.1) is satisﬁed.
Revised: Lipschitz continuous for all ✓ 2 (0 , ( ⌧ + ⌘ )  1 ) Hence, condition (P.1) is satisﬁed.

Original: Using ↵ k ! 0 ,we can apply Theorem B.1 to the recursion obtained in Lemma 3.8 for all k sufﬁciently large and itfollows
Revised: Using ↵ k ! 0 ,we can apply Theorem B.1 to the recursion obtained in Lemma 3.8 for all k sufﬁciently large and itfollows

Original: P 1 k =0 ↵ k E
Revised: P 1 k =0 ↵ k E

Original: [ kr env ✓ ( x k ) k 2 ] < 1 .
Revised: [ kr env ✓ ( x k ) k 2 ] < 1 .

Original: Thus, condition (P.2) holds with a = 2 .
Revised: Thus, condition (P.2) holds with a = 2 .

Original: Phase II: Verifying (P.3)–(P.4) for showing convergence in expectation.
Revised: Phase II: Verifying (P.3)–(P.4) for showing convergence in expectation.

Original: Taking total expectation in Lemma 3.9 veriﬁes condition (P.3) with q = 2 , A
Revised: Taking total expectationin Lemma 3.9 veriﬁes condition (P.3) with q = 2 , A

Original: = (16( L + L ' ) 2 + 8 L 2 ) , p 1 = 2 , B = 0 .
Revised: = (16( L + L ' ) 2 + 8 L 2 ) , p 1 = 2 , B = 0 .

Original: Moreover,condition (P.4) is true by assumption (D.5) and the previous parameters choices.
Revised: Moreover, condition (P.4) is true by assumption (D.5) and the previous parameters choices.

Original: Thus, applying Theorem 2.1 gives E [ kr env ✓ ( x k ) k ] ! 0 .
Revised: Thus,applying Theorem 2.1 gives E [ kr env ✓ ( x k ) k ] ! 0 .

Original: Phase III: Verifying (P.3 0 )–(P.4 0 ) for showing almost sure convergence.
Revised: Phase III: Verifying (P.3 0 )–(P.4 0 ) for showing almost sure convergence.

Original: As in (4), we can set
Revised: As in (4), we can set

Original: A k = ↵  1 k ( x k +1  x k  E [ x k +1  x k | F k ]) , B k = ↵  1 k E [ x k +1  x k | F k ] .
Revised: A k = ↵  1 k ( x k +1  x k  E [ x k +1  x k | F k ]) , B k = ↵  1 k E [ x k +1  x k | F k ] .

Original: Applying Lemma 3.9and utilizing Jensen’s inequality, we have E [ A k | F k ] = 0 , E
Revised: Applying Lemma 3.9and utilizing Jensen’s inequality, we have E [ A k | F k ] = 0 , E

Original: [ k A k k 2 ]  (4 / ↵ 2 k ) E
Revised: [ k A k k 2 ]  (4 / ↵ 2 k ) E

Original: [ k x k +1  x k k 2 ]  4(16( L + L ' ) 2 + 8 L 2 ) and k B k k 2  16( L + L ' ) 2 + 8 L 2 .
Revised: [ k x k +1  x k k 2 ] 4(16( L + L ' ) 2 + 8 L 2 ) and k B k k 2  16( L + L ' ) 2 + 8 L 2 .

Original: Thus, condition (P.3 0 ) is satisﬁed withp 

Revised: Thus, condition (P.3 0 ) is satisﬁed withp

Original: = p 2 = 1 , q = 2 .
Revised: = p 2 = 1 , q = 2 .

Original: Assumption (D.5), together with the previous parameters choices veriﬁescondition (P.4 0 ) and hence, applying Theorem 2.1 yields kr env ✓ ( x k ) k ! 0 almost surely.
Revised: Assumption (D.5), together with the previous parameters choices veriﬁescondition (P.4 0 ) and hence, applying Theorem 2.1 yields kr env ✓ ( x k ) k ! 0 almost surely.

Original: Summarizing this discussion, we obtain the following convergence results for SMM .
Revised: Summarizing this discussion, we obtain the following convergence results for SMM .

Original: Corollary 3.10.
Revised: Corollary 3.10.

Original: We consider the family of stochastic model-based methods (14) for the optimizationproblem (13) under assumptions (D.1) – (D.5) .
Revised: We consider the family of stochastic model-based methods (14) for the optimizationproblem (13) under assumptions (D.1) – (D.5) .

Original: Let { x k } k  0 be a generated sequence.
Revised: Let { x k } k  0 be a generated sequence.

Original: Then, we havelim k !
Revised: Then, we havelim k !

Original: 1 E [ kr env ✓ ( x k ) k ] = 0 and lim k !
Revised: 1 E [ kr env ✓ ( x k ) k ] = 0 and lim k !

Original: 1 kr env ✓ ( x k ) k
Revised: 1 kr env ✓ ( x k ) k

Original: = 0 almost surely.
Revised: = 0 almost surely.

Revised: Remark 3.11.

Revised: The results presented in Corollary 3.10 also hold under certain extended settings.

Revised: Infact, we can replace (D.3) by a slightly more general Lipschitz continuity assumption on f .

Revised: Moreover,it is possible to establish convergence in the case where f is not Lipschitz continuous but has Lipschitzcontinuous gradient, which is particularly useful when we apply stochastic proximal point method forsmooth f .

Revised: A more detailed derivation and discussion of such extensions is deferred to Appendix F.3.

Original: 3.5 Related work and discussion
Revised: 3.5 Related work and discussion

Original: SGD and RR.
Revised: SGD and RR.

Original: The literature for SGD is extremely rich and several connected and recent works havebeen discussed in Section 1.
Revised: The literature for SGD is extremely rich and several connected and recent works havebeen discussed in Section 1.

Original: Our result in Corollary 3.1 uniﬁes many of the existing convergenceanalyses of SGD and is based on the general ABC condition (A.3) (see [20, 21, 17] for comparison)rather than on the standard bounded variance assumption.
Revised: Our result in Corollary 3.1 uniﬁes many of the existing convergenceanalyses of SGD and is based on the general ABC condition (A.3) (see [21, 22, 18] for comparison)rather than on the standard bounded variance assumption.

Original: Our expected convergence result generalizesthe one in [5] using much weaker assumptions.
Revised: Our expected convergence result generalizesthe one in [6] using much weaker assumptions.

Original: Our results for RR are in line with the recent theoreticalobservations in [26, 28, 22].
Revised: Our results for RR are in line with the recent theoreticalobservations in [28, 30, 23].

Original: In particular, Corollary 3.2 recovers the almost sure convergence resultshown in [22], while the expected convergence result appears to be new.
Revised: In particular, Corollary 3.2 recovers the almost sure convergence resultshown in [23], while the expected convergence result appears to be new.

Original: Prox-SGD and SMM.
Revised: Prox-SGD and SMM.

Original: The work [9] established one of the ﬁrst complexity results for prox - SGDusing the Moreau envelope.
Revised: The work [10] established one of the ﬁrst complexity results for prox - SGDusing the Moreau envelope.

Original: Under a bounded variance assumption ( C = 0 in condition (C.4)) andfor general nonconvex and smooth f , the authors showed E [ kr env ✓ ( x ¯ k ) k 2 ]
Revised: Under a bounded variance assumption ( C = 0 in condition (C.4)) andfor general nonconvex and smooth f , the authors showed E [ kr env ✓ ( x ¯ k ) k 2 ]

Original: =
Revised: =

Original: O (( T + 1)  1 / 2 ) ,where x ¯ k is sampled uniformly from the past T + 1 iterates x 0 , . . .
Revised: O (( T + 1)  1 / 2 ) ,where x ¯ k is sampled uniformly from the past T + 1 iterates x 0 , . . .

Original: , x T .
Revised: , x T .

Original: As mentioned, this resultcannot be easily extended to the asymptotic convergence results discussed in this paper.
Revised: As mentioned, this resultcannot be easily extended to the asymptotic convergence results discussed in this paper.

Original: Earlierstudies of prox - SGD for nonconvex f and C = 0 include [16] where convergence of prox - SGD isestablished if the variance parameter D
Revised: Earlierstudies of prox - SGD for nonconvex f and C = 0 include [17] where convergence of prox - SGD isestablished if the variance parameter D

Original: =
Revised: =

Original: D k ! 

Revised: D k !

Original: vanishes as k ! 1 .
Revised: vanishes as k ! 1 .

Original: This can be achieved by 343 progressively increasing the size of the selected mini-batches or via variance reduction techniques asin prox - SVRG and prox - SAGA , see [31].
Revised: This can be achieved by 349 progressively increasing the size of the selected mini-batches or via variance reduction techniques asin prox - SVRG and prox - SAGA , see [33].

Original: The question whether prox - SGD can converge and whetherthe accumulation points of the iterates { x k } k  0 correspond to stationary points was only addressed 346 recently in [24].
Revised: The question whether prox - SGD can converge and whetherthe accumulation points of the iterates { x k } k  0 correspond to stationary points was only addressed 352 recently in [25].

Original: The authors use a differential inclusion approach to study convergence of prox - SGD .
Revised: The authors use a differential inclusion approach to study convergence of prox - SGD .

Original: However, additional compact constraints x 2 X have to be introduced in the model (8) to guaranteesure boundedness of { x k } k  0 and applicability of the differential inclusion techniques.
Revised: However, additional compact constraints x 2 X have to be introduced in the model (8) to guaranteesure boundedness of { x k } k  0 and applicability of the differential inclusion techniques.

Original: Lipschitz 349 continuity of ' also appears as an essential requirement in [24, Theorem 5.4].
Revised: Lipschitz 355 continuity of ' also appears as an essential requirement in [25, Theorem 5.4].

Original: The analyses in [12, 10]establish asymptotic convergence guarantees for SMM .
Revised: The analyses in [13, 11]establish asymptotic convergence guarantees for SMM .

Original: However, both works require a priori (almost)sure boundedness of { x k } k  0 and a density / Sard-type condition in order to show convergence.
Revised: However, both works require a priori (almost)sure boundedness of { x k } k  0 and a density / Sard-type condition in order to show convergence.

Original: Werefer to [14] for an extension of the results in [24, 10] to prox - SGD in Hilbert spaces.
Revised: Werefer to [15] for an extension of the results in [25, 11] to prox - SGD in Hilbert spaces.

Original: By contrast, ourconvergence framework allows to complement these differential inclusion-based results and — for theﬁrst time — fully removes any stringent boundedness assumption on { x k } k  0 .
Revised: By contrast, ourconvergence framework allows to complement these differential inclusion-based results and — for theﬁrst time — fully removes any stringent boundedness assumption on { x k } k  0 .

Original: Instead, our analysis 355 relies on more transparent assumptions that are veriﬁable and common in stochastic optimizationand machine learning.
Revised: Instead, our analysis 361 relies on more transparent assumptions that are veriﬁable and common in stochastic optimizationand machine learning.

Original: In summary, we are now able to claim: prox - SGD and SMM converge understandard stochastic conditions if ' is Lipschitz continuous .
Revised: In summary, we are now able to claim: prox - SGD and SMM converge understandard stochastic conditions if ' is Lipschitz continuous .

Original: In the easier convex case, analogousresults have been obtained, e.g., in [16, 1, 36].
Revised: In the easier convex case, analogousresults have been obtained, e.g., in [17, 1, 38].

Revised: We provide an overview of several related and representative results in Table 1 in Appendix G.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we provided a novel convergence framework that allows to derive expected and almostsure convergence results for a vast class of stochastic optimization methods under state-of-the-artassumptions and in a uniﬁed way.
Revised: In this work, we provided a novel convergence framework that allows to derive expected and almostsure convergence results for a vast class of stochastic optimization methods under state-of-the-artassumptions and in a uniﬁed way.

Original: We speciﬁed the steps on how to utilize our theorem in order toestablish convergence results for a given stochastic algorithm.
Revised: We speciﬁed the steps on how to utilize our theorem in order toestablish convergence results for a given stochastic algorithm.

Original: As concrete examples, we applied ourtheorem to derive asymptotic convergence guarantees for SGD , RR , prox - SGD , and SMM .
Revised: As concrete examples, we applied ourtheorem to derive asymptotic convergence guarantees for SGD , RR , prox - SGD , and SMM .

Original: To oursurprise, some of the obtained results appear to be new and provide new insights into the convergencebehavior of some well-known and standard stochastic methodologies.
Revised: To oursurprise, some of the obtained results appear to be new and provide new insights into the convergencebehavior of some well-known and standard stochastic methodologies.

Original: These applications revealedthat our uniﬁed theorem can serve as a plugin-type tool with the potential to facilitate the convergenceanalysis of a wide class of stochastic optimization methods.
Revised: These applications revealedthat our uniﬁed theorem can serve as a plugin-type tool with the potential to facilitate the convergenceanalysis of a wide class of stochastic optimization methods.

Original: Finally, it is important to investigate in which situations our convergence results in terms of thestationarity measure  can be strengthened — say to almost sure convergence guarantees for theiterates { x k } k  0 .
Revised: Finally, it is important to investigate in which situations our convergence results in terms of thestationarity measure  can be strengthened — say to almost sure convergence guarantees for theiterates { x k } k  0 .

Original: We plan to consider such a possible extension in future work.
Revised: We plan to consider such a possible extension in future work.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...(

Revised: a

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and 
Revised: ) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?

Original: scope?

Original: Yes]
Revised: [Yes](

Original: (b) Did you describe the limitations of your work?
Revised: b ) Did you describe the limitations of your work?

Original: [Yes] The limitations are written in equivalent forms as future works in the conclusion section; see Section 
Revised: [Yes] The limitations are written inequivalent forms as future works in the conclusion section see Section 4.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] We con- duct theoretical investigation about the fundamental stochastic optimization methods, which will not bring any negative societal impact.
Revised: [N/A] We conduct theoretical investigation about the fundamental stochastic optimization methods,which will not bring any negative societal impact.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: d) Have you read the ethics review guidelines and ensured that your paper conforms tothem

Original: [Yes]
Revised: [Yes

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] We put them in the
Revised: [Yes] We put them in the

Original: appendix.
Revised: appendix.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [N/A]
Revised: [N/A]

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review 
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review [Table]

Original: Board (IRB) approvals, if applicable?
Revised: Board IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: ) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A]
Revised: [N/A]