Original: Global Convergence and Stability of Stochastic Gradient Descent
Revised: Global Convergence and Stability of Stochastic Gradient Descent

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: In machine learning, stochastic gradient descent (SGD) is widely deployed to trainmodels using highly non-convex objectives with equally complex noise models.
Revised: In machine learning, stochastic gradient descent (SGD) is widely deployed to train models using highly non-convex objectives with equally complex noise models.

Original: Unfortunately, SGD theory often makes restrictive assumptions that fail to capturethe non-convexity of real problems, and almost entirely ignore the complex noisemodels that exist in practice.
Revised: Unfortunately, SGD theory often makes restrictive assumptions that fail to capture the non-convexity of real problems, and almost entirely ignore the complex noise models that exist in practice.

Original: In this work, we demonstrate the restrictivenessof these assumptions using three canonical models in machine learning, then wedevelop novel theoretical tools to address this shortcoming in two ways.
Revised: In this work, we demonstrate the restrictiveness of these assumptions using three canonical models in machine learning. Then, we develop novel theory to address this shortcoming in two ways.

Original: First, weestablish that SGD’s iterates will either globally converge to a stationary point ordiverge under nearly arbitrary nonconvexity and noise models.
Revised: First, we establish that SGD’s iterates will either globally converge to a stationary point or diverge under nearly arbitrary nonconvexity and noise models.

Original: Under a slightlymore restrictive assumption on the joint behavior of the non-convexity and noisemodel that generalizes current assumptions in the literature, we show that theobjective function cannot diverge, even if the iterates diverge.
Revised: Under a slightly more restrictive assumption on the joint behavior of the non-convexity and noise model that generalizes current assumptions in the literature, we show that the objective function cannot diverge, even if the iterates diverge.

Original: As a consequenceof our results, SGD can be applied to a greater range of stochastic optimizationproblems with conﬁdence about its global convergence behavior and stability.
Revised: As a consequence of our results, SGD can be applied to a greater range of stochastic optimization problems with conﬁdence about its global convergence behavior and stability.

Original:  Introduction
Revised:  Introduction

Original: Stochastic Gradient Descent (SGD) and its variants are dominant algorithms for solving stochasticoptimization problems arising in machine learning, and have expanded their reach to more complexproblems from estimating Gaussian Processes [Chen et al., 2020], covariance estimation in stochasticﬁlters [Kim et al., 2021] and systems identiﬁcation [Hardt et al., 2016, Zhang and Patel, 2020].
Revised: Stochastic Gradient Descent (SGD) and its variants are dominant algorithms for solving stochastic optimization problems arising in machine learning, and have expanded their reach to more complex problems from estimating Gaussian Processes [Chen et al., 2020], covariance estimation in stochastic ﬁlters [Kim et al., 2021], and systems identiﬁcation [Hardt et al., 2016, Zhang and Patel, 2020].

Original: Accordingly, understanding the behavior of SGD and its variants has been crucial to their reliableapplication in machine learning and beyond.
Revised: Accordingly, understanding the behavior of SGD and its variants has been crucial to their reliable application in machine learning and beyond.

Original: As a result, the theory of these methods has greatlyadvanced, most notably for SGD as it is the basis for, and simplest of, these methods.
Revised: As a result, the theory of these methods has greatly advanced, most notably for SGD as it is the basis for, and simplest of, these methods.

Original: Indeed, SGDhas been analyzed from many perspectives: global convergence analysis [Lei et al., 2019, Goweret al., 2020, Khaled and Richtárik, 2020, Mertikopoulos et al., 2020, Patel, 2021], local convergenceanalysis [Mertikopoulos et al., 2020], greedy and global complexity analysis [Gower et al., 2020,
Revised: Indeed, SGD has been analyzed from many perspectives: global convergence analysis [Lei et al., 2019, Gower et al., 2020, Khaled and Richtárik, 2020, Mertikopoulos et al., 2020, Patel, 2021], local convergence analysis [Mertikopoulos et al., 2020], greedy and global complexity analysis [Gower et al., 2020,

Original: Khaled and Richtárik, 2020], asymptotic weak convergence [Wang et al., 2021], and saddle pointanalysis [Fang et al., 2019, Mertikopoulos et al., 2020, Jin et al., 2021].
Revised: Khaled and Richtárik, 2020], asymptotic weak convergence [Wang et al., 2021], and saddle point analysis [Fang et al., 2019, Mertikopoulos et al., 2020, Jin et al., 2021].

Original: While all of these perspectives add new dimensions to our understanding of SGD, the global convergence analysis of SGD is the foundation as it dictates whether local analyses, complexity analysesor saddle point analyses are even warranted.
Revised: While all of these perspectives add new dimensions to our understanding of SGD, the global convergence analysis of SGD is the foundation as it dictates whether local analyses, complexity analyses or saddle point analyses are even warranted.

Original: As surveyed in Patel [2021], these current globalconvergence analyses of SGD make a wide variety of assumptions, most commonly: (1) the objectivefunction is bounded from below, (2) the gradient function is globally Lipschitz continuous, (3) thestochastic gradients are unbiased, and (4) the variance of the stochastic gradients are bounded.
Revised: As surveyed in Patel [2021], these current global convergence analyses of SGD make a wide variety of assumptions, most commonly: (1) the objective function is bounded from below, (2) the gradient function is globally Lipschitz continuous, (3) the stochastic gradients are unbiased, and (4) the variance of the stochastic gradients are bounded.

Original: Whilethe ﬁrst and third assumption are generally reasonable, 1 the second and fourth assumptions  are not generally applicable to machine learning problems as we now demonstrate through three simpleexamples (see Appendix A for details). Example 1: Linear Regression.
Revised: While the ﬁrst and third assumption are generally reasonable, 1 the second and fourth assumptions and their more recent generalizations are not usually applicable to machine learning problems as we now demonstrate through three simple examples.

Original: Consider ﬁtting a linear regression model using squared-error lossgiven independent copies of the example ( Y, Z ) where for some θ ∗ and error term (cid:15) , Y = (cid:104) Z, θ ∗ (cid:105) + (cid:15) . The next results states that for a very nice linear regression problem, the stochastic gradients violatethe bounded variance assumption. Proposition 1.
Revised: Note, in the ﬁrst two examples, we make use of a penalty function, which can be removed without impacting the result.

Original: There exist random variables Z and (cid:15) such that Z has bounded fourth moment, and(cid:15) is mean zero with bounded variance such that the variance of the stochastic gradients evaluated atθ diverges to inﬁnity as (cid:107) θ (cid:107) 2 → ∞ .

Original: Example 2: Feed Forward Neural Network.
Revised: Example 1: Feed Forward Neural Network.

Original: Consider the example ( Y, Z ) where Y is a binarylabel and Z is a feature vector.
Revised: Consider the example ( Y, Z ) where Y is a binary label and Z is a feature vector.

Original: We will attempt to predict Y from Z using a simple multi-layer feedforward network as shown in Fig.
Revised: We will attempt to predict Y from Z using a simple multi-layer feed forward network as shown in Fig.

Original: The next result states that for a simple distribution over theexample space and for a simple, archetype network, the gradient function is not globally Lipschitzcontinuous and the stochastic gradient does not have globally bounded variance.
Revised: The next result states that for a simple distribution over the example space and for a simple, archetype network, the gradient function is not globally Lipschitz continuous, nor does it satisfy the (possibly) more general ( L 0 , L 1 ) -smooth assumption [Zhang et al., 2019, Deﬁnition 1, Assumption 3]. Moreover, the variance of the stochastic gradients is unbounded. See Appendix A.2 for a proof.

Original: [Equation]
Revised: [Figure]

Original: Proposition 2.
Revised: Proposition 1.

Original: Consider the feed forward network in Fig.
Revised: Consider the feed forward network in Fig.

Original: 1 with σ linear and ϕ sigmoid trainedwith a binary cross-entropy loss with a Ridge penalty.
Revised: 1 with σ linear and ϕ sigmoid trained with a binary cross-entropy loss with a Ridge penalty.

Original: There exists a ﬁnite, discrete distribution for( Y, Z ) such that the risk function’s gradient is not globally Lipschitz continuous, nor is the varianceof the stochastic gradients  bounded.
Revised: There exists a ﬁnite, discrete distribution for ( Y, Z ) such that the risk function’s gradient is not globally Lipschitz continuous, nor does it satisfy the ( L 0 , L 1 ) -smooth assumption. Moreover, the variance of the stochastic gradients is not bounded.

Original: Example 3: Recurrent Neural Network.
Revised: Example 2: Recurrent Neural Network.

Original: Consider the example ( Y, Z 0 , Z 1 , Z 2 , Z 3 ) where Y is abinary label and { Z 0 , Z 1 , Z 2 , Z 3 } are sequentially observed.
Revised: Consider the example ( Y, Z 0 , Z 1 , Z 2 , Z 3 ) where Y is a binary label and { Z 0 , Z 1 , Z 2 , Z 3 } are sequentially observed.

Original: We will attempt to predict Y from Zusing a simple recurrent network as shown in Fig.
Revised: We will attempt to predict Y from Z using a simple recurrent network as shown in Fig.

Original: The next result states that for a simple distributionover the example space and for a simple, archetype network, the gradient function violatesglobal Lipschitz continuity and the stochastic gradient violates the bounded variance assumption. [Equation]
Revised: The next result states that for a simple distribution over the example space and for a simple, archetype network, the training function violates the aforementioned assumptions.

Original: [Figure] [Equation] [Figure] [Equation]
Revised: See Appendix A.3 for a proof.

Original: [Figure]
Revised: [Figure]

Original: Proposition 3. Consider the recurrent network in Fig.
Revised: Proposition 2. Consider the recurrent network in Fig.

Original: 1 with σ linear and ϕ sigmoid trained witha binary cross-entropy loss with a Ridge penalty.
Revised: 2 with σ linear and ϕ sigmoid trained with a binary cross-entropy loss with a Ridge penalty.

Original: There exists a ﬁnite, discrete distribution for( Y, Z 0 , Z 1 , Z 2 , Z 3 ) such that the risk function’s gradient is not globally Lipschitz continuous, nor isthe variance of the stochastic gradients  bounded.
Revised: There exists a ﬁnite, discrete distribution for ( Y, Z 0 , Z 1 , Z 2 , Z 3 ) such that the risk function’s gradient is not globally Lipschitz continuous, nor does it satisfy the ( L 0 , L 1 ) -smooth assumption. Moreover, the variance of the stochastic gradients is not bounded.

Original: As these three examples show, global convergence analyses that make use of the aforementionedassumptions do not apply to these canonical examples of machine learning problems.
Revised: Example 3: Poisson Regression. Consider ﬁtting a Poisson regression model by the standard maximum likelihood framework using independent copies of the example ( Y, Z ) , where Y is a count response variable and Z is a predictor.

Original: In fact, whilethe linear regression case can be covered by more recent analyses (see Bottou et al.
Revised: The next result states that for a very nice Poisson regression problem, the stochastic gradients violate the bounded variance assumption, its generalization [Bottou et al., 2018, Assumption 4.3c], and, in turn, its generalization, expected smoothness [Khaled and Richtárik, 2020, Assumption 2]. See Appendix A.4 for a proof. Proposition 3.

Original: [2018]), to our
Revised: Let Y and Z be independent Poisson random variables with mean one.

Original: Assumption Works Reddi et al. [2016a], Ma and Klabjan [2017], Zhou et al.
Revised: Consider estimating a Poisson regression model of Y as a function of Z .

Original: Bassily et al. [2018], Lei et al. [2019], Li and Orabona [2019], Gower et al.
Revised: Then, the gradient function is not globally Lipschitz continuous, nor does it satisfy the ( L 0 , L 1 ) -smooth assumption.

Original: [2020], Khaled and Richtárik [2020], Mertikopoulos et al. [2020], Patel [2021], Jin et al. [2021], Wang et al. Reddi et al.
Revised: Moreover, the variance of the stochastic gradients is not bounded, does not satisfy [Bottou et al., 2018, Assumption 4.3c], nor does it satisfy [Khaled and Richtárik, 2020, Assumption 2].

Original: [2016b], Ma and Klabjan [2017], Majewski et al. [2018], Hu et al. [2019], Bi and Gunn [2019], Zou et al. Mertikopoulos et al.
Revised: As these three examples show, global convergence analyses that make use of the aforementioned assumptions do not apply to these canonical examples of machine learning problems.

Original: [2020].knowledge and as summarized  Table 1, there are no global convergence analyses of SGD that applyto the neural network training examples.
Revised: In fact, to our knowledge and as summarized in Table 1, there are no global convergence analyses of SGD that apply to these examples.

Original: The Problem:
Revised: The Problem:

Original: As a result, we do not have guarantees about how SGD behaves on these simple neural network problems, which calls into question what SGD and its variants are doing on more complicated machine learning models.
Revised: As a result, we do not have guarantees about how SGD behaves on these simple machine learning problems, which calls into question what SGD and its variants are doing on more complicated machine learning models.

Original: Contributions.
Revised: Contributions.

Original: To address this problem, 
Revised: To address this problem, 1.

Original: We relax the global Lipschitz continuous gradient assumption and the bounded variance assump- tion to assumptions that are applicable to the examples above.Speciﬁcally, we relax the global
Revised: We relax the global Lipschitz continuous gradient assumption, the bounded variance assumption, and their aforementioned generalizations to assumptions that are applicable to the examples above.

Original:  Lipschitz continuity assumption  to local α -Hölder continuity of the gradient for α ∈ (0 , 1] (see Assumption 2), which is even a relaxation even for deterministic gradient algorithms (c.f. Nocedal and Wright [2006],Theorems 3.2, 3.8, 4.5, 4.6).
Revised: Speciﬁcally, we relax the global Lipschitz continuity assumption and the ( L 0 , L 1 ) -smooth assumption to local α -Hölder continuity of the gradient for α ∈ (0 , 1] (see Assumption 2), which is even a relaxation even for deterministic gradient algorithms (c.f. Nocedal and Wright [2006],Theorems 3.2, 3.8, 4.5, 4.6).

Original: For the α in the local Hölder assumption, we also relax the bounded variance assumption to only require that the (1 + α ) -moment of the stochastic gradient is bounded by an arbitrary upper semi-continuous function (see Assumption 
Revised: For the α in the local Hölder assumption, we also relax the bounded variance assumption to only require that the (1 + α ) -moment of the stochastic gradient is bounded by an arbitrary upper semi-continuous function ( see Assumption 4).

Original: Ourassumption allows stochastic gradients whose noise may not have a variance.
Revised: Our assumption allows stochastic gradients whose noise may not have a variance.

Original: Moreover, ourassumption generalizes the noise assumption of Bottou et al.
Revised: Moreover, our assumption generalizes the noise assumption of Bottou et al.

Original: [2018], the expected smoothnessassumption of Gower et al.
Revised: [2018], the expected smoothness assumption of Gower et al.

Original: [2020] and Khaled and Richtárik [2020], and the noise assumptionof Asi and Duchi [2019].
Revised: [2020] and Khaled and Richtárik [2020], and the noise assumption of Asi and Duchi [2019].

Original: We also point out that we do not require the coercivity or asymptoticﬂatness assumptions that are commonly considered in the analysis of SGD (e.g., Mertikopouloset al.
Revised: We also point out that we do not require the coercivity or asymptotic ﬂatness assumptions that are commonly considered in the analysis of SGD (e.g., Mertikopoulos et al.

Original: [2020], Assumptions 2 and 3).2.
Revised: [2020], Assumptions 2 and 3).

Original: Owing to the relaxation in the assumptions, we cannot apply the standard analysis as the local Hölder constant and the iterate difference are conditionally dependent random variables (see thediscussion after Lemma 1).
Revised: Owing to the relaxation in the assumptions, we cannot apply the standard analysis as the local Hölder constant and the iterate difference are conditionally dependent random variables (see the discussion after Lemma 1).

Original: As a result,  we innovate a new analysis strategy (see Section 4.1) toprove that, with probability one, either SGD’s iterates will converge to a stationary point or theydiverge (see Theorem 2).
Revised: As a result, by generalizing our previous techniques in Patel [2021], Patel and Zhang [2021] to the α -Hölder continuous setting, we innovate a new analysis strategy (see Section 4.1) to prove that, with probability one, either SGD’s iterates will converge to a stationary point or they diverge (see Theorem 2).

Original: Importantly, our new analysis strategy can be broadly applied even todeterministic algorithms to relax the assumptions found in the literature.The divergence component
Revised: Importantly, our new analysis strategy can be broadly applied even to deterministic algorithms to relax the assumptions found in the literature.

Original:  of our Theorem  is somewhat disconcerting as we cannot say exactlywhat happens if the iterates diverge.
Revised: The divergence component of our Theorem 2 is somewhat disconcerting as we cannot say exactly what happens if the iterates diverge.

Original: To ameliorate this concern, we add an additional assumption(see Assumption 5) and introduce another analysis strategy (see Section 4.2) to prove, even if theiterates are diverging, the objective function converges to a ﬁnite random variableand the gradientfunction converges to zero with probability one (see Theorem 3).
Revised: To ameliorate this concern, we add an additional assumption (see Assumption 5) and introduce another analysis strategy (see Section 4.2) to prove, even if the iterates are diverging, the objective function converges to a ﬁnite random variable with probability one (see Theorem 3).

Original: Our results do not supply a rate of convergence as this is impossible for the broad class of nonconvexfunctions and the generality of the noise models studied in this work
Revised: Our results do not supply a rate of convergence as this is impossible for the broad class of nonconvex functions and the generality of the noise models studied in this work

Original: [Wolpert and Macready, 1997];in other words, we can always construct a nonconvex objective function, a noise process, and choosean initialization such that any rate of convergence statement will be violated.
Revised: [Wolpert and Macready, 1997]; in other words, we can always construct a nonconvex objective function, a noise process and choose an initialization such that any rate of convergence statement will be violated.

Original: Indeed, we ﬁnd itremarkable that it is even possible to provide a global convergence statement for such a broad classof nonconvex functions and general noise models
Revised: Indeed, we ﬁnd it remarkable that it is even possible to provide a global convergence statement for such a broad class of nonconvex functions and general noise models. Assumption

Revised: Works Depending on the Assumption

Original: Limitations.
Revised: Global Lipschitz or Hölder Continuity of Gradient Reddi et al.

Revised: [2016a], Ma and Klabjan [2017], Zhou et al.

Revised: Bassily et al.

Revised: [2018], Lei et al.

Revised: [2019], Li and Orabona [2019],

Revised: Gower et al.

Revised: [2020], Khaled and Richtárik [2020], Mertikopoulos et al.

Revised: [2020], Patel [2021], Jin et al.

Revised: [2021], Wang et al.

Revised: Bounded Variance

Revised: Reddi et al.

Revised: [2016b], Ma and Klabjan [2017], Majewski et al.

Revised: [2018], Hu et al.

Revised: [2019], Bi and Gunn [2019], Zou et al.

Revised: Mertikopoulos et al.

Revised: Limitations.

Original: We make note of two important limitations in our work.
Revised: We make note of two important limitations in our work.

Original: First, we do not considerthe important case of nonsmoothness in this work as we require that the gradients of the stochasticoptimization function are continuous.
Revised: First, we do not consider the important case of nonsmoothness in this work as we require that the gradients of the stochastic optimization function are continuous.

Original: However, we note that if the results of Bianchi et al.
Revised: However, we note that if the results of Bianchi et al.

Original: [2022] arebroadly applicable, then SGD never observes a point of nonsmoothness and our results would thenbe applicable.
Revised: [2022] are broadly applicable, then SGD never observes a point of nonsmoothness and our results would then be applicable.

Original: Second, we do not have a simple interpretation of Assumption 5—though it seemsto have a close relative in another analysis (see Wang and Wu [2020])—, nor have we been able toconstruct a relevant counterexample that can illuminate the limitations of this assumption.
Revised: Second, we do not have a simple interpretation of Assumption 5—though it seems to have a close relative in another analysis (see Wang and Wu [2020])—, nor have we been able to construct a relevant counterexample that can illuminate the limitations of this assumption.

Original:  Stochastic Optimization
Revised:  Stochastic Optimization

Original: We consider solving the optimization problem
Revised: We consider solving the optimization problem

Original: [Equation]
Revised: [Equation]

Original: where F maps R p into R ; f maps R p and the co-domain of the random variable X into R ; and E isthe expectation operator.
Revised: where F maps R p into R ; f maps R p and the co-domain of the random variable X into R ; and E is the expectation operator.

Original: As we require gradients, we take F and f to differentiable in θ , and denoteits derivatives with respect to θ by ˙
Revised: As we require gradients, we take F and f to differentiable in θ , and denote its derivatives with respect to θ by ˙

Original: F ( θ ) and ˙ f
Revised: F ( θ ) and ˙ f

Original: ( θ, X ) .
Revised: ( θ, X ) .

Original: With this notation, we make the followinggeneral assumptions about the deterministic portion of the objective function.
Revised: With this notation, we make the following general assumptions about the deterministic portion of the objective function.

Original: Assumption 1.
Revised: Assumption 1.

Original: There exists F l.b. ∈ R such that ∀ θ ∈ R p , F l.b. ≤ F ( θ ) .
Revised: There exists F l.b. ∈ R such that ∀ θ ∈ R p , F l.b. ≤ F ( θ ) .

Original: Assumption 2.
Revised: Assumption 2.

Original: There exists α ∈ (0 , 1] such that ˙ F ( θ ) is locally α -Hölder continuous.
Revised: There exists α ∈ (0 , 1] such that ˙ F ( θ ) is locally α -Hölder continuous.

Original: Remark 1 .
Revised: Remark 1 .

Revised: For ˙ F to be locally α -Hölder continuous for some α ∈ (0 , 1] , for every compact set

Original: {+K ⊂ R p there exists a constant L ≥ 
+}
Revised: {+K ⊂ R p there exists a constant L ≥ 
+}

Original:  As an example, an empirical risk minimization problem for a deep neural network withtwice continuously differentiable activation functions with a twice continuously differentiable lossfunction will readily satisfy the above conditions.
Revised: such that for every θ, ψ ∈ K , [Equation] Remark 2 . As an example, an empirical risk minimization problem for a deep neural network with twice continuously differentiable activation functions with a twice continuously differentiable loss function will readily satisfy the above conditions.

Original: Assumptions 1 and 2 would even be considered mild in the context of non-convex deterministicoptimization, in which it is also common to assume that the objective function  well-behaved levelsets [e.g., Nocedal and Wright, 2006, Theorems 3.2, 3.8, 4.5, 4.6].
Revised: Assumptions 1 and 2 would even be considered mild in the context of non-convex deterministic optimization, in which it is also common to assume that the objective function has well-behaved level sets [e.g., Nocedal and Wright, 2006, Theorems 3.2, 3.8, 4.5, 4.6].

Original: Importantly, Assumption 2 relaxesthe common restrictive assumption of globally Hölder continuous gradient functions that is commonin other analyses (see Table 1).
Revised: Importantly, Assumption 2 relaxes the common restrictive assumption of globally Hölder continuous gradient functions that is common in other analyses (see Table 1).

Original: Our ﬁnal step is to make some assumptions about the stochastic portion of the objective function.
Revised: Our ﬁnal step is to make some assumptions about the stochastic portion of the objective function.

Original: Theﬁrst assumption requires the stochastic gradients to be unbiased, which can readily be relaxed [Bottouet al., 2018].
Revised: The ﬁrst assumption requires the stochastic gradients to be unbiased, which can readily be relaxed [Bottou et al., 2018].

Original: The second assumption allows for a generic noise model for an α -Hölder continuousgradient function, and even allows for the second moment to not exist when α < [-
-] [c.f. Wang et al. ,2021, which requires a decomposition of the noise term that we do not require].
Revised: The second assumption allows for a generic noise model for an α -Hölder continuous gradient function, and even allows for the second moment to not exist when α < [-
-]1 [c.f. Wang et al., 2021, which requires a decomposition of the noise term that we do not require].

Original: Assumption 3.
Revised: Assumption 3.

Original: For all θ ∈ R p , ˙ F ( θ ) =
Revised: For all θ ∈ R p , ˙ F ( θ ) =

Original: E [ ˙ f
Revised: E [ ˙ f

Original: ( θ, X )] .
Revised: ( θ, X )] .

Original: Assumption 4.
Revised: Assumption 4.

Original: Let α ∈ (0 , 1] be as in Assumption 2.
Revised: Let α ∈ (0 , 1] be as in Assumption 2.

Original: There exists an upper semi-continuous function
Revised: There exists an upper semi-continuous function

Original: G ( θ ) such that E [ (cid:107) ˙
Revised: G ( θ ) such that E [ (cid:107) ˙

Original: f ( θ, X ) (cid:107) 

Revised: f ( θ, X ) (cid:107)

Original: + α] ≤ G ( θ ) .
Revised: + α] ≤ G ( θ ) .

Revised: Remark 3 .

Revised: For G ( θ ) to be upper semi-continuous, then for all g > 0 , { θ ∈ R p : G ( θ ) < g } are open in R p .

Original: We will show that Assumptions 1 to 4 are sufﬁcient for a global convergence result (see
Revised: We will show that Assumptions 1 to 4 are sufﬁcient for a global convergence result (see

Original: Theorem 2).
Revised: Theorem 2).

Revised: Remark 4 .

Revised: As shown in §A, our examples from §1 satisfy Assumptions 1 to 4.

Original: Remark 2 .
Revised: Remark 5 .

Original: It is entirely possible that E [ (cid:107) ˙
Revised: It is entirely possible that E [ (cid:107) ˙

Original: f
Revised: f

Original: ( θ, X ) (cid:107) 

Revised: ( θ, X ) (cid:107)

Original: + α] is (at least) upper semi-continuous, and toset G ( θ ) equal to this function.
Revised: + α] is (at least) upper semi-continuous, and to set G ( θ ) equal to this function.

Original: In the case that E [ (cid:107) ˙
Revised: In the case that E [ (cid:107) ˙

Original: f
Revised: f

Original: ( θ, X ) (cid:107) 

Revised: ( θ, X ) (cid:107)

Original: + α] is not upper semi-continuous, itis possible to specify G ( θ ) as the upper envelope of E [ (cid:107) ˙
Revised: + α] is not upper semi-continuous, it is possible to specify G ( θ ) as the upper envelope of E [ (cid:107) ˙

Original: f ( θ, X ) (cid:107) 

Revised: f ( θ, X ) (cid:107)

Original: + α] (i.e.,the its limit supremumfunction).
Revised: + α] (i.e., its limit supremum function).

Original: However, it is unlikely that E [ (cid:107) ˙
Revised: However, it is unlikely that E [ (cid:107) ˙

Original: f ( θ, X ) (cid:107) 

Revised: f ( θ, X ) (cid:107)

Original: + α] nor its upper envelope are easy to specifyexplicitly, and it is more likely to be able to ﬁnd an upper bound.
Revised: + α] nor its upper envelope are easy to specify explicitly, and it is more likely to be able to ﬁnd an upper bound.

Original: Remark 3 .
Revised: Remark 6 .

Original: We use Assumption 4 to conclude that on any compact set, the 1 + α moment of thestochastic gradient is bounded.
Revised: We use Assumption 4 to conclude that on any compact set, the 1 + α moment of the stochastic gradient is bounded.

Original: Of course, we can assume this directly (i.e., on any compact set, [-the
-] + α moment is bounded), which, at ﬁrst glance, appears to be a relaxation.
Revised: Of course, we can assume this directly (i.e., on any compact set, [-the
-]the 1 + α moment is bounded), which, at ﬁrst glance, appears to be a relaxation.

Original: However, if we assumethat on any compact set, the 1 + α moment is bounded, we can use this to construct a G ( θ ) that isupper semi-continuous.
Revised: However, if we assume that on any compact set, the 1 + α moment is bounded, we can use this to construct a G ( θ ) that is upper semi-continuous.

Original: Thus, the two assumptions are equivalent.
Revised: Thus, the two assumptions are equivalent.

Original: Remark 4 .
Revised: Remark 7 .

Original: A simple example that shows the utility of Assumption 4 is to optimize E [ θ X ] where Xis an exponential random variable with parameter 1 and θ ∈ [1 , u ] where u < exp(1) .
Revised: A simple example that shows the utility of Assumption 4 is to optimize E [ θ X ] where X is an exponential random variable with parameter 1 and θ ∈ [1 , u ] where u < exp(1) .

Original: First, it iseasy to conﬁrm that the objective function is differentiable and its derivative is globally Lipschitzcontinuous.
Revised: First, it is easy to conﬁrm that the objective function is differentiable and its derivative is globally Lipschitz continuous.

Original: Moreover, given that we are on a bounded interval, we conclude that the derivative is globally α -Hölder continuous for any α ∈ (0 , 1] ; therefore, we are free to choose the α as we see ﬁt.
Revised: Moreover, given that we are on a bounded interval, we conclude that the derivative is globally α -Hölder continuous for any α ∈ (0 , 1] ; therefore, we are free to choose the α as we see ﬁt.

Original: Now, when u < exp(1 / 2) , we have that second moment of the stochastic gradient function exists.
Revised: Now, when u < exp(1 / 2) , we have that second moment of the stochastic gradient function exists.

Original: However, when exp(1 / 2) < u < exp(1) , only smaller moments of the stochastic gradient will exist.
Revised: However, when exp(1 / 2) < u < exp(1) , only smaller moments of the stochastic gradient will exist.

Original: Speciﬁcally, only for u < exp(1 /
Revised: Speciﬁcally, only for u < exp(1 /

Original: (1 + (cid:15) )) with (cid:15) ∈ (0 , 1) will the 1 + (cid:15) moment of the stochasticgradient will exist.
Revised: (1 + (cid:15) )) with (cid:15) ∈ (0 , 1) will the 1 + (cid:15) moment of the stochastic gradient will exist.

Original: Thus, depending on the size of our interval, we may not have the existence of thesecond moment.
Revised: Thus, depending on the size of our interval, we may not have the existence of the second moment, and, consequently, we may not have the existence of the variance.

Original: In order to show that the objective function cannot diverge (i.e., to prove stability), we will need anadditional assumption.
Revised: In order to show that the objective function cannot diverge (i.e., to prove stability), we will need an additional assumption.

Original: This assumption will relate the gradient function, noise model and variationon the local Hölder constant.
Revised: This assumption will relate the gradient function, noise model and variation on the local Hölder constant.

Original: To begin, we deﬁne the variation on the local Hölder constant.
Revised: To begin, we deﬁne the variation on the local Hölder constant.

Original: Letα ∈ (0 , 1] be as in Assumption 2 and (cid:15) > 0 be arbitrary, and deﬁne 
Revised: Let α ∈ (0 , 1] be as in Assumption 2 and (cid:15) > 0 be arbitrary, and deﬁne L

Original: [Equation]
Revised: [Equation]

Original: if this quantity is nonzero
Revised: if this quantity is nonzero

Original: [Equation]
Revised: [Equation]

Original: otherwise , (
Revised: otherwise , (

Original: [Equation]
Revised: [Equation]

Original: where ∨ indicates the maximum between two quantities.
Revised: where ∨ indicates the maximum between two quantities.

Original: Note, the choice of (cid:15) is irrelevant, and theycan be distinct for the two cases in the deﬁnition of L (cid:15) , but we ﬁx them to be the same for simplicity.
Revised: Note, the choice of (cid:15) is irrelevant, and they can be distinct for the two cases in the deﬁnition of L (cid:15) , but we ﬁx them to be the same for simplicity.

Original: Importantly, the quantity, L (cid:15) , is deﬁned at every parameter θ under Assumption 2.
Revised: Importantly, the quantity, L (cid:15) , is deﬁned at every parameter θ under Assumption 2.

Original: With this quantity, we can state a nonintuitive,technical assumption that is needed to prove stability.
Revised: With this quantity, we can state a nonintuitive, technical assumption that is needed to prove stability.

Original: Assumption 5.
Revised: Assumption 5.

Original: There exists C 1 , C 2 , C 3 ≥ 0 such that, ∀ θ ∈ R p ,
Revised: There exists C 1 , C 2 , C 3 ≥ 0 such that, ∀ θ ∈ R p ,

Original: [Equation]
Revised: [Equation]

Original: Assumption 5 generalizes Assumption 4.3(c) of Bottou et al.
Revised: Assumption 5 generalizes Assumption 4.3(c) of Bottou et al.

Original: [2018], which is satisﬁed for a largeswath of statistical models.
Revised: [2018], which is satisﬁed for a large swath of statistical models.

Original: Moreover, Assumption 5 generalizes the notion of expected smoothness[see Khaled and Richtárik, 2020, for a history of the assumption], which expanded the optimizationproblems covered by the theory of Bottou et al.
Revised: Moreover, Assumption 5 generalizes the notion of expected smoothness [see Khaled and Richtárik, 2020, for a history of the assumption], which expanded the optimization problems covered by the theory of Bottou et al.

Original: Note, Assumption 5 is about the asymptoticproperties of the stochastic optimization problem as the left hand side of the inequality in Assumption 5 can be bounded inside of any compact set.
Revised: Note, Assumption 5 is about the asymptotic properties of the stochastic optimization problem as the left hand side of the inequality in Assumption 5 can be bounded inside of any compact set.

Original: Thus, Assumption 5 covers a variety of asymptoticbehaviors, such as exp( (cid:107) θ (cid:107) 22 ) , exp( (cid:107) θ (cid:107) 2 ) , (cid:107) θ (cid:107) r 2 for r ∈ R , log( (cid:107) θ (cid:107) 2 +1) , and log(log( (cid:107) θ (cid:107) 2 +1)+1) .
Revised: Thus, Assumption 5 covers a variety of asymptotic behaviors, such as exp( (cid:107) θ (cid:107) 22 ) , exp( (cid:107) θ (cid:107) 2 ) , (cid:107) θ (cid:107) r 2 for r ∈ R , log( (cid:107) θ (cid:107) 2 +1) , and log(log( (cid:107) θ (cid:107) 2 +1)+1) .

Original: Therefore, Assumption 5 holds for functions with a variety of different asymptotic behaviors.
Revised: Therefore, Assumption 5 holds for functions with a variety of different asymptotic behaviors.

Original: We will show that Assumptions 1 to 5 are sufﬁcient for a stability result (see Theorem 3).
Revised: We will show that Assumptions 1 to 5 are sufﬁcient for a stability result (see Theorem 3).

Original: Now that we have speciﬁed the nature of the stochastic optimization problem, we turn our attentionto the algorithm used to solve the problem, namely, stochastic gradient descent (SGD).
Revised: Now that we have speciﬁed the nature of the stochastic optimization problem, we turn our attention to the algorithm used to solve the problem, namely, stochastic gradient descent (SGD).

Original:  Stochastic Gradient Descent
Revised:  Stochastic Gradient Descent

Original: SGD starts with an arbitrary initial value, θ 0 ∈ R p , and generates a sequence of iterates { θ k : k ∈ N }according to the rule
Revised: SGD starts with an arbitrary initial value, θ 0 ∈ R p , and generates a sequence of iterates { θ k : k ∈ N } according to the rule

Original: [Equation]
Revised: [Equation]

Original: where { M k : k + 1 ∈ N } ⊂ R p × p ; and { X k : k ∈ N } are independent and identically distributedcopies of X .
Revised: where { M k : k + 1 ∈ N } ⊂ R p × p ; and { X k : k ∈ N } are independent and identically distributed copies of X .

Original: Importantly, { M k } cannot be arbitrary, and the following properties specify a generalization of the Robbins and Monro [1951] conditions for matrix-valued learning rates [c.f. Patel,2021].
Revised: Importantly, { M k } cannot be arbitrary, and the following properties specify a generalization of the Robbins and Monro [1951] conditions for matrix-valued learning rates [c.f. Patel, 2021].

Original: The ﬁrst condition requires a positive learning rate, and imposes symmetry to ensure the existence ofreal eigenvalues.
Revised: The ﬁrst condition requires a positive learning rate, and imposes symmetry to ensure the existence of real eigenvalues.

Original: Property 1.
Revised: Property 1.

Original: { M k : k + 1 ∈ N } are symmetric, positive deﬁnite matrices.
Revised: { M k : k + 1 ∈ N } are symmetric, positive deﬁnite matrices.

Original: The next two properties are a natural generalization of the Robbins-Monro conditions.
Revised: The next two properties are a natural generalization of the Robbins-Monro conditions.

Original: Let α ∈ (0 , 1]be as in Assumption 2.
Revised: Let α ∈ (0 , 1] be as in Assumption 2.

Original: Property 2.
Revised: Property 2.

Original: Let λ max ( · ) denote the largest eigenvalue of a symmetric, positive deﬁnite matrix.
Revised: Let λ max ( · ) denote the largest eigenvalue of a symmetric, positive deﬁnite matrix.

Original: Then,183 (cid:80) ∞
Revised: Then, (cid:80) ∞

Original: k =

Revised: k =

Original: λ max ( M k ) 

Revised: λ max ( M k )

Original: + α =:
Revised: + α =:

Original: S < ∞ .
Revised: S < ∞ .

Original: Property 3.
Revised: Property 3.

Original: Let λ min ( · ) denote the smallest eigenvalue of a symmetric, positive deﬁnite matrix.
Revised: Let λ min ( · ) denote the smallest eigenvalue of a symmetric, positive deﬁnite matrix.

Original: Then, (cid:80) ∞
Revised: Then, (cid:80) ∞

Original: k =

Revised: k =

Original: λ min ( M k ) = ∞ .
Revised: λ min ( M k ) = ∞ .

Original: We will show that Properties 1 to 3 are sufﬁcient for a global convergence result (see
Revised: We will show that Properties 1 to 3 are sufﬁcient for a global convergence result (see

Original: Theorem 2).
Revised: Theorem 2).

Original: The ﬁnal property ensures the stability of the condition number of { M k } .
Revised: The ﬁnal property ensures the stability of the condition number of { M k } .

Original: Note, this property isreadily satisﬁed for scalar learning rates satisfying the Robbins-Monro conditions.
Revised: Note, this property is readily satisﬁed for scalar learning rates satisfying the Robbins-Monro conditions.

Original: Property 4.
Revised: Property 4.

Original: Let κ ( · ) denote the ratio of the largest and smallest eigenvalues of a symmetric, positivedeﬁnite matrix.
Revised: Let κ ( · ) denote the ratio of the largest and smallest eigenvalues of a symmetric, positive deﬁnite matrix.

Original: Then, lim k →∞ λ max ( M k )
Revised: Then, lim k →∞ λ max ( M k )

Original: α κ ( M k )
Revised: α κ ( M k )

Original: = 0 .
Revised: = 0 .

Original: We will show that Properties 1 to 4 are sufﬁcient for stability (see Theorem 3).
Revised: We will show that Properties 1 to 4 are sufﬁcient for stability (see Theorem 3).

Original:  Global Convergence & Stability
Revised:  Global Convergence & Stability

Original: With the stochastic optimization problem and with stochastic gradient descent (SGD) speciﬁed, wenow turn our attention to what happens when SGD is applied to a stochastic optimization problem.
Revised: With the stochastic optimization problem and with stochastic gradient descent (SGD) speciﬁed, we now turn our attention to what happens when SGD is applied to a stochastic optimization problem.

Original: The key step in the analysis of SGD on any objective function is to establish a bound between theoptimality gap at θ k +1 with that of θ k .
Revised: The key step in the analysis of SGD on any objective function is to establish a bound between the optimality gap at θ k +1 with that of θ k .

Original: This step is achieved by using the local Hölder continuity ofthe gradient function and the fundamental theorem of calculus.
Revised: This step is achieved by using the local Hölder continuity of the gradient function and the fundamental theorem of calculus.

Original: Using Assumption 2, we ﬁrst specifythe local Hölder constant.
Revised: Using Assumption 2, we ﬁrst specify the local Hölder constant.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: For any θ, ϕ ∈ R p , deﬁne
Revised: For any θ, ϕ ∈ R p , deﬁne

Original: [Equation]
Revised: [Equation]

Original: where B ( θ, r ) is an open ball around θ of radius r > 0 .
Revised: where B ( θ, r ) is an open ball around θ of radius r > 0 , and B ( θ, r ) is its closure.

Original: Moreover, for any R ≥ 0 , let L R be thesupremum of L ( θ, ϕ ) for any distinct θ, ϕ ∈ B (0 , R ) .
Revised: Moreover, for any R ≥ 0 , let L R be the supremum of L ( θ, ϕ ) for any distinct θ, ϕ ∈ B (0 , R ) .

Original: Remark 5 .
Revised: Remark 8 .

Original: Note, when the gradient is locally Hölder continuous, L R is ﬁnite for any R ≥ 0 .
Revised: Note, when the gradient is locally Hölder continuous, L R is ﬁnite for any R ≥ 0 .

Original: With this deﬁnition, we can now relate the optimality gap of θ k +1 with that of θ k by using thefollowing result and proved in Appendix B.
Revised: With this deﬁnition, we can now relate the optimality gap of θ k +1 with that of θ k by using the following result and proved in Appendix B.

Original: Lemma 1.
Revised: Lemma 1.

Original: Suppose Assumptions 1 and 2 hold.
Revised: Suppose Assumptions 1 and 2 hold.

Original: Then, for any θ, ϕ ∈ R p ,
Revised: Then, for any θ, ϕ ∈ R p ,

Original: [Equation]
Revised: [Equation]

Original: Now, if we simply set ϕ = θ k +

Revised: Now, if we simply set ϕ = θ k +

Original: and θ = θ k in Lemma 1 and try to take expectations to manage the randomness of the stochastic gradient, we will run into the problem that L ( θ k , θ k +1 ) and(cid:107) θ k +1 − θ k (cid:107) 2 are  dependent, {+
+} and we cannot compute its expectation.
Revised: and θ = θ k in Lemma 1 and try to take expectations to manage the randomness of the stochastic gradient, we will run into the problem that L ( θ k , θ k +1 ) and (cid:107) θ k +1 − θ k (cid:107) 2 are potentially dependent, {+
+} and we cannot compute its expectation.

Original: In previous work, this technicalchallenge is waived away by using a global Hölder constant to upper bound L ( θ k , θ k +1 ) , which isunrealistic even for simple problems (see Section 1).
Revised: In previous work, this technical challenge is waived away by using a global Hölder constant to upper bound L ( θ k , θ k +1 ) , which is unrealistic even for simple problems (see Section 1).

Original: To address this technical challenge, we innovate two new strategies for handling the dependencebetween L ( θ k , θ k +1 ) and (cid:107) θ k +1 − θ k (cid:107) 2 .
Revised: To address this technical challenge, we innovate two new strategies for handling the dependence between L ( θ k , θ k +1 ) and (cid:107) θ k +1 − θ k (cid:107) 2 .

Original: In both strategies, we follow the same general approach:
Revised: In both strategies, we follow the same general approach:

Original: We begin by restricting our analysis to speciﬁc events which will allow us to decouple L ( θ k , θ k +1 ) and (cid:107) θ k +1 − θ k (cid:107) 2 . 
Revised: We begin by restricting our analysis to speciﬁc events, which will allow us to decouple L ( θ k , θ k +1 ) and (cid:107) θ k +1 − θ k (cid:107) 2 . 2.

Original: With these two quantities decoupled, we will develop a recurrence  the 
Revised: With these two quantities decoupled, we will develop a recurrence relationship between the optimality gap at θ k +1 and that of θ k . 3.

Original: We apply this recurrence relationship with reﬁnements of standard arguments or new onesto derive the desired property about the objective function.
Revised: We apply this recurrence relationship with reﬁnements of standard arguments or new ones to derive the desired property about the objective function.

Original: Finally, we state the generality of the speciﬁc events on which we have studied SGD’siterates.
Revised: Finally, we state the generality of the speciﬁc events on which we have studied SGD’s iterates.

Original: Thus, it follows, we will deﬁne two distinct series of events for the two strategies.
Revised: Thus, it follows, we will deﬁne two distinct series of events for the two strategies.

Original: The ﬁrst strategy,which we refer to as the pseudo-global strategy, will provide the global convergence analysis.
Revised: The ﬁrst strategy, which we refer to as the pseudo-global strategy, will provide the global convergence analysis.

Original: Thesecond strategy, which we refer to as the local strategy, will provide the stability result.
Revised: The second strategy, which we refer to as the local strategy, will provide the stability result.

Original: 4.1 Pseudo-Global Strategy and Global Convergence Analysis
Revised: 4.1 Pseudo-Global Strategy and Global Convergence Analysis

Original: For the ﬁrst strategy, which supplies the global convergence result, we study SGD on the events
Revised: For the ﬁrst strategy, which supplies the global convergence result, we study SGD on the events

Original: [Equation]
Revised: [Equation]

Original: for every R ≥ 0 .
Revised: for every R ≥ 0 .

Original: We now try to control the optimality gap at iteration k + 1 with that of iteration k ,which will result in two cases.
Revised: We now try to control the optimality gap at iteration k + 1 with that of iteration k , which will result in two cases.

Original: (Case 1)
Revised: (Case 1)

Original: B k +

Revised: B k +

Original: ( R ) holds.
Revised: ( R ) holds.

Original: We can bound L ( θ k , θ k + 1 ) by L R , and G ( θ ) is also bounded inthe ball of radius R about the origin (which follows from G being upper semi-continuousin Assumption 4).
Revised: We can bound L ( θ k , θ k +1 ) by L R , and G ( θ ) is also bounded in the ball of radius R about the origin (which follows from G being upper semi-continuous in Assumption 4).

Original: As a result, we could then proceed with the analysis in a manner that issimilar to having a global Hölder constant. 
Revised: As a result, we could then proceed with the analysis in a manner that is similar to having a global Hölder constant. 2.

Original: (Case 2) (cid:107) θ k +1 (cid:107)  > R and B k ( R ) holds.
Revised: (Case 2) (cid:107) θ k +1 (cid:107) 2 > R and B k ( R ) holds.

Original: In this case, controlling L ( θ k , θ k +1 ) is verychallenging and, to our knowledge, was not solved before our work.
Revised: In this case, controlling L ( θ k , θ k +1 ) is very challenging and, to our knowledge was not solved before our work.

Original: Our approach for controlling the optimality gap in both cases is supplied in the next lemma, whoseproof is in Appendix C.
Revised: Our approach for controlling the optimality gap in both cases is supplied in the next lemma, whose proof is in Appendix C.

Original: Lemma 2.
Revised: Lemma 2.

Original: Let { M k } satisfy Property 1 .
Revised: Let { M k } satisfy Property 1 .

Original: Suppose Assumptions 1 to 4 hold.
Revised: Suppose Assumptions 1 to 4 hold.

Original: Let { θ k } satisfy (4) .
Revised: Let { θ k } satisfy (5) .

Original: Then, ∀ R ≥ 0 ,
Revised: Then, ∀ R ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: k [Equation] where G R = sup θ ∈ B (0 ,R ) G ( θ ) < ∞ with G ( θ ) ; and ∂F R = sup θ ∈ B (0 ,R ) (cid:107) ˙
Revised: where G R = sup θ ∈ B (0 ,R ) G ( θ ) < ∞ with G ( θ ) ; and ∂F R = sup θ ∈ B (0 ,R ) (cid:107) ˙

Original: F ( θ ) (cid:107) 2 (1 + α ) < ∞ .
Revised: F ( θ ) (cid:107) 2 (1 + α ) < ∞ .

Original: With this recursion and standard martingale results [Robbins and Siegmund, 1971, Neveu and Speed,1975, Exercise II.4], the limit of [ F ( θ k ) − F l.b. ] 1{+
+} [ B k( R )] exists with probability oneand is ﬁnite forevery R ≥ 0 .
Revised: With this recursion and standard martingale results [Robbins and Siegmund, 1971, Neveu and Speed, 1975, Exercise II.4], the limit of [ F ( θ k ) − F l.b. ] {+
+} [ B k R )] exists with probability one ﬁnite for every R ≥ 0 .

Original: As a result, the limit of F ( θ k ) − F l.b. exists and is ﬁnite on the event { sup k (cid:107) θ k (cid:107) 2 < ∞}(see Corollary 1).
Revised: As a result, the limit of F ( θ k ) − F l.b. exists and is ﬁnite on the event { sup k (cid:107) θ k (cid:107) 2 < ∞} (see Corollary 1).

Original: We can also use Lemma 2 to make a statement about the gradient.
Revised: We can also use Lemma 2 to make a statement about the gradient.

Original: Speciﬁcally, we can show that thelimit inﬁmum of E [ (cid:107) ˙
Revised: Speciﬁcally, we can show that the limit inﬁmum of E [ (cid:107) ˙

Original: F ( θ k ) (cid:107) 22 

Revised: F ( θ k ) (cid:107) 22

Original: [ B k ( R )]] must be zero, which is now a standard argument that mimics Zoutendijk’s theorem [Nocedal and Wright, 2006, Theorem 3.2].
Revised: [ B k ( R )]] must be zero, which is now a standard argument that mimics Zoutendijk’s theorem [Nocedal and Wright, 2006, Theorem 3.2].

Original: By Markov’s inequality, this resultimplies that (cid:107) ˙ F ( θ k ) (cid:107) 2 

Revised: By Markov’s inequality, this result implies that (cid:107) ˙ F ( θ k ) (cid:107) 2

Original: [ B k ( R )] gets arbitrarily close to 0 inﬁnitely often (see Lemma 9).
Revised: [ B k ( R )] gets arbitrarily close to 0 inﬁnitely often (see Lemma 9).

Original: To showconvergence to zero, however, is not standard.
Revised: To show convergence to zero, however, is not standard.

Original: Several strategies have been developed, namely thoseof Li and Orabona [2019], Lei et al.
Revised: Several strategies have been developed, namely those of Li and Orabona [2019], Lei et al.

Original: [2019], Mertikopoulos et al.
Revised: [2019], Mertikopoulos et al.

Original: [2020], Patel [2021].
Revised: [2020], Patel [2021],

Revised: Patel and Zhang [2021].

Original: Unfortunately,the approaches of Li and Orabona [2019], Lei et al.
Revised: Unfortunately, the approaches of Li and Orabona [2019], Lei et al.

Original: [2019] rely intimately on the existence of a global Hölder constant, while that of Mertikopoulos et al.
Revised: [2019] rely intimately on the existence of a global Hölder constant, while that of Mertikopoulos et al.

Original: [2020] requires even more restrictive assumptions.
Revised: [2020] requires even more restrictive assumptions.

Original: Fortunately, the approach of Patel  [2021] can be improved and generalized to the current context (see Lemma 10).
Revised: Fortunately, the approach of Patel [2021], Patel and Zhang [2021] can be improved and generalized to the current context (see Lemma 10).

Original: Thus, we show that lim k →∞ (cid:107) ˙
Revised: Thus, we show that lim k →∞ (cid:107) ˙

Original: F ( θ k ) (cid:107) 2 = 0 on { sup k (cid:107) θ k (cid:107) 2 < ∞} (see Corollary 2).
Revised: F ( θ k ) (cid:107) 2 = 0 on { sup k (cid:107) θ k (cid:107) 2 < ∞} (see Corollary 2).

Original: Our ﬁnal step is to clarify the role of { sup k (cid:107) θ k (cid:107) 2 < ∞} in the asymptotic behavior of SGD’s iterates.
Revised: Our ﬁnal step is to clarify the role of { sup k (cid:107) θ k (cid:107) 2 < ∞} in the asymptotics of SGD’s iterates.

Original: At ﬁrst glance, this event seems to imply that the iterates converge to a point.
Revised: At ﬁrst glance, this event seems to imply that the iterates converge to a point.

Original: However, owing to thegeneral nature of the noise, it is also possible, say, that the iterates approach a limit cycle or oscillatebetween points with the same norm.
Revised: However, owing to the general nature of the noise, it is also possible, say, that the iterates approach a limit cycle or oscillate between points with the same norm.

Original: Even beyond this event, the generality of the noise model mayallow for substantial excursions between F l.b. and inﬁnity (c.f., a simple random walk, which has alimit supremum of inﬁnity and a limit inﬁnimum of negative inﬁnity).
Revised: Even beyond this event, the generality of the noise model may allow for substantial excursions between F l.b. and inﬁnity (c.f., a simple random walk, which has a limit supremum of inﬁnity and a limit inﬁnimum of negative inﬁnity).

Original: Thankfully, we can prove thateither the iterates converge to a point or they must diverge—a result that we refer to as the Capture
Revised: Thankfully, we can prove that either the iterates converge to a point or they must diverge—a result that we refer to as the Capture

Original: Theorem (see Appendix C).
Revised: Theorem (see Appendix C).

Original: Theorem 1 (Capture Theorem) .
Revised: Theorem 1 (Capture Theorem) .

Original: Let { θ k } be deﬁned as in (4) , and let { M k } satisfy Properties 1and 2 .
Revised: Let { θ k } be deﬁned as in (5) , and let { M k } satisfy Propertiesand 2 .

Original: If Assumption 4 holds, then either { lim k →∞ θ k exists } or { lim inf k →∞ (cid:107) θ k (cid:107) 2 = ∞} mustoccur.
Revised: If Assumption 4 holds, then either { lim k →∞ θ k exists } or { lim inf k →∞ (cid:107) θ k (cid:107) 2 = ∞} must occur.

Original: By putting together the above arguments and results, we can conclude that either SGD’s iteratesdiverge or SGD’s iterates converge to a stationary point.
Revised: By putting together the above arguments and results, we can conclude that either SGD’s iterates diverge or SGD’s iterates converge to a stationary point.

Original: This is formally stated in the followingtheorem.

Original: See Section 1 for a discussion of the practical value of this result.

Original: Theorem 2 (Global Convergence) .
Revised: Theorem 2 (Global Convergence) .

Original: Let θ
Revised: Let θ

Original: be arbitrary, and let { θ k : k ∈ N } be deﬁned accordingto (4) with { M k : k + 1 } satisfying Properties 1 to 3 .
Revised: be arbitrary, and let { θ k : k ∈ N } be deﬁned according to (5) with { M k : k + 1 } satisfying Properties 1 to 3 .

Original: Suppose Assumptions 1 to 4 hold.
Revised: Suppose Assumptions 1 to 4 hold.

Original: Let
Revised: Let

Original: A 1 = { lim inf k →∞ (cid:107) θ k (cid:107) 2 = ∞} and A 2 = { lim k →∞ θ k exists } .
Revised: A 1 = { lim inf k →∞ (cid:107) θ k (cid:107) 2 = ∞} and A 2 = { lim k →∞ θ k exists } .

Original: Then, the following statementshold.1.
Revised: Then, the following statements hold.

Original: P [ A 1 ] + P [ A 2 ] = 1 .2.
Revised: P [ A 1 ] + P [ A 2 ] = 1 . 2.

Original: On A 2 , there exists a ﬁnite random variable, F lim , such that lim k →∞ F ( θ k ) =
Revised: On A 2 , there exists a ﬁnite random variable, F lim , such that lim k →∞ F ( θ k ) =

Original: F lim andlim k →∞ ˙ F ( θ k ) = 0{+
+} with probability one.
Revised: F lim and lim k →∞ ˙ F ( θ k ) = {+
+} with probability one.

Original: Proof.
Revised: Proof.

Original: By Theorem 1, we have that P [ A 1 ] + P [ A 2 ] = 1 .
Revised: By Theorem 1, we have that P [ A 1 ] + P [ A 2 ] = 1 .

Original: Then, on A 2 , Corollaries 1 and 2 imply that
Revised: Then, on A 2 , Corollaries 1 and 2 imply that

Original: F ( θ k ) → F lim , which is ﬁnite, and ˙ F ( θ k ) → 0 .
Revised: F ( θ k ) → F lim , which is ﬁnite, and ˙ F ( θ k ) → 0 .

Revised: We pause to stress to a fact about Theorem 2: it is nonobvious.

Revised: To be speciﬁc, under such general nonconvexity and noise, we should anticipate any number of asymptotic behaviors for the iterates: convergence to a stationary point, convergence to a nonstationary point, being trapped in a cycle, convergence to a limit cycle, and divergence to inﬁnity.

Revised: However and very surprisingly, we are able to show that only two possible outcomes can occur: convergence to a stationary point or divergence.

Revised: Indeed, in previous results [e.g., Mertikopoulos et al., 2020, Patel, 2021], only less speciﬁc determinations could be made under much more limited settings.

Original: 4.2 Local Strategy and Stability Analysis
Revised: 4.2 Local Strategy and Stability Analysis

Original: While Theorem 2 provides a complete global convergence result, it allows for the possibility ofdiverging iterates.
Revised: While Theorem 2 provides a complete global convergence result, it allows for the possibility of diverging iterates.

Original: The possibility of divergent iterates raises the spectre of whether the objectivefunction can also diverge along this sequence.
Revised: The possibility of divergent iterates raises the spectre of whether the objective function can also diverge along this sequence.

Original: That is, there is a possibility that SGD may be unstable,which would be highly unexpected and undesirable, especially when the objective function is coercive(e.g. , has an (cid:96) 1 penalty on the parameters).
Revised: That is, there is a possibility that SGD may be unstable, which would be highly unexpected and undesirable, especially when the objective function is coercive (e.g., has an (cid:96) 1 penalty on the parameters).

Original: To formalize this concept, we deﬁne a relevant notion ofstability.
Revised: To formalize this concept, we deﬁne a relevant notion of stability.

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: Stochastic Gradient Descent is stable if
Revised: Stochastic Gradient Descent is stable if

Original: [Equation]
Revised: [Equation]

Original: where { θ k } satisfy (4).
Revised: where { θ k } satisfy (5).

Original: We now state the events that we will use to decouple the relationship between L ( θ k , θ k +1 ) and(cid:107) θ k +1 − θ k (cid:107) 2 .
Revised: We now state the stopping times that we will use to decouple the relationship between L ( θ k , θ k +1 ) and (cid:107) θ k +1 − θ k (cid:107) 2 .

Original: Unlike in our pseudo-global strategy, we will make use of two sequences of eventsthat are closely related.

Original: To deﬁne these sequences, we will ﬁrst need to deﬁne stopping times.

Original: Forevery j + 1 ∈ N , deﬁne
Revised: For every j + 1 ∈ N , deﬁne

Original: [Equation]
Revised: [Equation]

Original: Analogously, for every j + 1 ∈ N , deﬁne

Original: [Equation] Now, we will use (10) to establish the stability of the objective function, and we will use (11) to showthat the gradient function must tend to zero.
Revised: Now, we will use (11) to establish the stability of the objective function.

Original: Just as we did with B k
Revised: Just as we did with B k

Original: ( R ) , we will derive a recursionon the optimality gap over the events {{ τ j > k } : k + 1 ∈ N } .
Revised: ( R ) , we will derive a recursion on the optimality gap over the events {{ τ j > k } : k + 1 ∈ N } .

Original: Of course, just as before, the mainchallenge in deriving a recursive formula is to address { τ j = k } .
Revised: Of course, just as before, the main challenge in deriving a recursive formula is to address { τ j = k } .

Original: Our solution is supplied in thefollowing lemma, whose proof is in Appendix D.
Revised: Our solution is supplied in the following lemma, whose proof is in Appendix D.

Original: Lemma 3.
Revised: Lemma 3.

Original: Let { M k } satisfy Property 1 .
Revised: Let { M k } satisfy Property 1 .

Original: Suppose Assumptions 1 to 4 hold.
Revised: Suppose Assumptions 1 to 4 hold.

Original: Let { θ k } satisfy (4) .
Revised: Let { θ k } satisfy (5) .

Original: Then, for any j + 1 ∈ N and k > j ,
Revised: Then, for any j + 1 ∈ N and k > j ,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: From Lemma 3, there is a clear motivation for Assumption 5.
Revised: From Lemma 3, there is a clear motivation for Assumption 5.

Original: Indeed, if we apply Assumption 5,
Revised: Indeed, if we apply Assumption 5,

Original: Lemma 3 produces the following simple recursive relationship.
Revised: Lemma 3 produces the following simple recursive relationship.

Original: Lemma 4.
Revised: Lemma 4.

Original: If Assumptions 1 to 5 , and Properties 1 and 4 hold, and { θ k } satisfy (4) , then there existsa K ∈ N such that for any j + 1 ∈ N and any k ≥ min { K, j + 1 } ,
Revised: If Assumptions 1 to 5 , and Properties 1 and 4 hold, and { θ k } satisfy (5) , then there exists a K ∈ N such that for any j + 1 ∈ N and any k ≥ min { K, j + 1 } ,

Original: [Equation]
Revised: [Equation]

Original: Just as in the pseudo-global strategy, Lemma 4 can be combined with standard martingale results[Robbins and Siegmund, 1971, Neveu and Speed, 1975, Exercise II.4] to conclude that the limit of
Revised: Just as in the pseudo-global strategy, Lemma 4 can be combined with standard martingale results [Robbins and Siegmund, 1971, Neveu and Speed, 1975, Exercise II.4] to conclude that the limit of

Original: F ( θ k ) exists and is ﬁnite on the event ∪ ∞
Revised: F ( θ k ) exists and is ﬁnite on the event ∪ ∞

Original: j =0 { τ j = ∞} (see Corollary 3).
Revised: j =0 { τ j = ∞} (see Corollary 3).

Original: Also as in the pseudo-globalstrategy, by improving on the arguments in Patel [2021],  we show that lim  k ˙ F ( θ k ) = 0 on the event∪ ∞ j =0 { ν j = ∞} (see Corollary 4).
Revised: Also as in the pseudo-global strategy, by improving on the arguments in Patel [2021], Patel and Zhang [2021], we show that lim inf k ˙ F ( θ k ) = 0 on the event ∪ ∞ j =0 { τ j = ∞} (see Lemma 14).

Original: Finally, we show that ∪ ∞ j =0 { τ j = ∞} and ∪ ∞ j =0 { ν j = ∞} are probability one events (see Theorem 5).
Revised: Finally, we show that ∪ ∞ j =0 { τ j = ∞} is a probability one event (see Theorem 5).

Original: In other words, we show that eventually L (cid:15) ( θ k ) will always dominate L ( θ k , θ k +1 ) .

Original: This statementshould not come as a surprise on the event { lim k θ k exists } , but is slightly surprising that it mustalso hold on { lim k (cid:107) θ k (cid:107) 2 = ∞} .
Revised: This statement should not come as a surprise on the event { lim k θ k exists } , but is slightly surprising that it must also hold on { lim k (cid:107) θ k (cid:107) 2 = ∞} .

Original: By combining these results, we can conclude as follows.
Revised: By combining these results, we can conclude as follows.

Original: Theorem 3 (Stability) .
Revised: Theorem 3 (Stability) .

Original: Let θ
Revised: Let θ

Original: be arbitrary, and let { θ k : k ∈ N } be deﬁned according to (4) with{ M k : k + 1 } satisfying Properties 1 to 3 .
Revised: be arbitrary, and let { θ k : k ∈ N } be deﬁned according to (5) with { M k : k + 1 } satisfying Properties 1 to 4 .

Original: Suppose Assumptions 1 to 5 hold.
Revised: Suppose Assumptions 1 to 5 hold.

Original: Then,
Revised: Then,

Original: There exists a ﬁnite random variable, F lim , such that lim k →∞ F ( θ k ) =
Revised: There exists a ﬁnite random variable, F lim , such that lim k →∞ F ( θ k ) =

Original: F lim with probability one; 
Revised: F lim with probability one; 2.

Original: lim  k →∞ ˙ F ( θ k ) = 

Revised: lim inf k →∞ ˙ F ( θ k ) =

Original: with probability one.
Revised: with probability one.

Original: Proof.
Revised: Proof.

Original: Using Corollary 3, we conclude that ∃ F lim that is ﬁnite such that lim k F ( θ k ) =
Revised: Using Corollary 3, we conclude that ∃ F lim that is ﬁnite such that lim k F ( θ k ) =

Original: F lim on∪ ∞
Revised: F lim on ∪ ∞ j =0 { τ j = ∞} . Using Lemma 14, we conclude that lim inf k ˙ F ( θ k ) = 0 on ∪ ∞

Original: j =0 { τ j = ∞} .
Revised: j =0 { τ j = ∞} .

Original: Using Corollary 4, we  conclude that lim k ˙ F ( θ k )
Revised: Finally, we apply Theorem 5 to conclude that P [ ∪ ∞

Original: [-= 
 on ∪ ∞-] j =0 { ν j = ∞}  .
Revised: [-= 
 on ∪ ∞-] j =0 { τ j = ∞} ] = 1 .

Original: Finally,we apply Theorem 5 to conclude that P [ ∪ ∞ j =0 { ν j = ∞} ]
Revised: We would like to demonstrate a simple example of how we would use Theorem 3. Consider applying

Original: = P [ ∪ ∞ j =0 { τ j = ∞} ] = 1 .
Revised: SGD to linear regression as speciﬁed in Appendix A.1. For this example, it is straightforward to verify that the assumptions of Theorem 3 are satisﬁed. Therefore, if we are to apply SGD to linear regression, we know that with probability one, F ( θ k ) → F lim which is ﬁnite.

Original: Conclusion
Revised: Since F ( θ ) → ∞ as θ → ∞ , we know that { θ k } cannot diverge. Hence, { θ k } must remain ﬁnite with probability one. By Theorem 2, { θ k } must converge to a stationary point. Since this stationary point is unique in our speciﬁc example of linear regression, we know that SGD must converge to the global minimizer of the linear regression problem.

Revised: Note, we can follow this outline to draw similar conclusions in more complex situations.

Revised:  Conclusion

Original: In this work, we studied the global convergence analysis of Stochastic Gradient Descent withdiminishing step size.
Revised: In this work, we studied the global convergence analysis of Stochastic Gradient Descent with diminishing step size.

Original: We began our discussion by producing three simple problems for whichthe common assumptions (i.e., global Hölder continuity, bounded variance)  in the SGD literatureare violated, and, to our knowledge, there does not exist theory that covers the two simple neuralnetwork problems.
Revised: We began our discussion by producing three simple problems for which the common assumptions (i.e., global Hölder continuity, bounded variance) and their generalizations in the SGD literature are violated. Indeed, to our knowledge, there does not exist theory that covers these problems.

Original: That is, prior to our work, it was unknown what SGD with arbitrary initializationand diminishing step sizes will do onthese two simple neural network problems, which raised thequestion of what SGD is doing on more complicated learning problems.
Revised: For example, prior to our work, it was unknown what SGD with arbitrary initialization and diminishing step sizes will do on simple neural network problems, which raised the question of what SGD is doing on more complicated learning problems.

Original: Motivated by our example problems, we considered a more general set of assumptions (see Assumptions 2 and 4).
Revised: Motivated by our example problems, we considered a more general set of assumptions (see Assumptions 2 and 4).

Original: Given the generality of our assumptions, we needed to develop a new analysistechnique that is of interest beyond this work, and we proved that SGD’s iterates either convergeto a stationary point or diverge.
Revised: Given the generality of our assumptions, we developed a new analysis technique that is of interest beyond this work, and we proved that SGD’s iterates either converge to a stationary point or diverge.

Original: Thus, we now know how SGD with arbitrary initialization anddiminishing step sizes will do on a much larger class of learning problems.
Revised: Thus, we now know how SGD with arbitrary initialization and diminishing step sizes will behave on a much larger class of learning problems.

Original: We note that we do not provide rate of convergence results mainly because it is impossible for thebroad class of functions admitted by our assumptions [Wolpert and Macready, 1997].
Revised: We note that we do not provide rate of convergence results mainly because it is impossible for the broad class of functions admitted by our assumptions [Wolpert and Macready, 1997].

Original: We stress thatglobal rates of convergence (e.g., complexity statements) results that exist do not apply to the twosimple neural network problems that we supplied at the beginning of this work without making morerestrictive assumptions.
Revised: We stress that global rates of convergence (e.g., complexity statements) results that exist do not apply to the two simple neural network problems that we supplied at the beginning of this work.

Original: We also studied what happens when SGD’s iteratescan diverge.
Revised: We also studied what happens when SGD’s iterates diverge.

Original: To this end, we required an additionalassumption under which we developed another novel analysis technique and showed that, regardlessof SGD’s iterates’ behavior, the objective function will converge to a ﬁnite random variableand thegradient function converges to zero with probability one.
Revised: To this end, we required an additional assumption under which we developed another novel analysis technique and showed that, regardless of SGD’s iterates’ behavior, the objective function will converge to a ﬁnite random variable with probability one.

Original: Unfortunately, we make an assumption thatwe were not able to interpret, but we will leave this to future work.
Revised: Unfortunately, we make an assumption that we were not able to interpret, but we will leave this to future work.

Original: Checklist

Original: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?

Original: [Yes] Each claim in the abstract corresponds to a statement inthe introduction, which then references the corresponding result in the paper.

Original: (b) Did you describe the limitations of your work?

Original: [Yes] See the end of Section 1(c)

Original: Did you discuss any potential negative societal impacts of your work?

Original: [N/A]

Original: SGD isalready a widely existing and deployed method.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] Each resultreferences the precise assumptions that are used and these are clearly stated in Section 2.(b)

Original: Did you include complete proofs of all theoretical results?

Original: [Yes] Outlines are pro-

Original: vided in the main document, while complete proofs of each result are provided in theappendix/supplement.3.

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [N/A]

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [N/A]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [N/A]

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]

Original: (b) Did you mention the license of the assets?

Original: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?

Original: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?

Original: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?

Original: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?

Original: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?

Original: [N/A]

Original: A Details for Counter Examples 
Revised: A Details for Counter Examples A.1 Simple Linear Regression

Original: A.1 Simple Linear Regression

Original: Here, we prove Proposition 1 for a rather simple problem.
Revised: We begin our exploration of assumptions with a rather simple problem.

Original: Let Z ∈ R be a random variable such that448 E (cid:2)
Revised: Let Z ∈ R be a random variable such that E (cid:2)

Original: Z 2 (cid:3) = 1 and E (cid:2) Z 4 (cid:3) = 2 .
Revised: Z 2 (cid:3) = 1 and E (cid:2) Z 4 (cid:3) = 2 .

Original: Moreover, let (cid:15) be an independent random variable with mean zeroand variance 1 .
Revised: Moreover, let (cid:15) be an independent random variable with mean zero and variance 1 .

Original: Finally, let θ ∗ ∈ R and deﬁne Y =
Revised: Finally, let θ ∗ ∈ R and deﬁne Y =

Original: Zθ ∗ + (cid:15) .
Revised: Zθ ∗ + (cid:15) .

Original: Consider the estimation problem ofminimizing F ( θ ) where
Revised: Consider the estimation problem of minimizing F ( θ ) where

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Letting X =
Revised: Letting X =

Original: ( Y, Z ) , let f ( θ, X )
Revised: ( Y, Z ) , let f ( θ, X )

Original: = 0 .
Revised: = 0 .

Original: 5( Zθ − Y ) 2 .
Revised: 5( Zθ − Y ) 2 .

Original: Now, the variance of ˙ f
Revised: Now, the variance of ˙ f

Original: ( θ, X ) is
Revised: ( θ, X ) is

Original: [Equation]
Revised: [Equation]

Original: Clearly, the variance scales with the error in the parameter, which violates the common boundednoise model assumption.
Revised: Clearly, the variance scales with the error in the parameter, which violates the common bounded noise model assumption.

Original: In particular, as | θ | → ∞ , the variance diverges.
Revised: In particular, as | θ | → ∞ , the variance diverges.

Original: On the other hand, the simple linear regression problem does satisfy our assumptions.
Revised: On the other hand, the simple linear regression problem does satisfy our assumptions.

Original: In particular,1.
Revised: In particular, 1.

Original: Assumptions 1 and 3 are easily veriﬁed.2.
Revised: Assumptions 1 and 3 are easily veriﬁed.

Original: Given that ˙ F is globally Lipschitz continuous, it is locally Lipschitz continuous.
Revised: Given that ˙ F is globally Lipschitz continuous, it is locally Lipschitz continuous.

Original: Therefore,
Revised: Therefore,

Original: Assumption 2 is satisﬁed.3.
Revised: Assumption 2 is satisﬁed.

Original: From the variance calculation of ˙ f
Revised: From the variance calculation of ˙ f

Original: ( θ, X ) , we conclude
Revised: ( θ, X ) , we conclude

Original: [Equation]
Revised: [Equation]

Original: (17) which is a continuous function.
Revised: 17) which is a continuous function.

Original: Hence, Assumption 4 is satisﬁed.
Revised: Hence, Assumption 4 is satisﬁed.

Original: A.2 Feed Forward Network for Binary Classiﬁcation
Revised: A.2 Feed Forward Network for Binary Classiﬁcation

Original: We now prove Proposition 2.
Revised: We now prove Proposition 1.

Original: Consider the binary classiﬁcation problem with label Y and feature
Revised: Consider the binary classiﬁcation problem with label Y and feature

Original: Z where ( Y, Z ) =
Revised: Z where ( Y, Z ) =

Original: (0 , 0) with probability 1 / 

Revised: (0 , 0) with probability 1 /

Original: and ( Y, Z ) =
Revised: and ( Y, Z ) =

Original: (1 , 1) with probability 1 / 2 .
Revised: (1 , 1) with probability 1 / 2 .

Original: Wesolve this classiﬁcation problem using the network shown in Fig.
Revised: We solve this classiﬁcation problem using the network shown in Fig.

Original: 1 with σ linear and ϕ sigmoid.
Revised: 1 with σ linear and ϕ sigmoid.

Original: We will train this model using the binary cross entropy loss function.
Revised: We will train this model using the binary cross entropy loss function.

Original: Letting X
Revised: Letting X

Original: =
Revised: =

Original: ( Y, Z ) andθ =
Revised: ( Y, Z ) and θ =

Original: [Equation]

Original:  1 , W 2 , W 3 , W 4 ) ,
Revised: ( W 1 , W 2 , W 3 , W 4 ) ,

Original: [Equation]
Revised: [Equation]

Original: i
Revised: and

Original: [Equation]
Revised: [Equation]

Original: (18) andˆ [Equation] = [Equation]
Revised: From this, we compute,

Original: Z =

Original: [Equation]
Revised: [Equation]

Original: Moreover,˙
Revised: Moreover,

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: and, consequently,

Original: [Equation]
Revised: [Equation]

Original: 

Original: [Equation]
Revised: [Equation]

Original: 
Revised: Finally, letting ¨ F ( θ ) = ∇ 2 F ( ψ ) | ψ = θ ,

Original: [Equation]
Revised: [Equation]

Original: and, consequently, [Equation]
Revised: where I 4 is the 4 × 4 identity matrix.

Original: We ﬁrst establish that ˙ F ( θ ) is not globally Lipschitz continuous.
Revised: We ﬁrst establish that ˙ F ( θ ) is not globally Lipschitz continuous.

Original: With θ = (1 , − 1 , W 3 , W 3 ) andφ = (1 , 0 , 0 , 0) , it is enough to ﬁnd a lower bound for the ﬁrst component of ˙ F ( θ ) − ˙ F ( φ ) , denotedby ˙ F 1 ( θ ) − ˙ F 1 ( φ ) .
Revised: With θ = (1 , − 1 , W 3 , W 3 ) and φ = (1 , − 1 , W 3 , 0) , it is enough to ﬁnd a lower bound for the ﬁrst component of ˙ F ( θ ) − ˙ F ( φ ) , denoted by ˙ F 1 ( θ ) − ˙ F 1 ( φ ) .

Original: To this end,
Revised: To this end,

Original: [Equation]
Revised: [Equation]

Original: Thus, ˙ F is not globally Lipschitz.
Revised: Thus, ˙ F is not globally Lipschitz.

Revised: We now establish that F does not satisfy ( L 0 , L 1 ) -smoothness.

Revised: That is, we show that there is no

Revised: L 0 , L 1 ≥ 0 such that (cid:107) ¨ F ( θ ) (cid:107) ≤ L 0 (cid:107) ˙

Revised: F ( θ ) (cid:107) + L 1 , where the norms can be chosen arbitrarily owing to the equivalence of norms in ﬁnite-dimensional vector spaces.

Original: To show that the variance is not bounded, we study the variance of the ﬁrst component of ˙ f
Revised: To see this, note that the Frobenius norm of ¨ F ( θ ) is lower bounded by the absolute value of the [1 , 1] entry. Using notation, [Equation] Let θ =

Revised: (0 , W 4 , W 4 , W 4 ) , then the lower bound is

Revised: [Equation]

Revised: [Equation]

Revised: Notice, for this same choice of θ , the l 1 norm of the gradient is bounded above by

Revised: [Equation]

Revised: For any choice of L 0 , L 1 > 0 , we conclude that there is a W 4 sufﬁciently large such that, for this parametrization of θ ,

Revised: [Equation]

Revised: Thus, we see that no L 0 nor L 1 can exist that will satisfy the ( L 0 , L 1 ) -smooth assumption for all choices of θ .

Revised: To show that the variance is not bounded, we study the variance of the ﬁrst component of ˙ f

Original: ( θ, X )which we denote by ˙
Revised: ( θ, X ) which we denote by ˙

Original: f 1 ( θ, X ) .
Revised: f 1 ( θ, X ) .

Original: By direct calculation,
Revised: By direct calculation,

Original: [Equation]
Revised: [Equation]

Original: We again consider θ =
Revised: We again consider θ =

Original: (1 , − 1 , W 3 , W 3 ) , then the variance at this value of θ is
Revised: (1 , − 1 , W 3 , W 3 ) , then the variance at this value of θ is

Original: [Equation]
Revised: [Equation]

Original: Therefore, as W 3 → ∞ , the variance goes to inﬁnity.
Revised: Therefore, as W 3 → ∞ , the variance goes to inﬁnity.

Original: That is, the variance of the stochastic gradientsis unbounded.
Revised: That is, the variance of the stochastic gradients is unbounded.

Original: On the other hand, the problem does satisfy our assumptions.
Revised: On the other hand, the problem does satisfy our assumptions.

Original: In particular,1.
Revised: In particular, 1.

Original: Assumptions 1 and 3 are easily veriﬁed.2.
Revised: Assumptions 1 and 3 are easily veriﬁed.

Original: Given that ˙ F is continuously differentiable, then compactness and continuity of the derivativeof ˙ F imply that it is locally Lipschitz continuous.
Revised: Given that ˙ F is continuously differentiable, then compactness and continuity of the derivative of ˙ F imply that it is locally Lipschitz continuous.

Original: Therefore, Assumption 2 is satisﬁed.3.
Revised: Therefore, Assumption 2 is satisﬁed.

Original: Given the computation of the variance for the ﬁrst component, we have E [ ˙ f 1 ( θ, X ) 2 ] is 
Revised: Given the computation of the variance for the ﬁrst component, we have E [ ˙ f 1 ( θ, X ) 2 ] is [Equation]

Original: [Equation]
Revised: [Equation]

Original: 25) which is a continuous function.
Revised: which is a continuous function.

Original: By repeating this argument for each component, we concludethat Assumption 4 is satisﬁed.
Revised: By repeating this argument for each component, we conclude that Assumption 4 is satisﬁed.

Original: A.3 Recurrent Neural Network for Binary Classiﬁcation
Revised: A.3 Recurrent Neural Network for Binary Classiﬁcation

Original: Consider observing one of two sequences (1 , 0 , 0 , 0) or (0 , 0 , 0 , 0) with equal probabilities, andsuppose that each sequence corresponds to the label 1 or 0 , respectively.
Revised: Consider observing one of two sequences (1 , 0 , 0 , 0) or (0 , 0 , 0 , 0) with equal probabilities, and suppose that each sequence corresponds to the label 1 or 0 , respectively.

Original: Now consider Fig.
Revised: Now consider Fig.

Original: 2 to be a1-dimensional linear recurrent neural network which reads each element of the sequence and uses alogistic output layer to predict either a label of one or zero.
Revised: 2 to be a 1-dimensional linear recurrent neural network which reads each element of the sequence and uses a logistic output layer to predict either a label of one or zero.

Original: If we ﬁx H 0 = 0, W 2 = 1 and W 3 = 1 ,then the model predicts the probability of a 1 label as
Revised: If we ﬁx H 0 = 0 and W 3 = 1 , then the model predicts the probability of a 1 label as

Original: [Equation]
Revised: [Equation]

Original: If we use the binary cross entropy loss with (cid:96) 2 regularization, and let X
Revised: If we use the binary cross entropy loss with (cid:96) 2 regularization, and let X

Original: =
Revised: =

Original: ( Y, Z 0 , Z 1 , Z 2 , Z 3 ) andθ =
Revised: ( Y, Z 0 , Z 1 , Z 2 , Z 3 ) and θ =

Original:  W 1  then 
Revised: ( W 1 , W 2 ) then [Equation] [Equation] 35) and

Original: [Equation]
Revised: [Equation]

Revised: Taking the expectations, we compute

Revised: [Equation]

Revised: 37) and

Original: [Equation]
Revised: [Equation]

Revised: Taking another derivative and letting ¨ F ( θ )

Revised: =

Revised: ∇ 2 F ( ψ ) | ψ = θ ,

Original: [Equation]
Revised: [Equation]

Original:  and 
Revised: We ﬁrst establish that ˙ F is not globally Lipschitz continuous. Notice, if we set W 2 = 1 , then the ﬁrst and second component of ˙ F ( θ ) are proportional to − W 21 and − W 31 respectively, which are not globally Lipschitz continuous functions. We now show that F also does not satisfy ( L 0 , L 1 ) -smoothness. Notice that, using the bottom right entry of ¨ F ( θ ) ,

Original: [Equation]
Revised: [Equation]

Original: Taking the expectations, we compute
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Revised: [Equation]

Revised: [Equation]

Revised: If we choose W 2 = 0 , then, for any L 0 , L 1 ≥ 0 there exists a | W 1 | sufﬁciently large such that

Original: [Equation]
Revised: [Equation]

Original: From this calculation alone, it is easy to see that as θ → −∞ , ˙ F ( θ )∝ − θ 2 , which is not a Lipschitzcontinuous function.
Revised: + L [Equation] [Equation] Hence, F ( θ ) is not ( L 0 , L 1 ) -smooth.

Original: Moreover, computing the  variance of ˙ f
Revised: Moreover, computing the trace of the variance of ˙ f

Original: ( θ, X ) , we recover
Revised: ( θ, X ) , we recover

Original: [Equation]
Revised: [Equation]

Original: which does not satisfy a bounded variance assumption.
Revised: which does not satisfy a bounded variance assumption (choose W 2 = 0 and let W 1 → ∞ ).

Original: Thus, any work that makes either a global Lipschitz bound on the gradient or a global noise model bound fails to apply to this simple recurrentneural network training problem.
Revised: Thus, any work that makes either a global Lipschitz bound on the gradient or a global noise model bound fails to apply to this simple recurrent neural network training problem.

Original: On the other hand, the problem does satisfy our assumptions.
Revised: On the other hand, the problem does satisfy our assumptions.

Original: In particular,1.
Revised: In particular, 1.

Original: Assumptions 1 and 3 are easily veriﬁed.2.
Revised: Assumptions 1 and 3 are easily veriﬁed.

Original: Given that ˙ F is continuously differentiable, then compactness and continuity of the derivativeof ˙ F imply that it is locally Lipschitz continuous.
Revised: Given that ˙ F is continuously differentiable, then compactness and continuity of the derivative of ˙ F imply that it is locally Lipschitz continuous.

Original: Therefore, Assumption 2 is satisﬁed.3.
Revised: Therefore, Assumption 2 is satisﬁed.

Original: Moreover,
Revised: Moreover

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: which is a continuous function.
Revised: which is a continuous function.

Original: Hence, Assumption 4 is satisﬁed.
Revised: Hence, Assumption 4 is satisﬁed.

Original: A.4 Noiseless Feed Forward Network for Binary Classiﬁcation
Revised: A.4 Poisson Regression

Revised: Here, we consider the task of estimating a Poisson regression model for data X = ( Y, Z ) where Y is a count response variable and Z is the covariate. To make this problem simpler, we will assume that both Y and Z are independent Poisson random variables with mean 1 , which implies that the parameter in the model, θ ∗ = 0 .

Revised: If we use a likelihood framework, then, up to a constant depending on Y ,

Revised: [Equation]

Revised: [Equation]

Revised: and

Revised: [Equation]

Revised: From this, we compute

Revised: [Equation]

Revised: [Equation]

Revised: ˙

Revised: [Equation]

Revised: and, letting ∇ 2 F ( ψ ) | ψ = θ = ¨ F ( θ

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: We begin by showing that ˙ F ( θ ) is not globally Lipschitz continuous.

Revised: To do so, for any θ > 0 , note | ˙ F ( θ ) − ˙ F (0) | =

Revised: exp(exp( θ ) + θ − 1) −

Revised: [Equation]

Revised: > exp

Revised: [Equation]

Revised: Thus, for any L > 0 there exists a θ > 0 such that | ˙ F ( θ ) − ˙ F (0) | > L | θ | .

Revised: We now show that F ( θ ) does not satisfy the ( L 0 , L 1 ) -smooth assumption.

Revised: Note, for θ ≥ 0 ,

Revised: [Equation]

Revised: and

Revised: [Equation]

Revised: [Equation]

Revised: It follows that for any L 0 , L 1 > 0 , there exists a θ > 0 such that L 0 | ˙ F ( θ ) | + L 1 < ¨ F ( θ ) .

Revised: For the noise, we compute the second moment of ˙ f

Revised: ( θ, X ) .

Revised: That is,

Revised: [Equation]

Revised: It is clear from this calculation that the variance (computed by subtracting off ˙ F ( θ ) 2 ) will diverge as θ tends to inﬁnity.

Revised: To show that [Bottou et al., 2018, Assumption 4.3c] does not apply, it is enough to show that its generalization, [Khaled and Richtárik, 2020, Assumption 2] does not apply.

Revised: To this end, we must show that there does not exists a C 0 , C 1 , C 2 ≥ 0 such that, ∀ θ ,

Revised: [Equation]

Revised: [Equation]

Revised: From our calculations, it is easy to verify that F ( θ ) and ˙ F ( θ ) are dominated by exp(2 exp( θ )) , and that the second moment of the stochastic gradient is bounded from below by exp(exp(2 θ )) for θ ≥ log(4) .

Revised: Hence, for any C 0 , C 1 , C 2 ≥ 0 , there exists θ sufﬁciently large such that

Revised: [Equation]

Revised: [Equation]

Revised: Thus, [Bottou et al., 2018, Assumption 4.3c] and [Khaled and Richtárik, 2020, Assumption 2] do not hold.

Revised: On the other hand, the problem does satisfy our assumptions.

Revised: In particular,

Revised: Assumptions 1 and 3 are easily veriﬁed.

Revised: Given that ¨ F is continuous, Assumption 2 is satisﬁed.

Revised: Moreover, we can use

Revised: the calculated

Revised: value E [ ˙ f ( θ, X ) 2 ] , which is

Revised: a continuous

Revised: function, as G ( θ ) to satisfy Assumption 4.

Revised: A.5 Noiseless Feed Forward Network for Binary Classiﬁcation

Original: Out of interest, we reconsider the second example but construct a different data distribution thatproduces noiseless stochastic gradient.
Revised: Out of interest, we reconsider the second example but construct a different data distribution that produces noiseless stochastic gradient.

Original: Consider the binary classiﬁcation problem with label Y andfeature Z where ( Y, Z ) 
Revised: Consider the binary classiﬁcation problem with label Y and feature Z where ( Y, Z ) =

Original: = (0 , − 1) with probability 1 / 2 and ( Y, Z ) =
Revised: (0 , − 1) with probability 1 / 2 and ( Y, Z ) =

Original: (1 , 1) with probability 1 / 2 .
Revised: (1 , 1) with probability 1 / 2 .

Original: We solve this classiﬁcation problem using the network shown in Fig.
Revised: We solve this classiﬁcation problem using the network shown in Fig.

Original: 1 with σ linear and ϕ sigmoid.
Revised: 1 with σ linear and ϕ sigmoid.

Original: We will train this model using the binary cross entropy loss function.
Revised: We will train this model using the binary cross entropy loss function.

Original: Letting X
Revised: Letting X

Original: =
Revised: =

Original: ( Y, Z ) and [Equation] = 
Revised: ( Y, Z ) and θ = ( W 1 ,

Original: [Equation]
Revised: [Equation]

Original: W 1 , W 2 , W 3 , W 4 ) ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Moreover,
Revised: Moreover,

Original: [Equation]
Revised: [Equation]

Original: and, consequently,
Revised: and, consequently,

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst establish that ˙ F ( θ ) is not globally Lipschitz continuous.
Revised: We ﬁrst establish that ˙ F ( θ ) is not globally Lipschitz continuous.

Original: With θ = (1 , − 1 , W 3 , W 3 ) andφ = (1 , 0 , 0 , 0) , it is enough to ﬁnd a lower bound for the ﬁrst component of ˙ F ( θ ) − ˙ F ( φ ) , denotedby ˙ F 1 ( θ ) − ˙ F 1 ( φ ) .
Revised: With θ = (1 , − 1 , W 3 , W 3 ) and φ = (1 , 0 , 0 , 0) , it is enough to ﬁnd a lower bound for the ﬁrst component of ˙ F ( θ ) − ˙ F ( φ ) , denoted by ˙ F 1 ( θ ) − ˙ F 1 ( φ ) .

Original: To this end,
Revised: To this end,

Original: [Equation]
Revised: [Equation]

Original: Thus, ˙ F is not globally Lipschitz.
Revised: Thus, ˙ F is not globally Lipschitz.

Original: On the other hand, the problem does satisfy our assumptions.
Revised: On the other hand, the problem does satisfy our assumptions.

Original: In particular,1.
Revised: In particular, 1.

Original: Assumptions 1 and 3 are easily veriﬁed.2.
Revised: Assumptions 1 and 3 are easily veriﬁed.

Original: Given that ˙ F is continuously differentiable, then compactness and continuity of the derivativeof ˙ F imply that it is locally Lipschitz continuous.
Revised: Given that ˙ F is continuously differentiable, then compactness and continuity of the derivative of ˙ F imply that it is locally Lipschitz continuous.

Original: Therefore, Assumption 2 is satisﬁed.3.
Revised: Therefore, Assumption 2 is satisﬁed.

Original: Moreover, ˙ f ( θ, Z ) = ˙
Revised: Moreover, ˙ f ( θ, Z ) = ˙

Original: F ( θ ) —that is, there ˙ f
Revised: F ( θ ) —that is, there ˙ f

Original: ( θ, Z ) has zero variance for the distribution thatwe have constructed.
Revised: ( θ, Z ) has zero variance for the distribution that we have constructed.

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original:  which is a continuous function.
Revised: ) which is a continuous function.

Original: Hence, Assumption 4 is satisﬁed.
Revised: Hence, Assumption 4 is satisﬁed.

Original: B Technical Lemmas
Revised: B Technical Lemmas

Original: Lemma 5 (Lemma 1) .
Revised: Lemma 5 (Lemma 1) .

Original: Suppose Assumptions 1 and 2 hold.
Revised: Suppose Assumptions 1 and 2 hold.

Original: Then, for any θ, ϕ ∈ R p ,
Revised: Then, for any θ, ϕ ∈ R p ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Proof.
Revised: Proof.

Original: By Taylor’s theorem,
Revised: By Taylor’s theorem,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Now, add and subtract ˙
Revised: Now, add and subtract ˙

Original: F ( θ ) to ˙ F ( θ + t ( ϕ − θ )) in the integral, then apply Assumption 2.
Revised: F ( θ ) to ˙ F ( θ + t ( ϕ − θ )) in the integral, then apply Assumption 2.

Original: By
Revised: By

Original: Deﬁnition 1,
Revised: Deﬁnition 1,

Original: [Equation]
Revised: [Equation]

Original: We conclude,
Revised: We conclude,

Original: [Equation]
Revised: [Equation]

Original: By computing the integral, the result follows.
Revised: By computing the integral, the result follows.

Original: Lemma 6.
Revised: Lemma 6.

Original: Suppose { M k : k + 1 ∈ N } satisfy Properties 1 and 4 .
Revised: Suppose { M k : k + 1 ∈ N } satisfy Properties 1 and 4 .

Original: Then ∀ C > 0 , ∃ K ∈ N such that∀ k ≥ K ,
Revised: Then ∀ C > 0 , ∃ K ∈ N such that ∀ k ≥ K ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Proof.
Revised: Proof.

Original: Fix C > 0 .
Revised: Fix C > 0 .

Original: Rearranging the conclusion, we see that it is equivalent to prove that ∃ K ∈ Nsuch that ∀ k ≥ K , 1 /C ≥ λ max ( M k ) α κ ( M k ) .
Revised: Rearranging the conclusion, we see that it is equivalent to prove that ∃ K ∈ N such that ∀ k ≥ K , 1 /C ≥ λ max ( M k ) α κ ( M k ) .

Original: This follows from Property 4.
Revised: This follows from Property 4.

Original: Lemma 7.
Revised: Lemma 7.

Original: For any θ ∈ R p , v ∈ R , L > 0 and α ∈ (0 , 1] ,
Revised: For any θ ∈ R p , v ∈ R , L > 0 and α ∈ (0 , 1] ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: If we minimize the left hand side of the inequality, we see that a minimum value occurs whenv α = (cid:13)(cid:13)(cid:13) ˙
Revised: If we minimize the left hand side of the inequality, we see that a minimum value occurs when v α = (cid:13)(cid:13)(cid:13) ˙

Original: F ( θ ) (cid:13)(cid:13)(cid:13) 2 /L ≥ 0 .
Revised: F ( θ ) (cid:13)(cid:13)(cid:13) 2 /L ≥ 0 .

Original: Solving for v and plugging this back into the left hand side, we concludethat the inequality holds.
Revised: Solving for v and plugging this back into the left hand side, we conclude that the inequality holds.

Original: C Global Convergence Analysis
Revised: C Global Convergence Analysis

Original: We begin by ﬁrst deriving a recursive relationship between the optimality gap at iteration k + 1 andthe optimality gap at iteration k on the events {B j ( R ) } as deﬁned in (7) for arbitrary R ≥ 0 .
Revised: We begin by ﬁrst deriving a recursive relationship between the optimality gap at iteration k + 1 and the optimality gap at iteration k on the events {B j ( R ) } as deﬁned in (8) for arbitrary R ≥ 0 .

Original: Usingthis result, we then provide an analysis of the convergence of the objective function.
Revised: Using this result, we then provide an analysis of the convergence of the objective function.

Original: Then, we turnour attention to the gradient function.
Revised: Then, we turn our attention to the gradient function.

Original: Note, B ( θ, r ) is the open ball around θ of radius r .
Revised: Note, B ( θ, r ) is the open ball around θ of radius r .

Original: C.1 A Recursive Relationship
Revised: C.1 A Recursive Relationship

Original: Lemma 8 (Lemma 2) .
Revised: Lemma 8 (Lemma 2) .

Original: Let { M k } satisfy Property 1 .
Revised: Let { M k } satisfy Property 1 .

Original: Suppose Assumptions 1 to 4 hold.
Revised: Suppose Assumptions 1 to 4 hold.

Original: Let { θ k }satisfy (4) .
Revised: Let { θ k } satisfy (5) .

Original: Then, ∀ R ≥ 0 ,
Revised: Then, ∀ R ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: where G R = sup θ ∈ B (0 ,R ) G ( θ ) < ∞ with G ( θ ) ; and ∂F R = sup θ ∈ B ( [Equation] R
Revised: where G R = sup θ ∈ B (0 ,R ) G ( θ ) < ∞ with G ( θ ) ; and ∂F R = sup θ ∈ B (0 ,R ) (cid:107) ˙

Original: [Equation]

Original: ˙ F ( θ ) (cid:107) [-
-] (1 + α ) < ∞ .
Revised: F ( θ ) (cid:107) [-
-]2 (1 + α ) < ∞ .

Original: Proof.
Revised: Proof.

Original: Fix R ≥ 0 .
Revised: Fix R ≥ 0 .

Original: For any k + 1 ∈ N , the deﬁnition of local Hölder continuity implies that L R +1 iswell deﬁned (see Deﬁnition 1).
Revised: For any k + 1 ∈ N , the deﬁnition of local Hölder continuity implies that L R +1 is well deﬁned (see Deﬁnition 1).

Original: Therefore, Lemma 1 implies
Revised: Therefore, Lemma 1 implies

Original: [Equation]
Revised: [Equation]

Original: Now, since B (0 , R ) ⊂ B (0 , R + 1) , it also holds true that
Revised: Now, since B (0 , R ) ⊂ B (0 , R + 1) , it also holds true that

Original: [Equation]
Revised: [Equation]

Original: Our goal now is to replace B k
Revised: Our goal now is to replace B k

Original: +1 ( R ) on the right hand side by B k
Revised: +1 ( R ) on the right hand side by B k

Original: ( R ) .
Revised: ( R ) .

Original: However, there is a technicaldifﬁculty which we must address.
Revised: However, there is a technical difﬁculty which we must address.

Original: First, it follows from the preceding inequality that
Revised: First, it follows from the preceding inequality that

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term on the right hand side of the inequality only contributes meaningfully if it is positive.
Revised: The ﬁrst term on the right hand side of the inequality only contributes meaningfully if it is positive.

Original: Since 1 [ B k ( R )] ≥ 1 [ B k +1 ( R )] , then two statements hold: (i) 

Revised: Since 1 [ B k ( R )] ≥ 1 [ B k +1 ( R )] , then two statements hold: (i)

Original: [ B k ( R )] 1 [ B k +1 ( R )]
Revised: [ B k ( R )] 1 [ B k +1 ( R )]

Original: = 

Revised: =

Original: [ B k +1 ( R )] ;and (ii) the ﬁrst term of the right hand side of (50) is positive if and only if 
Revised: [ B k +1 ( R )] ; and (ii) the ﬁrst term of the right hand side of (73) is positive if and only if (cid:18)

Original: [Equation]
Revised: [Equation]

Original: By the choice of L R +1 , Assumption 1 and Lemma 1 imply that if (51) occurs, then (cid:107) θ k +1 (cid:107) 2 > R +1 ≥(cid:107) θ k (cid:107) 2 +1 .
Revised: By the choice of L R +1 , Assumption 1 and Lemma 1 imply that if (74) occurs, then (cid:107) θ k + [Equation] 1 ≥

Original:  By the reverse triangle inequality and (4), if (51) occurs, then (cid:107) M k ˙ f
Revised: [Equation] θ k (cid:107) 2 +1 . By the reverse triangle inequality and (5), if (74) occurs, then (cid:107) M k ˙ f

Original: ( θ k , X k +1 ) (cid:107) 2 ≥ 1 .
Revised: ( θ k , X k +1 ) (cid:107) 2 ≥ 1 .

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: We now compute another coarse upper bound for this inequality.
Revised: We now compute another coarse upper bound for this inequality.

Original: Note, by Assumption 1 and
Revised: Note, by Assumption 1 and

Original: Cauchy-Schwarz,
Revised: Cauchy-Schwarz,

Original: [Equation]
Revised: [Equation]

Original: ≤ [Equation] where ∂F R = sup θ ∈ B (0 ,R ) (cid:107) ˙
Revised: where ∂F R = sup θ ∈ B (0 ,R ) (cid:107) ˙

Original: F ( θ ) (cid:107) 2 (1 + α ) < ∞ given that (cid:107) ˙
Revised: F ( θ ) (cid:107) 2 (1 + α ) < ∞ given that (cid:107) ˙

Original: F ( θ ) (cid:107) 2 is a continuous function of θ .
Revised: F ( θ ) (cid:107) 2 is a continuous function of θ .

Original: Applying this inequality to (50)
Revised: Applying this inequality to (73)

Original: [Equation]
Revised: [Equation]

Original: we conclude[ F [Equation] θ k +
Revised: we conclude [ F ( θ k +1 )

Original: [Equation]

Original: − F l.b. ] 

Revised: − F l.b. ]

Original: [ B k +1 ( R )] ≤
Revised: [ B k +1 ( R )] ≤

Original: [Equation]
Revised: [Equation]

Original:  × 1 [ B k ( R )] .[Equation]
Revised: (cid:19) × 1 [ B k ( R )] .

Original: By Assumption 3,
Revised: By Assumption 3,

Revised: E

Original: [Equation]
Revised: [Equation]

Original:  × 1 [ B k ( R )] .[Equation]
Revised: [Equation] × 1 [ B k ( R )] .

Original: Using Property 1 and Assumption 4,
Revised: Using Property 1 and Assumption 4,

Original: [Equation]
Revised: [Equation]

Original: By Assumption 4, G is upper semicontinuous and B (0 , R ) is compact, which implies that G R is welldeﬁned and ﬁnite.
Revised: By Assumption 4, G is upper semicontinuous and B (0 , R ) is compact, which implies that G R is well deﬁned and ﬁnite.

Original: The result follows.
Revised: The result follows.

Original: C.2 Objective Function Analysis
Revised: C.2 Objective Function Analysis

Original: Corollary 1.
Revised: Corollary 1.

Original: Let { θ k } be deﬁned as in (4) satisfying Properties 1 and 2 .
Revised: Let { θ k } be deﬁned as in (5) satisfying Properties 1 and 2 .

Original: Suppose Assumptions 1to 4 hold.
Revised: Suppose Assumptionsto 4 hold.

Original: Then, there exists a ﬁnite random variable F lim such that on the event { sup k (cid:107) θ k 
Revised: Then, there exists a ﬁnite random variable F lim such that on the event { sup k (cid:107) θ k (cid:107) 2 < ∞} , lim k →∞ F ( θ k ) = F lim with probability one.

Original: Proof.
Revised: Proof.

Original: By Lemma 2, for every R ≥ 0 ,
Revised: By Lemma 2, for every R ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: By Neveu and Speed [1975, Exercise II.4] (cf.
Revised: By Neveu and Speed [1975, Exercise II.4] (cf.

Original: Robbins and Siegmund [1971]) and Property 2,lim k →∞ [ F ( θ k ) − F l.b. ] 

Revised: Robbins and Siegmund [1971]) and Property 2, lim k →∞ [ F ( θ k ) − F l.b. ]

Original: [ B k ( R )] converges to a ﬁnite random variable with probability one.
Revised: [ B k ( R )] converges to a ﬁnite random variable with probability one.

Original: Since
Revised: Since

Original: R ≥ 0 is arbitrary, we conclude that there exists a ﬁnite random variable F lim such that the set{ sup k (cid:107) θ k (cid:107) 2 ≤ R } is a subset of { lim k F ( θ k ) =
Revised: R ≥ 0 is arbitrary, we conclude that there exists a ﬁnite random variable F lim such that the set { sup k (cid:107) θ k (cid:107) 2 ≤ R } is a subset of { lim k F ( θ k ) =

Original: F lim } up to a measure zero set.
Revised: F lim } up to a measure zero set.

Original: Since the countableunion of measure zero sets has measure zero,
Revised: Since the countable union of measure zero sets has measure zero,

Original: [Equation]
Revised: [Equation]

Original: up to a measure zero set.
Revised: up to a measure zero set.

Original: The result follows.
Revised: The result follows.

Original: C.3 Gradient Function Analysis
Revised: C.3 Gradient Function Analysis

Original: We now prove that the gradient norm evaluated at SGD’s iterates must, repeatedly, get arbitrarilyclose to zero.
Revised: We now prove that the gradient norm evaluated at SGD’s iterates must, repeatedly, get arbitrarily close to zero.

Original: Lemma 9.
Revised: Lemma 9.

Original: Let { θ k } be deﬁned as in (4) satisfying Properties 1 to 3 .
Revised: Let { θ k } be deﬁned as in (5) satisfying Properties 1 to 3 .

Original: Suppose Assumptions 1 to 4hold.
Revised: Suppose Assumptions 1 tohold.

Original: Then, ∀ R ≥ 0 and for all δ > 0 ,
Revised: Then, ∀ R ≥ 0 and for all δ > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By Lemma2,
Revised: By Lemma

Original: [Equation]
Revised: [Equation]

Original: Taking the sum of this equation for all k from 0 to j ∈ N, we have
Revised: Taking the sum of this equation for all k from 0 to j ∈ N

Original: [Equation]
Revised: [Equation]

Revised: we have

Original: [Equation]
Revised: [Equation]

Original: By Assumption 1 and Property 2, the right hand side is bounded by
Revised: By Assumption 1 and Property 2, the right hand side is bounded by

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: (65) which is ﬁnite.
Revised: which is ﬁnite.

Original: Therefore, (cid:80) ∞
Revised: Therefore, (cid:80) ∞

Original: k =

Revised: k =

Original: λ min ( M k ) E
Revised: λ min ( M k ) E

Original: [ (cid:107) ˙
Revised: [ (cid:107) ˙

Original: F ( θ k ) (cid:107) 22 

Revised: F ( θ k ) (cid:107) 22

Original: [ B k ( R )]] is ﬁnite.
Revised: [ B k ( R )]] is ﬁnite.

Original: Furthermore, by Property 3, lim inf k E [ (cid:107) ˙
Revised: Furthermore, by Property 3, lim inf k E [ (cid:107) ˙

Original: F ( θ k ) (cid:107) 22 

Revised: F ( θ k ) (cid:107) 22

Original: [ B k ( R )]] = 0 .
Revised: [ B k ( R )]] = 0 .

Original: Now, for any δ > 0 , Markov’s inequality implies that for all j + 1 ∈ N , 
Revised: Now, for any δ > 0 , Markov’s inequality implies that for all j + 1 ∈ N , and for all k ≥ j

Original: [Equation]
Revised: [Equation]

Revised: Since the last inequality holds for every k ≥ j , then, in particular, for all j + 1 ∈ N ,

Original:  where the right hand side is zero because lim inf k E [(cid:107) ˙
Revised: [Equation] [Equation] where the right hand side is zero because lim inf k E [

Original:  F [-( θ k ) (cid:107) 22 
-]
Revised: [Equation] F [-( θ k ) (cid:107) 22 
-][Equation]

Original: [ B k ( R )]] = 0 .
Revised: [Equation] R )]] = 0 .

Original: As the countable union of measure zero sets has measure zero, we conclude that for all δ > 0 ,
Revised: As the countable union of measure zero sets has measure zero, we conclude that for all δ > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Unfortunately, Lemma 9 does not guarantee that the gradient norm will be captured within a regionof zero.
Revised: Unfortunately, Lemma 9 does not guarantee that the gradient norm will be captured within a region of zero.

Original: In order to prove this, we ﬁrst show that it is not possible (i.e., a zero probability event) forthe limit supremum and limit inﬁmum of the gradients to be distinct.
Revised: In order to prove this, we ﬁrst show that it is not possible (i.e., a zero probability event) for the limit supremum and limit inﬁmum of the gradients to be distinct.

Original: Lemma 10.
Revised: Lemma 10.

Original: Let { θ k } be deﬁned as in (4) satisfying Properties 1 and 2 .
Revised: Let { θ k } be deﬁned as in (5) satisfying Properties 1 and 2 .

Original: Suppose Assumptions 1 to 4hold.
Revised: Suppose Assumptions 1 tohold.

Original: Then, ∀ R ≥ 0 and for all δ > 0 ,
Revised: Then, ∀ R ≥ 0 and for all δ > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let γ > 0 .
Revised: Let γ > 0 .

Original: Let L R be as in Deﬁnition 1, and G R be as in Lemma 2.
Revised: Let L R be as in Deﬁnition 1, and G R be as in Lemma 2.

Original: Then, for δ > 0 ,
Revised: Then, for δ > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Using the reverse triangle inequality, (cid:107) ˙
Revised: Using the reverse triangle inequality, (cid:107) ˙

Original: F [Equation] θ k +1 ) [Equation]
Revised: F ( θ k +1 ) (cid:107) 2 − (cid:107) ˙

Original: 2 − [Equation] ˙ F ( θ k ) (cid:107) 2 ≤ (cid:107) ˙
Revised: F ( θ k ) (cid:107) 2 ≤ (cid:107) ˙

Original: F ( θ k +1 ) − ˙ F ( θ k ) (cid:107) 2 .
Revised: F ( θ k +1 ) − ˙ F ( θ k ) (cid:107) 2 .

Original: Now,making use of the restriction to B k +1 ( R ) , (cid:107) ˙
Revised: Now, making use of the restriction to B k +1 ( R ) , (cid:107) ˙

Original: F ( θ k +1 ) − ˙ F ( θ k ) (cid:107) 2 ≤ L R (cid:107)
Revised: F ( θ k +1 ) − ˙ F ( θ k ) (cid:107) 2 ≤ L R (cid:107)

Original: θ k +1 − θ k (cid:107)
Revised: θ k +1 − θ k (cid:107)

Original: α 2 .
Revised: α 2 .

Original: Moreover, on(cid:107) ˙
Revised: Moreover, on (cid:107) ˙

Original: F ( θ k ) (cid:107) 2 ≤ δ , (cid:107) ˙
Revised: F ( θ k ) (cid:107) 2 ≤ δ , (cid:107) ˙

Original: F ( θ k ) (cid:107) 2 

Revised: F ( θ k ) (cid:107) 2

Original: [ B k +1 ( R )] ≤ δ .
Revised: [ B k +1 ( R )] ≤ δ .

Original: Putting these two observations together,
Revised: Putting these two observations together,

Original: [Equation]
Revised: [Equation]

Original: [Equation] Now, using1 [ B k +1 ( R )] 1 (cid:104)(cid:13)(cid:13)(cid:13) ˙ F ( θ k )
Revised: Now, using

Original: [Equation]
Revised: [Equation]

Original: 2 1 [ B k ( R )] [Equation] ≤ 1 [ B k ( R )] and Markov’s inequality, 
Revised: where the last inequality is a consequence of Markov’s inequality, (cid:107) M k ˙

Original: [Equation] [Equation] ≤ [Equation]
Revised: f ( θ k , X k +1 ) (cid:107) 2 ≤ (cid:107) M k (cid:107) 2 (cid:107) ˙

Original: B
Revised: f

Original: [Equation] [Equation] [Equation] R )](cid:21) .
Revised: ( θ k , X k +1 ) (cid:107) 2 , and 1 [ B k ( R )]

Revised: being measurable with respect to F k .

Original: By Assumption 4, E (cid:104) (cid:107) ˙
Revised: By Assumption 4, E (cid:104) (cid:107) ˙

Original: f
Revised: f

Original: ( θ k , X k +1 ) (cid:107) 1+ α(cid:12)(cid:12) (cid:12) F k (cid:105) ≤ G ( θ k ) .
Revised: ( θ k , X k +1 ) (cid:107) 1+ α(cid:12)(cid:12)(cid:12) F k (cid:105) ≤ G ( θ k ) .

Original: Moreover, on B k ( R ) , G ( θ k ) ≤ G R .
Revised: Moreover, on B k ( R ) , G ( θ k ) ≤ G R .

Original: Usingthis in the expectation, we conclude
Revised: Using this in the expectation, we conclude

Original: [Equation]

Original: δ
Revised: P

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: By Property 2, the sum of the last expression over all k + 1 ∈ N is ﬁnite.
Revised: By Property 2, the sum of the last expression over all k + 1 ∈ N is ﬁnite.

Original: By the Borel-Cantellilemma, for all R ≥ 0 , δ > 0 and γ > 0 ,
Revised: By the Borel-Cantelli lemma, for all R ≥ 0 , δ > 0 and γ > 0 ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Since this holds for any γ > 0 , it will hold for every value in a sequence γ n ↓ 0 .
Revised: Since this holds for any γ > 0 , it will hold for every value in a sequence γ n ↓ 0 .

Original: Since the countableunion of measure zero events has measure zero, 
Revised: Since the countable union of measure zero events has measure zero, for any R ≥ 0 and δ > 0 ,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: where Ω δ =

Revised: { lim sup k (cid:107) ˙

Original: {+F ( θ k +1 ) (cid:107) 2 1 [ B k +
+}
Revised: {+F ( θ k +1 ) (cid:107) 2 1 [ B k +
+}

Revised: ( R )]

Revised: = δ } .

Revised: We now show that Ω δ is a probability zero event.

Revised: Notice, by Lemma 9 and the deﬁnition of Ω δ , Ω

Revised: [Equation]

Original:  We now put together Lemmas 9 and 10 to show that, on the event { sup k (cid:107) θ k (cid:107) 2 < ∞} , (cid:107) ˙ F ( θ k ) (cid:107) 2converges to 0 with probability one.
Revised: (108) up to a set of measure zero. By applying (107) with δ/ 2 , P [Ω δ ] = 0 . The conclusion of the result follows. We now put together Lemmas 9 and 10 to show that, on the event { sup k (cid:107) θ k (cid:107) 2 < ∞} , (cid:107) ˙ F ( θ k ) (cid:107)converges to 0 with probability one.

Original: Corollary 2.
Revised: Corollary 2.

Original: Let { θ k } be deﬁned as in (4) satisfying Properties 1 to 3 .
Revised: Let { θ k } be deﬁned as in (5) satisfying Properties 1 to 3 .

Original: Suppose Assumptions 1 to 4hold.
Revised: Suppose Assumptions 1 tohold.

Original: Then, on the event { sup k (cid:107) θ k (cid:107) 2 < ∞} , lim k →∞ (cid:107) ˙
Revised: Then, on the event { sup k (cid:107) θ k (cid:107) 2 < ∞} , lim k →∞ (cid:107) ˙

Original: F ( θ k ) (cid:107) 2 = 0 .
Revised: F ( θ k ) (cid:107) 2 = 0 .

Original: Proof.
Revised: Proof.

Original: For any R ≥ 0 and δ > 0 , Lemma 9 implies
Revised: For any R ≥ 0 and δ > 0 , Lemma 9 implies

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: We see that this latter event is exactly,
Revised: We see that this latter event is exactly,

Original: [Equation]
Revised: [Equation]

Original: .o. (cid:105) , (83) which, by Lemma 10, is zero with probability one.
Revised: which, by Lemma 10, is zero with probability one.

Original: Therefore, P [ (cid:107) ˙
Revised: Therefore, P [ (cid:107) ˙

Original: F ( θ k +1 ) (cid:107) 2 1 [ B k +

Revised: F ( θ k +1 ) (cid:107) 2 1 [ B k +

Original: ( R )] > δ, i.o. ]is zero.
Revised: ( R )] > δ, i.o. ] is zero.

Original: Letting δ n ↓ 0 and noting that the countable union of measure zero sets has measure zero, weconclude P [ (cid:107) ˙
Revised: Letting δ n ↓ 0 and noting that the countable union of measure zero sets has measure zero, we conclude P [ (cid:107) ˙

Original: F ( θ k +1 ) (cid:107) 2 1 [ B k +

Revised: F ( θ k +1 ) (cid:107) 2 1 [ B k +

Original: ( R )] > 0 , i.o. ]
Revised: ( R )] > 0 , i.o. ]

Original: = 0 .
Revised: = 0 .

Original: Therefore, for all R ≥ 0 , { sup k (cid:107) θ k (cid:107) 2 ≤ R } ⊂ { lim k →∞ (cid:107) ˙
Revised: Therefore, for all R ≥ 0 , { sup k (cid:107) θ k (cid:107) 2 ≤ R } ⊂ { lim k →∞ (cid:107) ˙

Original: F ( θ k ) (cid:107) 2 = 0 } up to a measure zero set.
Revised: F ( θ k ) (cid:107) 2 = 0 } up to a measure zero set.

Original: Since { sup k (cid:107) θ k (cid:107) 2 < ∞} = ∪ R ∈ N { sup k (cid:107) θ k (cid:107) 2 ≤ R } , the result follows.
Revised: Since { sup k (cid:107) θ k (cid:107) 2 < ∞} = ∪ R ∈ N { sup k (cid:107) θ k (cid:107) 2 ≤ R } , the result follows.

Original: C.4 Capture Theorem
Revised: C.4 Capture Theorem

Original: The ﬁnal step in our proof is to study the event { sup k (cid:107) θ k (cid:107) < ∞} .
Revised: The ﬁnal step in our proof is to study the event { sup k (cid:107) θ k (cid:107) < ∞} .

Original: Theorem 4 (Theorem 1) .
Revised: Theorem 4 (Theorem 1) .

Original: Let { θ k } be deﬁned as in (4) , and let { M k } satisfy Properties 1 and 2 .
Revised: Let { θ k } be deﬁned as in (5) , and let { M k } satisfy Properties 1 and 2 .

Original: If
Revised: If

Original: Assumption 4 holds, then either { lim k →∞ θ k exists } or { lim inf k →∞ (cid:107) θ k (cid:107) 2 = ∞} must occur.
Revised: Assumption 4 holds, then either { lim k →∞ θ k exists } or { lim inf k →∞ (cid:107) θ k (cid:107) 2 = ∞} must occur.

Original: Proof.
Revised: Proof.

Original: Let ¯ θ ∈ R p .
Revised: Let ¯ θ ∈ R p .

Original: Fix R ≥ 0 and let γ > 0 .
Revised: Fix R ≥ 0 and let γ > 0 .

Original: Then,
Revised: Then,

Original: [Equation]

Original: [Equation]

Original: R

Original: [Equation]
Revised: [Equation]

Original: ≥ R + γ [Equation] Now, (cid:107) θ k − ¯ θ (cid:107) 2 1 (cid:2) (cid:107) θ k − ¯ θ (cid:107) 2 ≤ R (cid:3) ≤ R .
Revised: Now, (cid:107) θ k − ¯ θ (cid:107) 2 1 (cid:2) (cid:107) θ k − ¯ θ (cid:107) 2 ≤ R (cid:3) ≤ R .

Original: Therefore,
Revised: Therefore,

Original: P [Equation] (86) ≤ [Equation]
Revised: P (cid:2)(cid:0)(cid:13)(cid:13)

Original: −

Original: [Equation]

Original: [Equation]

Original: θ k −

Original: [Equation]
Revised: [Equation]

Original: where the last line follows by applying the reverse triangle inequality
Revised: where the last line follows by applying the reverse triangle inequality

Original: [Equation]
Revised: [Equation]

Original: By using (4) and Markov’sinequality,
Revised: By using (5) and Markov’s inequality,

Original: [Equation]
Revised: [Equation]

Original: By applying Assumption 
Revised: By applying Assumption 4, E (cid:104) (cid:107) ˙ f ( θ k , X k +1 ) (cid:107) 1+ α(cid:12)(cid:12)(cid:12) F k (cid:105) ≤ G ( θ k ) .

Revised: Moreover, on (cid:107) θ k − ¯ θ (cid:107) 2 ≤ R ,

Revised: G

Original: [Equation]
Revised: [Equation]

Original: E (cid:104)
Revised: θ k ) ≤ sup θ :

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12) F k (cid:105) ≤ G ( θ k ) . Moreover, on (cid:107)θ k − ¯ θ (cid:107) 2 ≤ R ,
Revised: (cid:107)

Original: G ( θ k ) ≤ sup θ : (cid:107) θ (cid:107) 2 ≤ R + (cid:107) ¯ θ (cid:107) 2 G( θ )
Revised: [Equation] ≤ [Equation] G

Original: =:
Revised: [Equation] = [Equation] G

Original: G R + (cid:107) ¯ θ (cid:107) 2 < ∞ since G is upper semi-continuous.
Revised: [Equation] ¯ [Equation] 2 < ∞ since G is upper semi-continuous.

Original: Combiningthese steps,
Revised: Combining these steps,

Original: [Equation]
Revised: [Equation]

Original: By Property 2, we see that the sum of the probabilities is ﬁnite.
Revised: By Property 2, we see that the sum of the probabilities is ﬁnite.

Original: Together with the Borel-Cantellilemma, P [ (cid:107) θ k +1 − ¯ θ (cid:107) 2 ≥ R + γ, (cid:107) θ k − ¯ θ (cid:107) 2 ≤ R, i.o. ] = 0 .
Revised: Together with the Borel-Cantelli lemma, ∀ R ≥ 0 , ∀ γ > 0 , and for all ¯ θ ∈ R p , [Equation]

Original: Since γ > 0 is arbitrary, we can show that this statement holds for a countable sequence of γ n ↓ 0 .
Revised: Since γ > 0 is arbitrary, we can show that this statement holds for a countable sequence of γ n ↓ 0 .

Original: Therefore, P [ (cid:107) θ k +1 − ¯ θ (cid:107) 2 >
Revised: Therefore, ∀ R ≥ 0 and all ¯ θ ∈ R p ,

Original: R, (cid:107) θ k − ¯ θ (cid:107) 2 ≤ R, i.o. ] = 0 .
Revised: [Equation]

Original: Since R is arbitrary as well, we conclude that for each element in a ﬁnite sequence R n ↑ ∞ ,640 P [ (cid:107) θ k +1 − ¯ θ (cid:107) 2 > R n ,  (cid:107) θ k − ¯ θ (cid:107) 2 ≤ R n, i.o. ] = 0  .
Revised: Since R is arbitrary, we conclude that for any ordering of positive rational numbers, { R n } , P [lim sup k (cid:107) θ k +1 − ¯ θ (cid:107) 2 > R n , lim inf k (cid:107) θ k − ¯ θ (cid:107) 2 ≤ R n ] = 0 for every n .

Original: Again, the countable union of measure zero setsis measure zero.
Revised: Again, the countable union of measure zero sets is measure zero.

Original: Hence, we conclude that P [lim sup k (cid:107) θ k − ¯ θ (cid:107) 2 > lim inf k (cid:107) θ k − ¯ θ (cid:107) 2 ] = 0 .
Revised: Hence, we conclude that P [lim sup k (cid:107) θ k − ¯ θ (cid:107) 2 > lim inf k (cid:107) θ k − ¯ θ (cid:107) 2 ] = 0 .

Original: Thus,either lim k (cid:107) θ k − ¯ θ (cid:107) 2 exists and is either inﬁnite or ﬁnite.
Revised: Thus, either lim k (cid:107) θ k − ¯ θ (cid:107) 2 exists and is either inﬁnite or ﬁnite.

Original: Moreover, on the event that the limit is ﬁnite, since ¯ θ is arbitrary, we can choose three distinct valuesof ¯ θ which are not colinear, and, by triangulation, the lim k θ k converges to a ﬁxed point (up to a setof measure zero).
Revised: Moreover, on the event that the limit is ﬁnite, since ¯ θ is arbitrary, we can choose p + 1 distinct values of ¯ θ which do not belong to a hyperplane of dimension smaller than p , and, by triangulation, the lim k θ k converges to a ﬁxed point (up to a set of measure zero).

Original: D Stability Analysis
Revised: D Stability Analysis

Original: We begin with a recursive relationship on the events { τ j > k } .
Revised: We begin with a recursive relationship on the events { τ j > k } .

Original: We use this result to prove that theobjective function converges to a ﬁnite limit on these events.
Revised: We use this result to prove that the objective function converges to a ﬁnite limit on these events.

Original: Then, we use this result to concludethat the gradient function converges to  zero on { ν j > k } .
Revised: Then, we use this result to conclude that the gradient function visits to a region of zero on the same event.

Original: Finally, we study the combination of theseevents to establish that the two statements above hold with probability one.
Revised: Finally, we study this event to establish that the two statements above hold with probability one.

Original: D.1 A Recursive Relationship
Revised: D.1 A Recursive Relationship

Original: Lemma 11 (Lemma 3) .
Revised: Lemma 11 (Lemma 3) .

Original: Let { M k } satisfy Property 1 .
Revised: Let { M k } satisfy Property 1 .

Original: Suppose Assumptions 1 to 4 hold.
Revised: Suppose Assumptions 1 to 4 hold.

Original: Let { θ k }satisfy (4) .
Revised: Let { θ k } satisfy (5) .

Original: Then, for any j + 1 ∈ N and k > j ,
Revised: Then, for any j + 1 ∈ N and k > j ,

Original: E [( F ( θ k +

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By the construction of τ j [Equation] when τ j > k 
Revised: By the construction of τ j , when τ j > k , then

Original: [Equation]
Revised: [Equation]

Original: then

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Using this relationship and using 1 [ τ j > k ] = 1 [ τ j > k − 1] − 1 [ τ j = k ] 
Revised: Using this relationship and using 1 [ τ j > k ] = 1 [ τ j > k − 1] − 1 [ τ j = k ] ,

Original: [Equation] [Equation]
Revised: E [ { F ( θ k +1 )

Original: ≤{+− F l.b. } 
+}
Revised: {+− F l.b. } 
+}

Original: [Equation] − E (cid:20)(cid:26)
Revised: [ τ j > k ] | F k ]

Original: [Equation]
Revised: [Equation]

Original: For the ﬁrst term on the right hand side, we can apply Assumptions 3 and 4, Property 1, and (4) tocalculate
Revised: For the ﬁrst term on the right hand side, we can apply Assumptions 3 and 4, Property 1, and (5) to calculate

Original: [Equation]
Revised: [Equation]

Original: For the second term on the right hand side of (95), we  ﬁrst note 1 [ τ j = k ] ≤ 1 [ τ j > k − 1] whichimplies 1 [ τ j = k ] = 1 [ τ j = k ] 1 [ τ j > k − 1] .
Revised: For the second term on the right hand side of (124), we require two facts. The ﬁrst fact is 1 [ τ j = k ] ≤ 1 [ τ j > k − 1] which implies 1 [ τ j = k ] = 1 [ τ j = k ] 1 [ τ j > k − 1] .

Original: Second, the Cauchy-Schwarz inequality and Lemma 7 imply
Revised: For the second fact, the CauchySchwarz inequality and Lemma 7 imply

Original: [Equation]
Revised: [Equation]

Original: Hence, using (4)
Revised: Hence, using (5),

Original: [Equation]
Revised: [Equation]

Revised: Putting together these two preceding facts together,

Original: [Equation]
Revised: [Equation]

Original: With the help of Theorem 5, we conclude [Equation]
Revised: where we bound P [ τ j = k |F k ] using Theorem 5.

Original: By putting the bounds on the ﬁrst  and second term together, the result follows.
Revised: By applying the bounds on the ﬁrst term, (125), and second term, (130), to (124), the result follows.

Original: By applying Assumption 5 to Lemma 3, we have the following simpliﬁed form.
Revised: By applying Assumption 5 to Lemma [Equation] we have the following simpliﬁed form.

Original: Lemma 12 (Lemma 4) .
Revised: Lemma 12 (Lemma 4) .

Original: If Assumptions 1 to 5 , and Properties 1 and 4 hold, and { θ k } satisfy (4) ,then there exists a K ∈ N such that for any j + 1 ∈ N and any k ≥ min { K, j + 1 } ,
Revised: If Assumptions 1 to 5 , and Properties 1 and 4 hold, and { θ k } satisfy (5) , then there exists a K ∈ N such that for any j + 1 ∈ N and any k ≥ min { K, j + 1 } ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The result follows by ﬁrst using Assumption 5 in Lemma 3.
Revised: The result follows by ﬁrst using Assumption 5 in Lemma 3.

Original: Then, collecting similar terms,we apply Lemma 6 to ﬁnd K .
Revised: Then, collecting similar terms, we apply Lemma 6 to ﬁnd K .

Original: D.2 Objective Function Analysis
Revised: D.2 Objective Function Analysis

Original: With this recursive formula, we now have the ﬁrst result.
Revised: With this recursive formula, we now have the ﬁrst result.

Original: Corollary 3.
Revised: Corollary 3.

Original: If Assumptions 1 to 5 and Properties 1 , 2 and 4 hold, and { θ k } satisfy (4) , thenlim k →∞ F ( θ k ) exists and is ﬁnite on ∪ ∞ j =0 { τ j = ∞} .
Revised: If Assumptions 1 to 5 and Properties 1 , 2 and 4 hold, and { θ k } satisfy (5) , then lim k →∞ F ( θ k ) exists and is ﬁnite on ∪ ∞ j =0 { τ j = ∞} .

Original: Proof.
Revised: Proof.

Original: By Lemma 4 and Robbins and Siegmund [1971], Neveu and Speed [1975, Exercise II.4], thelimit as k goes to inﬁnity of ( F ( θ k ) − F l.b. ) 1 [ τ j > k ] exists with probability one and is integrable.
Revised: By Lemma 4 and Robbins and Siegmund [1971], Neveu and Speed [1975, Exercise II.4], the limit as k goes to inﬁnity of ( F ( θ k ) − F l.b. ) 1 [ τ j > k − 1] exists with probability one and is integrable.

Original: Therefore, on the event { τ j = ∞} , the limit of F ( θ k ) − F l.b. exists and is integrable.
Revised: Therefore, on the event { τ j = ∞} , the limit of F ( θ k ) − F l.b. exists and is integrable.

Original: As a result, thelimit of F ( θ k ) − F l.b. exists and is ﬁnite on ∪ ∞ j =0 { τ j = ∞} .
Revised: As a result, the limit of F ( θ k ) − F l.b. exists and is ﬁnite on ∪ ∞ j =0 { τ j = ∞} .

Original: Additionally, we can state the following useful result.
Revised: Additionally, we can state the following useful result.

Original: Lemma 13.
Revised: Lemma 13.

Original: If Assumptions 1 to 5 , and Properties 1 , 2 and 4 hold, and { θ k } satisfy (4) , then ∃ K ∈ Nsuch that for any j > K , ∃ N j > 0 for which
Revised: If Assumptions 1 to 5 , and Properties 1 , 2 and 4 hold, and { θ k } satisfy (5) , then ∃ K ∈ N such that for any j > K , ∃ N j > 0 for which

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Revised: In Lemma 4, we upper bound the right hand side by removing the negative term, and, by

Revised: Property 2, we add C 1 (1 + α ) − 1 (cid:80) ∞ (cid:96) =

Original: {+k +1 λ max ( M (cid:96) ) 
+}
Revised: {+k +1 λ max ( M (cid:96) ) 
+}

Revised: + α to both side.

Original: By Lemma 4, Property 2, and any k ≥ j , 
Revised: Then, for all k ≥ j , E [ [Equation] Using 1 + C

Revised: [Equation]

Revised: [Equation]

Revised: 1 +

Revised: [Equation]

Revised: − 1 λ max ( M k ) 1+ α ) , it follows

Revised: E

Revised: [Equation]

Revised: ≤ exp

Original: [Equation]
Revised: [Equation]

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: where we have used 1 [ τ j > j − 1] = 1 .
Revised: where we have used 1 [ τ j > j − 1] = 1 .

Original: By Property 2, the summation in the exponent is ﬁnite,which implies the result.
Revised: By Property 2, the summation in the exponent is ﬁnite, which implies the result.

Original: D.3 Gradient Function Analysis
Revised: D.3 Gradient Function Analysis

Original: Lemma 14.
Revised: Lemma 14.

Original: If Assumptions 1 to 5 , and Properties 1 to 4 hold, and { θ k } satisfy (4) , then, for anyδ > 0 ,
Revised: If Assumptions 1 to 5 , and Properties 1 to 4 hold, and { θ k } satisfy (5) , then, for any δ > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By Lemma 4,
Revised: By Lemma 4,

Original: [Equation]
Revised: [Equation]

Original: By applying Lemma 13,
Revised: By applying Lemma 13,

Original: [Equation]
Revised: [Equation]

Original: By summing and using Assumption 1,
Revised: By summing and using Assumption 1,

Original: [Equation]
Revised: [Equation]

Original: By Property 2, the right hand side is bounded.
Revised: By Property 2, the right hand side is bounded.

Original: Now, by Property 3,
Revised: Now, by Property 3,

Original: [Equation]
Revised: [Equation]

Original: Using Markov’s inequality, for any (cid:96) ∈ N and any δ > 0 ,
Revised: Using Markov’s inequality

Original: [Equation]
Revised: [Equation]

Original: As the countable union of sets of measure zero have measure zero, the result follows.

Original: Corollary 4.

Original: If Assumptions 1 to 5 , and Properties 1 to 4 hold, and { θ k } satisfy (4) , then, on∪ ∞

Original: j =0 { ν j = ∞} , lim k →∞ (cid:107) ˙

Original: F ( θ k ) (cid:107) = 0 .

Original: Proof.

Original: Let δ > 0 and γ > 0 . Note, 1 [ ν j > k ] ≤ 1 [ ν j > k − 1] . Then, P
Revised: for any (cid:96) ∈ N and any δ > 0 ,

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: where we have made use of Assumption 4, Markov’s inequality, and Assumption 5 in the last two lines.

Original: Taking expectations, using sup k>j E [( F ( θ k ) − F l.b. ) 1 [ ν j > k ]] ≤ N j (Lemma 13), and applying

Original: Borel-Cantelli with Property 2, we conclude

Original: [Equation]
Revised: [Equation]

Original: Since γ is arbitrary, the result holds for a collection of γ n ↓ 0 .

Original: As the countable union of measurezero events has measure zero,
Revised: As the countable union of sets of measure zero have measure zero

Original: [Equation]
Revised: [Equation]

Original: θ k +1 ) (cid:13) (cid:13)(cid:13) 2 1 [ ν j > k ] > δ, (cid:13) (cid:13)(cid:13) ˙

Original: [-F ( θ k ) (cid:13) (cid:13)(cid:13) 2 
-]
Revised: [-F ( θ k ) (cid:13) (cid:13)(cid:13) 2 
-]

Original: [ ν j > k − 1] ≤ δ i.o. (cid:105) = 0 . Using this result in conjunction with Lemma 14, P (cid:104) lim
Revised: the result follows.

Original: k →∞ (cid:13)(cid:13)(cid:13) ˙

Original: F ( θ k +1 ) (cid:13)(cid:13)(cid:13) 2 1 [ ν j > k ] = 0 (cid:105) = 1 .

Original: Hence, on the event { ν j = ∞} , lim k →∞ (cid:107) ˙

Original: F ( θ k ) (cid:107) = 0 .

Original: Thus, on ∪ ∞ j =0 { ν j = ∞} , we conclude thatlim k →∞ (cid:107) ˙

Original: F ( θ k ) (cid:107) = 0 .

Original: D.4 Stopping Time Analysis
Revised: D.4 Stopping Time Analysis

Original: By Lemma 1, { τ j = k } ⊂ { ν j = k } . Therefore, { τ j ≤ k } ⊂ { ν j ≤ k } , and { τ j > k } ⊃ { ν j > k } . Using these relationships, we can compute the probabilities of { τ j = k }and { ν j = k } .
Revised: WE compute the probability of { τ j = k } .

Original: Theorem 5.
Revised: Theorem 5.

Original: Let { τ j : j + 1 ∈ N } be deﬁned as in(10) , and { ν j : j + 1 ∈ N } be deﬁned as in (11) .
Revised: Let { τ j : j + 1 ∈ N } be deﬁned as in (11) .

Original: If Assumptions 1 , 2 and 4 and Property 1 hold, and { θ k } satisfy (4) , then, for any j + 1 ∈ N and anyk + 1 ∈ N ,
Revised: If Assumptions 1 , 2 and 4 and Propertyhold, and { θ k } satisfy (5) , then, for any j + 1 ∈ N and any k + 1 ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: [Equation] Moreover, if Property 2 also holds, then P (cid:2) ∪ 
Revised: Moreover, if Property 2 also holds, then P (cid:2) ∪ ∞ j =0 { τ j = ∞} (cid:3) = 1 .

Original: Proof.
Revised: Proof.

Original: The case of k ≤ j is trivial.
Revised: The case of k ≤ j is trivial.

Original: So consider only k > j .
Revised: So consider only k > j .

Original: By the construction of L ( · , · ) and L (cid:15) ( · ) ,ω ∈ { L ( θ k , θ k +1 ) >
Revised: By the construction of L ( · , · ) and L (cid:15) ( · ) , ω ∈ { L ( θ k , θ k +1 ) >

Original: L (cid:15) ( θ k ) } implies
Revised: L (cid:15) ( θ k ) } implies

Original: ω ∈ {(cid:107) θ k +1 − θ k (cid:107) 2 >
Revised: ω ∈ {(cid:107) θ k +1 − θ k (cid:107) 2 >

Original: ( G ( θ k ) ∨ (cid:15) )
Revised: ( G ( θ k ) ∨ (cid:15) )

Original: 11+ α } .
Revised: 11+ α } .

Original: Using (4), Markov’sinequality, Property 1, we conclude
Revised: Using (5), Markov’s inequality, Property 1, we conclude

Original: [Equation]
Revised: [Equation]

Original:  Applying Assumption 4 supplies the bound on P [ τ j = k | F k ]and P [ ν j = k | F k ] .
Revised: [Equation] Applying Assumption 4 supplies the bound on P [ τ j = k | F k ] .

Original: For the secondpart, note
Revised: For the second part, note

Original: [Equation]
Revised: [Equation]

Original: Moreover, { ν j = ∞} ⊂ { ν j +1 = ∞} .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Since lim j P [ ν j = ∞ ] ≥ 1 − lim j (cid:80) ∞ k = j +1 λ max ( M k ) 

Revised: Since lim j P [ τ j = ∞ ] ≥ 1 − lim j (cid:80) ∞ k = j +1 λ max ( M k )

Original: + α , applying Property 2 supplies the ﬁnalresult.
Revised: + α , applying Property 2 supplies the ﬁnal result.