Original: G LOBAL C ONVERGENCE AND S TABILITY OF S TOCHASTIC G RADIENT D ESCENT
Revised: G LOBAL C ONVERGENCE AND S TABILITY OF S TOCHASTIC G RADIENT D ESCENT

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In machine learning, stochastic gradient descent (SGD) is widely deployed to train models using highly non-convex objectives with equally complex noise models.
Revised: In machine learning, stochastic gradient descent (SGD) is widely deployed to train models using highly non-convex objectives with equally complex noise models.

Original: Unfortunately, SGD theory often makes restrictive assumptions that fail to capture the non-convexity of real problems, and almost entirely ignore the complex noise models that exist in practice.
Revised: Unfortunately, SGD theory often makes restrictive assumptions that fail to capture the non-convexity of real problems, and almost entirely ignore the complex noise models that exist in practice.

Original: In this work, we make substantial progress on this shortcoming.
Revised: In this work, we make substantial progress on this shortcoming.

Original: First, we establish that SGD’s iterates will either globally converge to a stationary point or diverge under nearly arbitrary nonconvexity and noise models.
Revised: First, we establish that SGD’s iterates will either globally converge to a stationary point or diverge under nearly arbitrary nonconvexity and noise models.

Original: Under a slightly more restrictive assumption on the joint behavior of the non-convexity and noise model that generalizes current assumptions in the literature, we show that the objective function cannot diverge, even if the iterates diverge.
Revised: Under a slightly more restrictive assumption on the joint behavior of the non-convexity and noise model that generalizes current assumptions in the literature, we show that the objective function cannot diverge, even if the iterates diverge.

Original: As a consequence of our results, SGD can be applied to a greater range of stochastic optimization problems with conﬁdence about its global convergence behavior and stability.
Revised: As a consequence of our results, SGD can be applied to a greater range of stochastic optimization problems with conﬁdence about its global convergence behavior and stability.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Stochastic Gradient Descent (SGD) is a dominant algorithm for solving stochastic optimization problems that arise in machine learning, and has expanded its reach to more complex problems from estimating Gaussian Processes (Chen et al., 2020), covariance estimation in stochastic ﬁlters (Kim et al., 2021), and systems identiﬁcation (Hardt et al., 2016; Zhang & Patel, 2020).
Revised: Stochastic Gradient Descent (SGD) is a dominant algorithm for solving stochastic optimization problems that arise in machine learning, and has expanded its reach to more complex problems from estimating Gaussian Processes (Chen et al., 2020), covariance estimation in stochastic ﬁlters (Kim et al., 2021), and systems identiﬁcation (Hardt et al., 2016; Zhang & Patel, 2020).

Original: Accordingly, understanding SGD’s behavior has been crucial to its reliable application in machine learning and beyond.
Revised: Accordingly, understanding SGD’s behavior has been crucial to its reliable application in machine learning and beyond.

Original: As a result, SGD’s theory has greatly advanced from a variety of perspectives: global convergence analysis (Lei et al., 2019; Gower et al., 2020; Khaled & Richt ´ arik, 2020; Mertikopoulos et al., 2020; Patel, 2020), local convergence analysis (Mertikopoulos et al., 2020), greedy and global complexity analysis (Gower et al., 2020; Khaled & Richt ´ arik, 2020), asymptotic weak convergence (Wang et al., 2021), and saddle point analysis (
Revised: As a result, SGD’s theory has greatly advanced from a variety of perspectives: global convergence analysis (Lei et al., 2019; Gower et al., 2020; Khaled & Richt ´ arik, 2020; Mertikopoulos et al., 2020; Patel, 2020), local convergence analysis (Mertikopoulos et al., 2020), greedy and global complexity analysis (Gower et al., 2020; Khaled & Richt ´ arik, 2020), asymptotic weak convergence (Wang et al., 2021), and saddle point analysis (

Original: Fang
Revised: Fang

Original: et al., 2019; Mertikopoulos et al.,
Revised: et al., 2019; Mertikopoulos et al.,

Original: Jin et al., 2021).
Revised: Jin et al., 2021).

Original: While all of these perspectives add new dimensions to our understanding of SGD, the global convergence analysis of SGD is the foundation as it dictates whether local analyses, complexity analyses or saddle point analyses are even warranted.
Revised: While all of these perspectives add new dimensions to our understanding of SGD, the global convergence analysis of SGD is the foundation as it dictates whether local analyses, complexity analyses or saddle point analyses are even warranted.

Original: Unfortunately, current global convergence analyses of SGD make restrictive assumptions that fail to capture actual stochastic optimization problems arising in machine learning.
Revised: Unfortunately, current global convergence analyses of SGD make restrictive assumptions that fail to capture actual stochastic optimization problems arising in machine learning.

Original: In particular, global convergence analyses of SGD assume:
Revised: In particular, global convergence analyses of SGD assume:

Original: a global H¨older constant for the gradient of the objective function (e
Revised: a global H¨older constant for the gradient of the objective function (e

Original: g., Reddi et al., 2016a; Ma & Klabjan, 2017; Zhou et al., 2018; Bassily et al., 2018; Lei et al., 2019; Li & Orabona, 2019; Gower et al., 2020; Khaled & Richt´arik, 2020; Mertikopoulos et al., 2020; Patel, 2020; Jin et al., 2021); 2.
Revised: g., Reddi et al., 2016a; Ma & Klabjan, 2017; Zhou et al., 2018; Bassily et al., 2018; Lei et al., 2019; Li & Orabona, 2019; Gower et al., 2020; Khaled & Richt´arik, 2020; Mertikopoulos et al., 2020; Patel, 2020; Jin et al., 2021); 2.

Original: unrealistic noise models (e
Revised: unrealistic noise models (e

Original: g., uniformly bounded variance) for the stochastic gradients (e.g., Reddi et al., 2016b; Ma & Klabjan, 2017; Hu et al., 2019; Bi & Gunn, 2019; Zou et al., 2019; Mertikopoulos et al., 2020).
Revised: g., uniformly bounded variance) for the stochastic gradients (e.g., Reddi et al., 2016b; Ma & Klabjan, 2017; Hu et al., 2019; Bi & Gunn, 2019; Zou et al., 2019; Mertikopoulos et al., 2020).

Original: While such assumptions often make the analysis simpler, the resulting global convergence results would not even apply to simple neural network models for binary classiﬁcation (see Appendix A).
Revised: While such assumptions often make the analysis simpler, the resulting global convergence results would not even apply to simple neural network models for binary classiﬁcation (see Appendix A).

Original: To address this gap, we analyze the global convergence of SGD on nonconvex stochastic optimization problems that capture many actual machine learning applications.
Revised: To address this gap, we analyze the global convergence of SGD on nonconvex stochastic optimization problems that capture many actual machine learning applications.

Original: In particular, we assume local H¨older continuity of the gradient function (see Assumption 2), which substantially relaxes the global H¨older assumption of all previous works.
Revised: In particular, we assume local H¨older continuity of the gradient function (see Assumption 2), which substantially relaxes the global H¨older assumption of all previous works.

Original: Second, we assume that the noise model of the stochastic gradients is bounded by an arbitrary upper semi-continuous function of the parameter (see Assumption 4), and can even have inﬁnite variance (c.f., Wang et al., 2021), which generalizes the assumptions of all previous work.
Revised: Second, we assume that the noise model of the stochastic gradients is bounded by an arbitrary upper semi-continuous function of the parameter (see Assumption 4), and can even have inﬁnite variance (c.f., Wang et al., 2021), which generalizes the assumptions of all previous work.

Original: With these two general assumptions, we prove that the iterates of SGD will either converge to a stationary point or diverge to inﬁnity with probability one (see Theorem 2).
Revised: With these two general assumptions, we prove that the iterates of SGD will either converge to a stationary point or diverge to inﬁnity with probability one (see Theorem 2).

Original: Owing to our result, SGD can be applied to actual empirical risk minimization problems with guarantees about its asymptotic behavior.
Revised: Owing to our result, SGD can be applied to actual empirical risk minimization problems with guarantees about its asymptotic behavior.

Original: In the process of proving Theorem 2, we also prove another almost remarkable claim about the behavior of SGD’s iterates: that of all the possible asymptotic behaviors of the iterates (e.g., convergence to a ﬁxed point, convergence to a manifold, limit cycles, oscillation between points, divergence), even with rather arbitrary noise models, the only two possibilities are either the iterates converge to a ﬁxed point or they diverge (see Theorem 1).
Revised: In the process of proving Theorem 2, we also prove another almost remarkable claim about the behavior of SGD’s iterates: that of all the possible asymptotic behaviors of the iterates (e.g., convergence to a ﬁxed point, convergence to a manifold, limit cycles, oscillation between points, divergence), even with rather arbitrary noise models, the only two possibilities are either the iterates converge to a ﬁxed point or they diverge (see Theorem 1).

Original: Note, we cannot expect this outcome apriori even in such a simple context as applying SGD with ﬁxed step sizes to solve a consistent linear system: the iterates may terminate in a cycle and, thus, fail to converge to a ﬁxed point (Motzkin & Schoenberg, 1954, Theorem 2).
Revised: Note, we cannot expect this outcome apriori even in such a simple context as applying SGD with ﬁxed step sizes to solve a consistent linear system: the iterates may terminate in a cycle and, thus, fail to converge to a ﬁxed point (Motzkin & Schoenberg, 1954, Theorem 2).

Original: Thus, from a practical perspective, applying SGD to nonconvex problems with exotic noise models, which may initially cause concern, will either result in convergence to a stationary point or divergence of the iterates.
Revised: Thus, from a practical perspective, applying SGD to nonconvex problems with exotic noise models, which may initially cause concern, will either result in convergence to a stationary point or divergence of the iterates.

Original: While Theorem 2 is patently useful, SGD’s iterates diverging may cause some concern, especially when optimizing regularized empirical risk functions that are guaranteed to be coercive (i.e., the objective diverges to inﬁnity as the argument tends to inﬁnity).
Revised: While Theorem 2 is patently useful, SGD’s iterates diverging may cause some concern, especially when optimizing regularized empirical risk functions that are guaranteed to be coercive (i.e., the objective diverges to inﬁnity as the argument tends to inﬁnity).

Original: To address this issue, we generalize the notion of expected smoothness (see Khaled & Richt´arik, 2020) to an assumption about the joint behavior of the gradient function, the noise model, and the local H¨older constant (see Assumption 5) to prove that, regardless SGD’s iterates’ behavior, the objective function will remain ﬁnite and the gradient function will converge to zero with probability one (see Theorem 3).
Revised: To address this issue, we generalize the notion of expected smoothness (see Khaled & Richt´arik, 2020) to an assumption about the joint behavior of the gradient function, the noise model, and the local H¨older constant (see Assumption 5) to prove that, regardless SGD’s iterates’ behavior, the objective function will remain ﬁnite and the gradient function will converge to zero with probability one (see Theorem 3).

Original: Thus, from a practical perspective, if we can apply Theorem 3 to a coercive objective function, we are guaranteed that the iterates cannot diverge, and, consequently, must converge to a stationary point.
Revised: Thus, from a practical perspective, if we can apply Theorem 3 to a coercive objective function, we are guaranteed that the iterates cannot diverge, and, consequently, must converge to a stationary point.

Original: Furthering the practical value of our results, as alluded to previously, our results enable the use of downstream analyses.
Revised: Furthering the practical value of our results, as alluded to previously, our results enable the use of downstream analyses.

Original: Speciﬁcally, our results allow for SGD’s iterates to converge to stationary point or diverge; as a result, when SGD’s iterates converge to a stationary point, saddle point escape analyses (e.g., Fang et al., 2019; Mertikopoulos et al., 2020; Jin et al., 2021) can be applied to ensure that the stationary point is a local minimizer.
Revised: Speciﬁcally, our results allow for SGD’s iterates to converge to stationary point or diverge; as a result, when SGD’s iterates converge to a stationary point, saddle point escape analyses (e.g., Fang et al., 2019; Mertikopoulos et al., 2020; Jin et al., 2021) can be applied to ensure that the stationary point is a local minimizer.

Original: Moreover, when SGD is converging to a stationary point, local convergence rate analyses can also be supplied, which can inform adaptive step size rules and stopping criteria (Patel, 2020).
Revised: Moreover, when SGD is converging to a stationary point, local convergence rate analyses can also be supplied, which can inform adaptive step size rules and stopping criteria (Patel, 2020).

Original: Finally, from a theoretical perspective, we innovate two new analysis strategies to deal with the generality of thethe local H ¨ older continuity assumption on the gradient, and our general noise model assumption.
Revised: Finally, from a theoretical perspective, we innovate two new analysis strategies to deal with the generality of the local H ¨ older continuity assumption on the gradient, and our general noise model assumption.

Original: We term these two strategies the pseudo-global strategy and the local strategy.
Revised: We term these two strategies the pseudo-global strategy and the local strategy.

Original: We develop the pseudo-global strategy to prove global convergence (i.e., Theorem 2), while we develop the local strategy to prove stability (i.e., Theorem 3).
Revised: We develop the pseudo-global strategy to prove global convergence (i.e., Theorem 2), while we develop the local strategy to prove stability (i.e., Theorem 3).

Original: We believe that both of our strategies are of independent interest to theoreticians.
Revised: We believe that both of our strategies are of independent interest to theoreticians.

Original: Contribution Summary.
Revised: Contribution Summary.

Original: To summarize, we study the behavior of SGD under much more realistic assumptions than what is currently in the literature; namely, we consider local H¨older continuity and general noise models.
Revised: To summarize, we study the behavior of SGD under much more realistic assumptions than what is currently in the literature; namely, we consider local H¨older continuity and general noise models.

Original: In this context, we are able to show (1) that the iterates must either converge to a ﬁxed point or that they might diverge (Theorem 1); and (2), when the iterates converge to a ﬁxed point, it must be a stationary point of the objective function (Theorem 2).
Revised: In this context, we are able to show (1) that the iterates must either converge to a ﬁxed point or that they might diverge (Theorem 1); and (2), when the iterates converge to a ﬁxed point, it must be a stationary point of the objective function (Theorem 2).

Original: Moreover, under a slightly more restrictive assumption—which still generalizes current assumptions in the literature—, we show that, regardless of iterate behavior, the objective function will converge to a ﬁnite random variable (i.e., SGD is stable) and the gradient function will converge to zero (Theorem 3).
Revised: Moreover, under a slightly more restrictive assumption—which still generalizes current assumptions in the literature—, we show that, regardless of iterate behavior, the objective function will converge to a ﬁnite random variable (i.e., SGD is stable) and the gradient function will converge to zero (Theorem 3).

Original: Finally, we develop two new analysis strategies, the pseudo-global strategy and the local strategy, that are of independent interest to theoreticians in machine learning and stochastic optimization.
Revised: Finally, we develop two new analysis strategies, the pseudo-global strategy and the local strategy, that are of independent interest to theoreticians in machine learning and stochastic optimization.

Original: Organization.
Revised: Organization.

Original: The remainder of the paper is organized as follows.
Revised: The remainder of the paper is organized as follows.

Original: In Section 2, we specify the stochastic optimization problem that we will study, including a formal statement of all assumptions.
Revised: In Section 2, we specify the stochastic optimization problem that we will study, including a formal statement of all assumptions.

Original: In Section 3, we specify Stochastic Gradient Descent (SGD) and the properties of the needed properties of the learning rate.
Revised: In Section 3, we specify Stochastic Gradient Descent (SGD) and the properties of the needed properties of the learning rate.

Original: In Section 4, we prove and discuss our main results and highlight key steps, while leaving the rest to the appendix.
Revised: In Section 4, we prove and discuss our main results and highlight key steps, while leaving the rest to the appendix.

Original: In Section 5, we conclude this work.
Revised: In Section 5, we conclude this work.

Original:  S TOCHASTIC O PTIMIZATION
Revised:  S TOCHASTIC O PTIMIZATION

Original: We consider solving the optimization problem
Revised: We consider solving the optimization problem

Original: [Equation]
Revised: [Equation]

Original: where F maps R p into R ; f maps R p and the co-domain of the random variable X into R ; and E is the expectation operator.
Revised: where F maps R p into R ; f maps R p and the co-domain of the random variable X into R ; and E is the expectation operator.

Original: As we require gradients, we take F and f to differentiable in θ , and denote its derivatives with respect to θ by ˙
Revised: As we require gradients, we take F and f to differentiable in θ , and denote its derivatives with respect to θ by ˙

Original: F ( θ ) and ˙ f
Revised: F ( θ ) and ˙ f

Original: ( θ, X ) .
Revised: ( θ, X ) .

Original: With this notation, we make the following general assumptions about the deterministic portion of the objective function.
Revised: With this notation, we make the following general assumptions about the deterministic portion of the objective function.

Original: Assumption 1.
Revised: Assumption 1.

Original: There exists F l.b. ∈ R such that ∀ θ ∈ R p , F l.b. ≤ F ( θ ) .
Revised: There exists F l.b. ∈ R such that ∀ θ ∈ R p , F l.b. ≤ F ( θ ) .

Original: Assumption 2.
Revised: Assumption 2.

Original: There exists α ∈ (0 , 1] such that ˙ F ( θ ) is locally α -H¨older continuous.
Revised: There exists α ∈ (0 , 1] such that ˙ F ( θ ) is locally α -H¨older continuous.

Original: Assumptions 1 and 2 would even be considered mild in the context of non-convex deterministic optimization, in which it is also common to assume that the objective function well-behaved level sets (e.g., Nocedal & Wright, 2006, Theorems 3.2, 3.8, 4.5, 4.6).
Revised: Assumptions 1 and 2 would even be considered mild in the context of non-convex deterministic optimization, in which it is also common to assume that the objective function well-behaved level sets (e.g., Nocedal & Wright, 2006, Theorems 3.2, 3.8, 4.5, 4.6).

Original: Importantly, Assumption 2 relaxes the common restrictive assumption of globally H ¨ older continuous gradient functions that is common in other analyses.
Revised: Importantly, Assumption 2 relaxes the common restrictive assumption of globally H ¨ older continuous gradient functions that is common in other analyses.

Original: Our ﬁnal step is to make some assumptions about the stochastic portion of the objective function.
Revised: Our ﬁnal step is to make some assumptions about the stochastic portion of the objective function.

Original: The ﬁrst assumption requires the stochastic gradients to be unbiased, which can readily be relaxed (Bottou et al., 2018).
Revised: The ﬁrst assumption requires the stochastic gradients to be unbiased, which can readily be relaxed (Bottou et al., 2018).

Original: The second assumption allows for a generic noise model for an α -H ¨ older continuous gradient function, and even allows for the second moment to not exist when α < 1 (c.f.
Revised: The second assumption allows for a generic noise model for an α -H ¨ older continuous gradient function, and even allows for the second moment to not exist when α < 1 (c.f.

Original: Wang et al., 2021).
Revised: Wang et al., 2021).

Original: Assumption 3.
Revised: Assumption 3.

Original: For all θ ∈ R p , ˙ F ( θ ) =
Revised: For all θ ∈ R p , ˙ F ( θ ) =

Original: E [ ˙ f
Revised: E [ ˙ f

Original: ( θ, X )] .
Revised: ( θ, X )] .

Original: Assumption 4.
Revised: Assumption 4.

Original: Let α ∈ (0 , 1] be as in Assumption 2 .
Revised: Let α ∈ (0 , 1] be as in Assumption 2 .

Original: There exists an upper semi-continuous function G ( θ ) such that E [ (cid:107) ˙
Revised: There exists an upper semi-continuous function G ( θ ) such that E [ (cid:107) ˙

Original: f ( θ, X ) (cid:107) 

Revised: f ( θ, X ) (cid:107)

Original: + α] ≤ G ( θ ) .
Revised: + α] ≤ G ( θ ) .

Original: We will show that Assumptions 1 to 4 are sufﬁcient for a global convergence result. 
Revised: We will show that Assumptions 1 to 4 are sufﬁcient for a global convergence result. Remark 1.

Revised: It is entirely possible that E [ (cid:107) ˙

Revised: f

Original: In order{+( θ, X ) (cid:107) 
 + α] is (at least) upper semi-continuous, and+} to show that the objective function cannot diverge (i.e., to prove stability), we will need an additional assumption.
Revised: {+( θ, X ) (cid:107) 
 + α] is (at least) upper semi-continuous, and+} to set G ( θ ) equal to this function. In the case that E [ (cid:107) ˙ f

Original: {+( θ, X ) (cid:107) 
+}
Revised: {+( θ, X ) (cid:107) 
+}

Revised: + α] is not upper semi-continuous, it is possible to specify G ( θ ) as the upper envelope of E [ (cid:107) ˙

Original: {+f ( θ, X ) (cid:107) 
+}
Revised: {+f ( θ, X ) (cid:107) 
+}

Revised: + α] (i.e., the its limit supremum function).

Revised: However, it is unlikely that E [ (cid:107) ˙

Original: {+f ( θ, X ) (cid:107) 
+}
Revised: {+f ( θ, X ) (cid:107) 
+}

Revised: + α] nor its upper envelope are easy to specify explicitly, and it is more likely to be able to ﬁnd an upper bound.

Revised: In order to show that the objective function cannot diverge (i.e., to prove stability), we will need an additional assumption.

Original: This assumption will relate the gradient function, noise model and variation on the local H¨older constant.
Revised: This assumption will relate the gradient function, noise model and variation on the local H¨older constant.

Original: To begin, we deﬁne the variation on the local H¨older constant.
Revised: To begin, we deﬁne the variation on the local H¨older constant.

Original: Let α ∈ (0 , 1] be as in Assumption 2 and (cid:15) > 0 be arbitrary, and deﬁne
Revised: Let α ∈ (0 , 1] be as in Assumption 2 and (cid:15) > 0 be arbitrary, and deﬁne

Original: [Equation]
Revised: [Equation]

Original: if this quantity is nonzero
Revised: if this quantity is nonzero

Original: [Equation]
Revised: [Equation]

Original: otherwise , [Equation] where ∨ indicates the maximum between two quantities.
Revised: otherwise , (2) where ∨ indicates the maximum between two quantities.

Original: Note, the choice of (cid:15) is irrelevant, and they can be distinct for the two cases in the deﬁnition of L (cid:15) , but we ﬁx them to be the same for simplicity.
Revised: Note, the choice of (cid:15) is irrelevant, and they can be distinct for the two cases in the deﬁnition of L (cid:15) , but we ﬁx them to be the same for simplicity.

Original: Importantly, the quantity, L (cid:15) , is deﬁned at every parameter θ under Assumption 2.
Revised: Importantly, the quantity, L (cid:15) , is deﬁned at every parameter θ under Assumption 2.

Original: With this quantity, we can state the assumption  needed to prove stability, which we see is a generalization of expected smoothness (see Khaled &
Revised: With this quantity, we can state a nonintuitive,technical assumption that is needed to prove stability.

Original: Assumption 5.
Revised: Assumption 5.

Original: There exists C 1 , C 2 , C 3 ≥ 0 such that, ∀ θ ∈ R p ,
Revised: There exists C 1 , C 2 , C 3 ≥ 0 such that, ∀ θ ∈ R p ,

Original: [Equation]
Revised: [Equation]

Original: At ﬁrst glance, Assumption 5 seems rather contrived, but, upon further investigation, we have had difﬁculty in ﬁnding practical problems where Assumption 5 does not hold.
Revised: Assumption 5 generalizes Assumption 4.3(c) of Bottou et al. (2018), which is satisﬁed for a large swath of statistical models. Moreover, Assumption 5 generalizes the notion of expected smoothness (see Khaled & Richt´arik, 2020, for a history of the assumption), which expanded the optimization problems covered by the theory of Bottou et al. Note, Assumption 5 is about the asymptotic properties of the stochastic optimization problem as the left hand side of the inequality in Assumption 5 can be bounded inside of any compact set.

Original: For instance, the following functions will satisfy the deterministic portion of Assumption 5 for ϑ ∈ R : exp( ϑ 2 ) , exp( ϑ ) , ϑ r  for r ∈ R , log( | ϑ | + 1) , and log(log( | ϑ | + 1) + 1) .
Revised: Thus, Assumption 5 covers a variety of asymptotic behaviors, such as exp( (cid:107) θ (cid:107) 22 ) , exp( (cid:107) θ (cid:107) 2 ) , (cid:107) θ (cid:107) r 2 for r ∈ R , log( (cid:107) θ (cid:107) 2 +1) , and log(log( (cid:107) θ (cid:107) 2 +1)+1) .

Original: Therefore, Assumption 5 holds for functions  a variety of different asymptotic behaviors.
Revised: Therefore, Assumption 5 holds for functions with a variety of different asymptotic behaviors.

Original: We will show that Assumptions 1 to 5 are sufﬁcient for a stability result.
Revised: We will show that Assumptions 1 to 5 are sufﬁcient for a stability result.

Original: Now that we have speciﬁed the nature of the stochastic optimization problem, we turn our attention to the algorithm used to solve the problem, namely, stochastic gradient descent (SGD).
Revised: Now that we have speciﬁed the nature of the stochastic optimization problem, we turn our attention to the algorithm used to solve the problem, namely, stochastic gradient descent (SGD).

Original:  S TOCHASTIC G RADIENT D ESCENT
Revised:  S TOCHASTIC G RADIENT D ESCENT

Original: SGD starts with an arbitrary initial value, θ 0 ∈ R p , and generates a sequence of iterates { θ k : k ∈ N } according to the rule
Revised: SGD starts with an arbitrary initial value, θ 0 ∈ R p , and generates a sequence of iterates { θ k : k ∈ N } according to the rule

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where { M k : k + 1 ∈ N } ⊂ R p × p ; and { X k : k ∈ N } are independent and identically distributed copies of X .
Revised: where { M k : k + 1 ∈ N } ⊂ R p × p ; and { X k : k ∈ N } are independent and identically distributed copies of X .

Original: Importantly, { M k } cannot be arbitrary, and the following properties specify a generalization of the Robbins & Monro (1951) conditions for matrix-valued learning rates (c.f. Patel, 2020).
Revised: Importantly, { M k } cannot be arbitrary, and the following properties specify a generalization of the Robbins & Monro (1951) conditions for matrix-valued learning rates (c.f. Patel, 2020).

Original: The ﬁrst condition requires a positive learning rate, and imposes symmetry to ensure the existence of real eigenvalues.
Revised: The ﬁrst condition requires a positive learning rate, and imposes symmetry to ensure the existence of real eigenvalues.

Original: Property 1.
Revised: Property 1.

Original: { M k : k + 1 ∈ N } are symmetric, positive deﬁnite matrices.
Revised: { M k : k + 1 ∈ N } are symmetric, positive deﬁnite matrices.

Original: The next two properties are a natural generalization of the Robbins-Monro conditions.
Revised: The next two properties are a natural generalization of the Robbins-Monro conditions.

Original: Let α ∈ (0 , 1] be as in Assumption 2.
Revised: Let α ∈ (0 , 1] be as in Assumption 2.

Original: Property 2.
Revised: Property 2.

Original: Let λ max ( · ) denote the largest eigenvalue of a symmetric, positive deﬁnite matrix.
Revised: Let λ max ( · ) denote the largest eigenvalue of a symmetric, positive deﬁnite matrix.

Original: Then, (cid:80) ∞
Revised: Then, (cid:80) ∞

Original: k =

Revised: k =

Original: λ max ( M k ) 

Revised: λ max ( M k )

Original: + α =:
Revised: + α =:

Original: S < ∞ .
Revised: S < ∞ .

Original: Property 3.
Revised: Property 3.

Original: Let λ min ( · ) denote the smallest eigenvalue of a symmetric, positive deﬁnite matrix.
Revised: Let λ min ( · ) denote the smallest eigenvalue of a symmetric, positive deﬁnite matrix.

Original: Then, (cid:80) ∞
Revised: Then, (cid:80) ∞

Original: k =

Revised: k =

Original: λ min ( M k ) = ∞ .
Revised: λ min ( M k ) = ∞ .

Original: We will show that Properties 1 to 3 are sufﬁcient for a global convergence result.
Revised: We will show that Properties 1 to 3 are sufﬁcient for a global convergence result.

Original: The ﬁnal property ensures the stability of the condition number of { M k } .
Revised: The ﬁnal property ensures the stability of the condition number of { M k } .

Original: Note, this property is readily satisﬁedfor scalar learning rates satisfying the Robbins-Monro conditions.
Revised: Note, this property is readily satisﬁed

Original: Property 4.
Revised: Property 4.

Original: Let κ ( · ) denote the ratio of the largest and smallest eigenvalues of a symmetric, positive deﬁnite matrix.
Revised: Let κ ( · ) denote the ratio of the largest and smallest eigenvalues of a symmetric, positive deﬁnite matrix.

Original: Then, lim k →∞ λ max ( M k )
Revised: Then, lim k →∞ λ max ( M k )

Original: α κ ( M k )
Revised: α κ ( M k )

Original: = 0 .
Revised: = 0 .

Original: We will show that Properties 1 to 4 are sufﬁcient for stability.
Revised: We will show that Properties 1 to 4 are sufﬁcient for stability.

Original:  G LOBAL C ONVERGENCE & S TABILITY
Revised:  G LOBAL C ONVERGENCE & S TABILITY

Original: With the stochastic optimization problem and with stochastic gradient descent (SGD) speciﬁed, we now turn our attention to what happens when SGD is applied to a stochastic optimization problem.
Revised: With the stochastic optimization problem and with stochastic gradient descent (SGD) speciﬁed, we now turn our attention to what happens when SGD is applied to a stochastic optimization problem.

Original: The key step in the analysis of SGD on any objective function is to establish a bound between the optimality gap at θ k +1 with that of θ k .
Revised: The key step in the analysis of SGD on any objective function is to establish a bound between the optimality gap at θ k +1 with that of θ k .

Original: This step is achieved by using the local H¨older continuity of the gradient function and the fundamental theorem of calculus.
Revised: This step is achieved by using the local H¨older continuity of the gradient function and the fundamental theorem of calculus.

Original: Using Assumption 2, we ﬁrst specify the local H¨older constant.
Revised: Using Assumption 2, we ﬁrst specify the local H¨older constant.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: For any θ, ϕ ∈ R p , deﬁne
Revised: For any θ, ϕ ∈ R p , deﬁne

Original: [Equation]
Revised: [Equation]

Original: Moreover, for any R ≥ 0 , let L R
Revised: Moreover, for any R ≥ 0 , let L R

Original: =
Revised: =

Original: L (0 , ϕ ) for (cid:107) ϕ (cid:107) 2 = R .
Revised: L (0 , ϕ ) for (cid:107) ϕ (cid:107) 2 = R .

Original: Remark 1.
Revised: Remark 2.

Original: Note, when the gradient is locally H¨older continuous, L R is well deﬁned for any R ≥ 0 .
Revised: Note, when the gradient is locally H¨older continuous, L R is well deﬁned for any R ≥ 0 .

Original: With this deﬁnition, we can now relate the optimality gap of θ k +1 with that of θ k by using the following result.
Revised: With this deﬁnition, we can now relate the optimality gap of θ k +1 with that of θ k by using the following result.

Original: Lemma 1.
Revised: Lemma 1.

Original: Suppose Assumptions 1 and 2 hold.
Revised: Suppose Assumptions 1 and 2 hold.

Original: Then, for any θ, ϕ ∈ R p ,
Revised: Then, for any θ, ϕ ∈ R p ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By Taylor’s theorem,
Revised: By Taylor’s theorem,

Original: [Equation]
Revised: [Equation]

Original: Now, add and subtract ˙
Revised: Now, add and subtract ˙

Original: F ( θ ) to ˙ F ( θ + t ( ϕ − θ )) in the integral, then apply Assumption 2.
Revised: F ( θ ) to ˙ F ( θ + t ( ϕ − θ )) in the integral, then apply Assumption 2.

Original: We conclude,
Revised: We conclude,

Original: [Equation]
Revised: [Equation]

Original: By computing the integral, the result follows.
Revised: By computing the integral, the result follows.

Original: Now, if we simply set ϕ = θ k +

Revised: Now, if we simply set ϕ = θ k +

Original: and θ = θ k in Lemma 1 and try to take expectations to manage the randomness of the stochastic gradient, we will run into the problem that L ( θ k , θ k +1 ) and (cid:107) θ k +1 − θ k (cid:107) 2 are dependent, and we cannot compute its expectation.
Revised: and θ = θ k in Lemma 1 and try to take expectations to manage the randomness of the stochastic gradient, we will run into the problem that L ( θ k , θ k +1 ) and (cid:107) θ k +1 − θ k (cid:107) 2 are dependent, and we cannot compute its expectation.

Original: In previous work, this issue is dealt with simply by using a global H¨older constant to upper bound L ( θ k , θ k +1 ) , which is unrealistic even for simple problems (see Appendix A).
Revised: In previous work, this technical challenge is waived away by using a global H¨older constant to upper bound L ( θ k , θ k +1 ) , which is unrealistic even for simple problems (see Appendix A).

Original: To address this issue, we innovate two new strategies for handling the dependence between L ( θ k , θ k +1 ) and (cid:107) θ k +1 − θ k (cid:107) 2 .
Revised: To address this technical challenge, we innovate two new strategies for handling the dependence between L ( θ k , θ k +1 ) and (cid:107) θ k +1 − θ k (cid:107) 2 .

Original: In both strategies, we follow the same general approach:
Revised: In both strategies, we follow the same general approach:

Original: We begin by restricting our analysis to speciﬁc events, which will allow us to decouple L ( θ k , θ k +1 ) and (cid:107) θ k +1 − θ k (cid:107) 2 . 2.
Revised: We begin by restricting our analysis to speciﬁc events, which will allow us to decouple L ( θ k , θ k +1 ) and (cid:107) θ k +1 − θ k (cid:107) 2 . 2.

Original: With these two quantities decoupled, we will develop a recurrence relationship between the optimality gap at θ k +1 and that of θ k .3.
Revised: With these two quantities decoupled, we will develop a recurrence relationship between the optimality gap at θ k +1 and that of θ k .

Original: We apply this recurrence relationship with reﬁnements of standard arguments or new ones to derive the desired property about the objective function.
Revised: We apply this recurrence relationship with reﬁnements of standard arguments or new ones to derive the desired property about the objective function.

Original: Finally, we state the generality of the speciﬁc events on which we have studied SGD’s iterates.
Revised: Finally, we state the generality of the speciﬁc events on which we have studied SGD’s iterates.

Original: Thus, it follows, we will deﬁne two distinct series of events for the two strategies.
Revised: Thus, it follows, we will deﬁne two distinct series of events for the two strategies.

Original: The ﬁrst strategy, which we refer to as the pseudo-global strategy, will provide the global convergence analysis.
Revised: The ﬁrst strategy, which we refer to as the pseudo-global strategy, will provide the global convergence analysis.

Original: The second strategy, which we refer to as the local strategy, will provide the stability result.
Revised: The second strategy, which we refer to as the local strategy, will provide the stability result.

Original: 4. P SEUDO -G LOBAL S TRATEGY AND G LOBAL C ONVERGENCE A NALYSIS
Revised: 4. P SEUDO -G LOBAL S TRATEGY AND G LOBAL C ONVERGENCE A NALYSIS

Original: For the ﬁrst strategy, which supplies the global convergence result, we study SGD on the events
Revised: For the ﬁrst strategy, which supplies the global convergence result, we study SGD on the events

Original: [Equation]
Revised: [Equation]

Original: for every R ≥ 0 .
Revised: for every R ≥ 0 .

Original: We now try to control the optimality gap at iteration k + 1 with that of iteration k , which will result in two cases.
Revised: We now try to control the optimality gap at iteration k + 1 with that of iteration k , which will result in two cases.

Original: (Case 1)
Revised: (Case 1)

Original: B k +

Revised: B k +

Original: ( R ) holds we can bound L ( θ k , θ k +1 ) by L R , and G ( θ ) is also bounded in the ball of radius R about the origin (which follows from G being upper semi-continuous in Assumption 4).
Revised: ( R ) holds we can bound L ( θ k , θ k +1 ) by L R , and G ( θ ) is also bounded in the ball of radius R about the origin (which follows from G being upper semi-continuous in Assumption 4).

Original: As a result, we could then proceed with the analysis in a manner that is similar to having a global H¨older constant.2.
Revised: As a result, we could then proceed with the analysis in a manner that is similar to having a global H ¨ older constant.

Original: (Case 2) (cid:107) θ k +1 (cid:107) 2 > R and B k ( R ) holds.
Revised: (Case 2) (cid:107) θ k +1 (cid:107) 2 > R and B k ( R ) holds.

Original: In this case, controlling L ( θ k , θ k +1 ) is very challenging and, to our knowledge, was not solved before our work.
Revised: In this case, controlling L ( θ k , θ k +1 ) is very challenging and, to our knowledge, was not solved before our work.

Original: Our  for controlling the optimality gap in both cases is supplied in the next lemma, whose proof is in Appendix C.
Revised: Our approach for controlling the optimality gap in both cases is supplied in the next lemma, whose proof is in Appendix C.

Original: Lemma 2.
Revised: Lemma 2.

Original: Let { M k } satisfy Property 1 .
Revised: Let { M k } satisfy Property 1 .

Original: Suppose Assumptions 1 to 4 hold.
Revised: Suppose Assumptions 1 to 4 hold.

Original: Let { θ k } satisfy (4) .
Revised: Let { θ k } satisfy (4) .

Original: Then, ∀ R ≥ 0 ,
Revised: Then, ∀ R ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: 10) where G R = sup θ ∈ B ( R ) G ( θ ) < ∞ with G ( θ ) ; and ∂F R = sup θ ∈ B
Revised: where G R = sup θ ∈ B ( R ) G ( θ ) < ∞ with G ( θ ) ; and ∂F R = sup θ ∈ B

Original: ( R ) (cid:107) ˙
Revised: ( R ) (cid:107) ˙

Original: F ( θ ) (cid:107) 2 (1 + α ) < ∞ .
Revised: F ( θ ) (cid:107) 2 (1 + α ) < ∞ .

Original: With this recursion and standard martingale results (Robbins & Siegmund, 1971; Neveu & Speed, 1975, Exercise II.4), the limit of [ F ( θ k ) − F l.b. ] 1 [ B k ( R )] exists with probability one and is ﬁnite for every R ≥ 0 .
Revised: With this recursion and standard martingale results (Robbins & Siegmund, 1971; Neveu & Speed, 1975, Exercise II.4), the limit of [ F ( θ k ) − F l.b. ] 1 [ B k ( R )] exists with probability one and is ﬁnite for every R ≥ 0 .

Original: As a result, the limit of F ( θ k ) − F l.b. exists and is ﬁnite on the event { sup k (cid:107) θ k (cid:107) 2 < ∞} (see Corollary 1).
Revised: As a result, the limit of F ( θ k ) − F l.b. exists and is ﬁnite on the event { sup k (cid:107) θ k (cid:107) 2 < ∞} (see Corollary 1).

Original: We can also use Lemma 2 to make a statement about the gradient.
Revised: We can also use Lemma 2 to make a statement about the gradient.

Original: Speciﬁcally, we can show that the limit inﬁmum of E [ (cid:107) ˙
Revised: Speciﬁcally, we can show that the limit inﬁmum of E [ (cid:107) ˙

Original: F ( θ k ) (cid:107) 22 

Revised: F ( θ k ) (cid:107) 22

Original: [ B k ( R )]] must be zero, which is now a standard argument that mimics
Revised: [ B k ( R )]] must be zero, which is now a standard argument that mimics

Original: Zoutendijk’s theorem (Nocedal & Wright, 2006, Theorem 3.2).
Revised: Zoutendijk’s theorem (Nocedal & Wright, 2006, Theorem 3.2).

Original: By Markov’s inequality, this result implies that (cid:107) ˙ F ( θ k ) (cid:107) 2 

Revised: By Markov’s inequality, this result implies that (cid:107) ˙ F ( θ k ) (cid:107) 2

Original: [ B k ( R )] be get arbitrarily close to 0 inﬁnitely often (see Lemma 8).
Revised: [ B k ( R )] gets arbitrarily close to 0 inﬁnitely often (see Lemma 8).

Original: To show convergence to zero, however, is not standard.
Revised: To show convergence to zero, however, is not standard.

Original: Several strategies have been developed, namely those of Li & Orabona (2019); Lei et al.
Revised: Several strategies have been developed, namely those of Li & Orabona (2019); Lei et al.

Original: ; Mertikopoulos et al.
Revised: ; Mertikopoulos et al.

Original: Patel (2020).
Revised: Patel (2020).

Original: Unfortunately, the approaches of Li & Orabona (2019); Lei et al.
Revised: Unfortunately, the approaches of Li & Orabona (2019); Lei et al.

Original: (2019) rely intimately on the existence of a global H ¨ older constant, while that of Mertikopoulos et al.
Revised: (2019) rely intimately on the existence of a global H¨older constant, while that of Mertikopoulos et al.

Original: (2020) requires even more restrictive assumptions.
Revised: (2020) requires even more restrictive assumptions.

Original: Fortunately, the approach of Patel (2020) can be improved and generalized to the current context (see Lemma 9).
Revised: Fortunately, the approach of Patel (2020) can be improved and generalized to the current context (see Lemma 9).

Original: Thus, we show that lim k →∞ (cid:107) ˙
Revised: Thus, we show that lim k →∞ (cid:107) ˙

Original: F ( θ k ) (cid:107) 2 = 0 on { sup k (cid:107) θ k (cid:107) 2 < ∞} (see Corollary 
Revised: F ( θ k ) (cid:107) 2 = 0 on { sup k (cid:107) θ k (cid:107) 2 < ∞} (see Corollary 2).

Original: Our ﬁnal step is to clarify the role of { sup k (cid:107) θ k (cid:107) 2 < ∞} in the asymptotic behavior of SGD’s iterates.
Revised: Our ﬁnal step is to clarify the role of { sup k (cid:107) θ k (cid:107) 2 < ∞} in the asymptotic behavior of SGD’s iterates.

Original: At ﬁrst glance, this event seems to imply that the iterates converge to a point.
Revised: At ﬁrst glance, this event seems to imply that the iterates converge to a point.

Original: However, owing to the general nature of the noise, it is also possible, say, that the iterates approach a limit cycle or oscillate between points with the same norm.
Revised: However, owing to the general nature of the noise, it is also possible, say, that the iterates approach a limit cycle or oscillate between points with the same norm.

Original: Even beyond this event, the generality of the noise model may allow for substantial excursions between F l.b. and inﬁnity (c.f., a simple random walk, which has a limit supremum of inﬁnity and a limit inﬁnimum of negative inﬁnity).
Revised: Even beyond this event, the generality of the noise model may allow for substantial excursions between F l.b. and inﬁnity (c.f., a simple random walk, which has a limit supremum of inﬁnity and a limit inﬁnimum of negative inﬁnity).

Original: Thankfully, we can prove that either the iterates converge to a point or they must diverge—a result that we refer to as the Capture Theorem (see Appendix C).
Revised: Thankfully, we can prove that either the iterates converge to a point or they must diverge—a result that we refer to as the Capture Theorem (see Appendix C).

Original: Theorem 1 (Capture Theorem) .
Revised: Theorem 1 (Capture Theorem) .

Original: Let { θ k } be deﬁned as in (4) , and let { M k } satisfy Propertiesand 2 .
Revised: Let { θ k } be deﬁned as in (4) , and let { M k } satisfy Propertiesand 2 .

Original: If Assumption 4 holds, then either { lim k →∞ θ k exists } or { lim inf k →∞ (cid:107) θ k (cid:107) 2 = ∞} must occur.
Revised: If Assumption 4 holds, then either { lim k →∞ θ k exists } or { lim inf k →∞ (cid:107) θ k (cid:107) 2 = ∞} must occur.

Original: By putting together the above arguments and results, we can conclude that either SGD’s iterates diverge or SGD’s iterates converge to a stationary point.
Revised: By putting together the above arguments and results, we can conclude that either SGD’s iterates diverge or SGD’s iterates converge to a stationary point.

Original: This is formally stated in the following theorem.
Revised: This is formally stated in the following theorem.

Original: See Section 1 for a discussion of the practical value of this result.
Revised: See Section 1 for a discussion of the practical value of this result.

Original: Theorem 2 (Global Convergence) .
Revised: Theorem 2 (Global Convergence) .

Original: Let θ
Revised: Let θ

Original: be arbitrary, and let { θ k : k ∈ N } be deﬁned according to (4) with { M k : k + 1 } satisfying Properties 1 to 3 .
Revised: be arbitrary, and let { θ k : k ∈ N } be deﬁned according to (4) with { M k : k + 1 } satisfying Properties 1 to 3 .

Original: Suppose Assumptions 1 to 4 hold.
Revised: Suppose Assumptions 1 to 4 hold.

Original: Let
Revised: Let

Original: A 1 = { lim inf k →∞ (cid:107) θ k (cid:107) 2 = ∞} and A 2 = { lim k →∞ θ k exists } .
Revised: A [Equation] = ∞} and A 2 = { lim k →∞ θ k exists } .

Original: Then, the following statements hold.
Revised: Then, the following statements hold.

Original: P [ A 1 ] + P [ A 2 ] = 1 . 2.
Revised: P [ A 1 ] + P [ A 2 ] = 1 . 2.

Original: On A 2 , there exists a ﬁnite random variable, F lim , such that lim k →∞ F ( θ k ) =
Revised: On A 2 , there exists a ﬁnite random variable, F lim , such that lim k →∞ F ( θ k ) =

Original: F lim and lim k →∞ ˙ F ( θ k )
Revised: F lim and lim k →∞ ˙ F ( θ k )

Original: = 

Revised: =

Original: with probability one.
Revised: with probability one.

Original: Proof.
Revised: Proof.

Original: By Theorem 1, we have that P [ A 1 ] + P [ A 2 ] = 1 .
Revised: By Theorem 1, we have that P [ A 1 ] + P [ A 2 ] = 1 .

Original: Then, on A 2 , Corollaries 1 and 2 supply that F ( θ k ) → F lim , which is ﬁnite, and ˙ F ( θ k ) → 0 .
Revised: Then, on A 2 , Corollaries 1 and 2 imply that F ( θ k ) → F lim , which is ﬁnite, and ˙ F ( θ k ) → 0 .

Original: 4. L OCAL S TRATEGY AND S TABILITY A NALYSIS
Revised: 4. L OCAL S TRATEGY AND S TABILITY A NALYSIS

Original: While Theorem 2 supplies a complete global convergence result, it allows for the possibility of diverging iterates.
Revised: While Theorem 2 provides a complete global convergence result, it allows for the possibility of diverging iterates.

Original: The possibility of divergent iterates raises the spectre of whether the objective function can also diverge along this sequence.
Revised: The possibility of divergent iterates raises the spectre of whether the objective function can also diverge along this sequence.

Original: That is, there is a possibility that SGD may be unstable, which would be highly unexpected and undesirable, especially when the objective function is coercive (e.g., has an (cid:96) 1 penalty on the parameters).
Revised: That is, there is a possibility that SGD may be unstable, which would be highly unexpected and undesirable, especially when the objective function is coercive (e.g., has an (cid:96) 1 penalty on the parameters).

Original: To formalize this concept, we deﬁne a relevant notion of stability.
Revised: To formalize this concept, we deﬁne a relevant notion of stability.

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: Stochastic Gradient Descent is stable if
Revised: Stochastic Gradient Descent is stable if

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where { θ k } satisfy (4) .
Revised: where { θ k } satisfy (4) .

Original: We now state the events that we will use to decouple the relationship between L ( θ k , θ k +1 ) and (cid:107) θ k +1 − θ k (cid:107) 2 .
Revised: We now state the events that we will use to decouple the relationship between L ( θ k , θ k +1 ) and (cid:107) θ k +1 − θ k (cid:107) 2 .

Original: Unlike in our pseudo-global strategy, we will make use of two sequences of events that are closely related.
Revised: Unlike in our pseudo-global strategy, we will make use of two sequences of events that are closely related.

Original: To deﬁne these sequences, we will ﬁrst need to deﬁne stopping times.
Revised: To deﬁne these sequences, we will ﬁrst need to deﬁne stopping times.

Original: For every j + 1 ∈ N , deﬁne
Revised: For every j + 1 ∈ N , deﬁne

Original: [Equation]
Revised: [Equation]

Original: Analogously, for every j + 1 ∈ N , deﬁne
Revised: Analogously, for every j + 1 ∈ N , deﬁne

Original: [Equation]
Revised: [Equation]

Original:  Now, we will use (12) to establish the stability of the objective function, and we will use (13) to show that the gradient function must tend to zero.
Revised: [Equation] Now, we will use (12) to establish the stability of the objective function, and we will use (13) to show that the gradient function must tend to zero.

Original: Just as we did with B k
Revised: Just as we did with B k

Original: ( R ) , we will derive a recursion on the optimality gap over the events {{ τ j > k } : k + 1 ∈ N } .
Revised: ( R ) , we will derive a recursion on the optimality gap over the events {{ τ j > k } : k + 1 ∈ N } .

Original: Of course, just as before, the main challenge in deriving a recursive formula is to address { τ j = k } .
Revised: Of course, just as before, the main challenge in deriving a recursive formula is to address { τ j = k } .

Original: Our solution is supplied in the following lemma, whose proof is in Appendix D.
Revised: Our solution is supplied in the following lemma, whose proof is in Appendix D.

Original: Lemma 3.
Revised: Lemma 3.

Original: Let { M k } satisfy Property 1 .
Revised: Let { M k } satisfy Property 1 .

Original: Suppose Assumptions 1 to 4 hold.
Revised: Suppose Assumptions 1 to 4 hold.

Original: Let { θ k } satisfy (4) .
Revised: Let { θ k } satisfy (4) .

Original: Then, for any j + 1 ∈ N and k > j ,
Revised: Then, for any j + 1 ∈ N and k > j ,

Original: [Equation]
Revised: [Equation]

Original: From Lemma 3, there is a clear motivation for Assumption 5.
Revised: From Lemma [Equation] there is a clear motivation for Assumption 5.

Original: Indeed, if we apply Assumption 5,
Revised: Indeed [Equation] if we apply Assumption [Equation]

Original: Lemma 3 produces the following simple recursive relationship.
Revised: Lemma 3 produces the following simple recursive relationship.

Original: Lemma 4.
Revised: Lemma 4.

Revised: If Assumptions 1 to 5 , and Properties 1 and 4 hold

Original: If Assumptions 1 to 5 , and Properties 1 and 4 hold, and { θ k } satisfy (4) , then there exists a K ∈ N such that for any j + 1 ∈ N and any k ≥ min { K, j + 1 } ,
Revised: [Equation] and { θ k } satisfy (4) [Equation] then there exists a K ∈ N such that for any j + 1 ∈ N and any k ≥ min { K, j + 1 } ,

Original: [Equation]
Revised: [Equation]

Original: Just as in the pseudo-global strategy, Lemma 4 can be combined with standard martingale results (Robbins & Siegmund, 1971; Neveu & Speed, 1975, Exercise II.4) to conclude that the limit of
Revised: Just as in the pseudo-global strategy, Lemma 4 can be combined with standard martingale results (Robbins & Siegmund, 1971; Neveu & Speed, 1975, Exercise II.4) to conclude that the limit of

Original: F ( θ k ) exists and is ﬁnite on the event ∪ ∞
Revised: F ( θ k ) exists and is ﬁnite on the event ∪ ∞

Original: j =0 { τ j = ∞} (see Corollary 3).
Revised: j =0 { τ j = ∞} (see Corollary 3).

Original: Also as in the pseudoglobal strategy, by improving on the arguments in Patel (2020), we show that lim k ˙ F ( θ k ) = 0 on the event ∪ ∞ j =0 { ν j = ∞} (see Corollary 4).
Revised: Also as in the pseudoglobal strategy, by improving on the arguments in Patel (2020), we show that lim k ˙ F ( θ k ) = 0 on the event ∪ ∞ j =0 { ν j = ∞} (see Corollary 4).

Original: Finally, we show that ∪ ∞ j =0 { τ j = ∞} and ∪ ∞ j =0 { ν j = ∞} are probability one events (see Theorem 5).
Revised: Finally, we show that ∪ ∞ j =0 { τ j = ∞} and ∪ ∞ j =0 { ν j = ∞} are probability one events (see Theorem 5).

Original: In other words, we show that eventually L (cid:15) ( θ k ) will always dominate L ( θ k , θ k +1 ) .
Revised: In other words, we show that eventually L (cid:15) ( θ k ) will always dominate L ( θ k , θ k +1 ) .

Original: This statement should not come as a surprise on the event { lim k θ k exists } , but is slightly surprising that it must also hold on { lim k (cid:107) θ k (cid:107) 2 = ∞} .
Revised: This statement should not come as a surprise on the event { lim k θ k exists } , but is slightly surprising that it must also hold on { lim k (cid:107) θ k (cid:107) 2 = ∞} .

Original: By combining these results, we can conclude as follows.
Revised: By combining these results, we can conclude as follows.

Original: Theorem 3 (Stability) .
Revised: Theorem 3 (Stability) .

Original: Let θ
Revised: Let θ

Original: be arbitrary, and let { θ k : k ∈ N } be deﬁned according to (4) with { M k : k + 1 } satisfying Properties 1 to 3 .
Revised: be arbitrary, and let { θ k : k ∈ N } be deﬁned according to (4) with { M k : k + 1 } satisfying Properties 1 to 3 .

Original: Suppose Assumptions 1 to 5 hold.
Revised: Suppose Assumptions 1 to 5 hold.

Original: Then,
Revised: Then,

Original: There exists a ﬁnite random variable, F lim , such that lim k →∞ F ( θ k ) =
Revised: There exists a ﬁnite random variable, F lim , such that lim k →∞ F ( θ k ) =

Original: F lim with probability one; 2.
Revised: F lim with probability one; 2.

Original: lim k →∞ ˙ F ( θ k ) = 

Revised: lim k →∞ ˙ F ( θ k ) =

Original: with probability one.
Revised: with probability one.

Original: Proof.
Revised: Proof.

Original: Using Corollary 3, we conclude that ∃ F lim that is ﬁnite such that lim k F ( θ k ) =
Revised: Using Corollary 3, we conclude that ∃ F lim that is ﬁnite such that lim k F ( θ k ) =

Original: F lim on ∪ ∞ j =0 { τ j = ∞} .
Revised: F lim on ∪ ∞ j =0 { τ j = ∞} .

Original: Using Corollary 4, we conclude that lim k ˙ F ( θ k )
Revised: Using Corollary 4, we conclude that lim k ˙ F ( θ k )

Original: = 

Revised: =

Original: on ∪ ∞ j =0 { ν j = ∞} .
Revised: on ∪ ∞ j =0 { ν j = ∞} .

Original: Finally, we apply Theorem 5 to conclude that P [ ∪ ∞ j =0 { ν j = ∞} ]
Revised: Finally, we apply Theorem 5 to conclude that P [ ∪ ∞ j =0 { ν j = ∞} ]

Original: = P [ ∪ ∞ j =0 { τ j = ∞} ] = 1 .
Revised: = P [ ∪ ∞ j =0 { τ j = ∞} ] = 1 .

Original: See Section 1 for a brief discussion of the practical consequences of Theorem 3.
Revised: See Section 1 for a brief discussion of the practical consequences of Theorem 3.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we further ﬁlled the gap between SGD’s use in practice and SGD’s theory; that is,
Revised: In this work, we further ﬁlled the gap between SGD’s use in practice and SGD’s theory; that is,

Original: SGD is often applied in situations where the non-convexity of the objective and noise model are not covered by existing theory.
Revised: SGD is often applied in situations where the non-convexity of the objective and noise model are not covered by existing theory.

Original: Thus, we focused on analyzing SGD in a context that was more realistic to machine learning problems in two ways.
Revised: Thus, we focused on analyzing SGD in a context that was more realistic to machine learning problems in two ways.

Original: First, we eliminated the unrealistic assumption that the gradient of the objective function is globally H¨older continuous, and replaced it with a local
Revised: First, we eliminated the unrealistic assumption that the gradient of the objective function is globally H¨older continuous, and replaced it with a local

Original: H¨older continuity assumption.
Revised: H¨older continuity assumption.

Original: Second, we allowed for arbitrary noise models.
Revised: Second, we allowed for arbitrary noise models.

Original: This latter innovation suggests the possibility of potentially undesirable outcomes: the iterates can enter a limit cycle or they can oscillate.
Revised: This latter innovation suggests the possibility of potentially undesirable outcomes: the iterates can enter a limit cycle or they can oscillate.

Original: Perhaps most interestingly, in the process of establishing our global convergence result, we showed that these undesirable outcomes were impossible (Theorem 1): either the iterates converge to a stationary point or they diverge (Theorem 2).
Revised: Perhaps most interestingly, in the process of establishing our global convergence result, we showed that these undesirable outcomes were impossible (Theorem 1): either the iterates converge to a stationary point or they diverge (Theorem 2).

Original: The possibility of the iterates diverging also raises the question of what happens to the objective function when the iterates diverge.
Revised: The possibility of the iterates diverging also raises the question of what happens to the objective function when the iterates diverge.

Original: Thus, under an additional assumption on the joint behavior of the local H¨older constant, noise model and gradient function, we showed that the objective function remains ﬁnite along any iterate sequence (i.e., SGD is stable), and the gradient function converges to zero along this iterate sequence (Theorem 3).
Revised: Thus, under an additional assumption on the joint behavior of the local H¨older constant, noise model and gradient function, we showed that the objective function remains ﬁnite along any iterate sequence (i.e., SGD is stable), and the gradient function converges to zero along this iterate sequence (Theorem 3).

Original: Surprisingly, in the process of proving this result, we showed that, eventually, the constant L ( θ k , θ k +1 ) is eventually well controlled by L (cid:15) ( θ k ) , even if the iterate sequence is diverging .
Revised: Surprisingly, in the process of proving this result, we showed that, eventually, the constant L ( θ k , θ k +1 ) is eventually well controlled by L (cid:15) ( θ k ) , even if the iterate sequence is diverging .

Original: These results have several practical consequences, which we enumerate presently.
Revised: These results have several practical consequences, which we enumerate presently.

Original: First, by establishing the global convergence and stability of SGD over nonconvex functions with general noise models, we enable all other types of analyses: local convergence rate analyses, local weak converge analyses, saddle point escape time analyses, and complexity analyses.
Revised: First, by establishing the global convergence and stability of SGD over nonconvex functions with general noise models, we enable all other types of analyses: local convergence rate analyses, local weak converge analyses, saddle point escape time analyses, and complexity analyses.

Original: Second, our global convergence and stability results can be applied to realistic machine learning optimization problems, and provide guarantees about whether SGD will ﬁnd a stationary point.
Revised: Second, our global convergence and stability results can be applied to realistic machine learning optimization problems, and provide guarantees about whether SGD will ﬁnd a stationary point.

Original: Finally, as SGD continues to be applied to more complex problems outside of machine learning, our results are able to supply conﬁdence in the performance of the algorithm on such problems.
Revised: Finally, as SGD continues to be applied to more complex problems outside of machine learning, our results are able to supply conﬁdence in the performance of the algorithm on such problems.

Original: From a theoretical perspective, we innovated two new analysis strategies and substantially reﬁned a number of argument approaches currently in the literature.
Revised: From a theoretical perspective, we innovated two new analysis strategies and substantially reﬁned a number of argument approaches currently in the literature.

Original: We anticipate that these new argument strategies will be of substantial interest to those working in machine learning and stochastic optimization.
Revised: We anticipate that these new argument strategies will be of substantial interest to those working in machine learning and stochastic optimization.

Original: One of the important issues not considered in this work is the nature of L (cid:15) ( θ ) .
Revised: One of the important issues not considered in this work is the nature of L (cid:15) ( θ ) .

Original: In particular, there are functions that may admit upper bounds on L (cid:15) ( θ ) that ensure Assumption 5 is satisﬁed, yet using actual value of L (cid:15) ( θ ) would suggest that Assumption 5 is not satisﬁed.
Revised: In particular, there are functions that may admit upper bounds on L (cid:15) ( θ ) that ensure Assumption 5 is satisﬁed, yet using actual value of L (cid:15) ( θ ) would suggest that Assumption 5 is not satisﬁed.

Original: Exploring this issue will be the subject of future work.
Revised: Exploring this issue will be the subject of future work.

Original: Another area of future effort will be to construct a realistic example in which Assumption 5 is not satisﬁed, and to show that either (1) the iterates diverge, which implies that Assumption 5 is sufﬁcient and may be necessary; or (2) the iterates converge, which implies that Assumption 5 is sufﬁcient but not necessary.
Revised: Another area of future effort will be to construct a realistic example in which Assumption 5 is not satisﬁed, and to show that either (1) the iterates diverge, which implies that Assumption 5 is sufﬁcient and may be necessary; or (2) the iterates converge, which implies that Assumption 5 is sufﬁcient but not necessary.