Original: O N THE CONVERGENCE OF SGD UNDER THE OVERPARAMETER SETTING
Revised: O N THE CONVERGENCE OF SGD UNDER THE OVERPARAMETER SETTING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: With the improvement of computing power, over-parameterized models get increasingly popular in machine learning.
Revised: With the improvement of computing power, over-parameterized models get increasingly popular in machine learning.

Original: This type of model is usually with a complicated, non-smooth, and non-convex loss function landscape.
Revised: This type of model is usually with a complicated, non-smooth, and non-convex loss function landscape.

Original: However, when we train the model, simply using the first-order optimization algorithm like stochastic gradient descent (SGD) could acquire some good results, in both training and testing, albeit that SGD is known to not guarantee convergence for nonsmooth and non-convex cases.
Revised: However, when we train the model, simply using the first-order optimization algorithm like stochastic gradient descent (SGD) could acquire some good results, in both training and testing, albeit that SGD is known to not guarantee convergence for nonsmooth and non-convex cases.

Original: Theoretically, it was previously proved that in training, SGD converges to the global optimum with probability 1 − ϵ , but only for certain models and ϵ depends on the model complexity.
Revised: Theoretically, it was previously proved that in training, SGD converges to the global optimum with probability 1 − ϵ , but only for certain models and ϵ depends on the model complexity.

Original: It was also observed that SGD tends to choose a flat minimum, which preserves its training performance in testing.
Revised: It was also observed that SGD tends to choose a flat minimum, which preserves its training performance in testing.

Original: In this paper, we first prove that SGD could iterate to the global optimum almost surely under arbitrary initial value and some mild assumptions on the loss function.
Revised: In this paper, we first prove that SGD could iterate to the global optimum almost surely under arbitrary initial value and some mild assumptions on the loss function.

Original: Then, we prove that if the learning rate is larger than a value depending on the structure of a global minimum, the probability of converging to this global optimum is zero.
Revised: Then, we prove that if the learning rate is larger than a value depending on the structure of a global minimum, the probability of converging to this global optimum is zero.

Original: Finally, we acquire the asymptotic convergence rate based on the local structure of the global optimum.
Revised: Finally, we acquire the asymptotic convergence rate based on the local structure of the global optimum.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: With the improvement of the computing power of computer hardware, an increasing number of over-parameterized models are deployed in the domain of machine learning.
Revised: With the improvement of the computing power of computer hardware, an increasing number of over-parameterized models are deployed in the domain of machine learning.

Original: One of the most representative and successful models is what we called deep neural network (LeCun et al. (2015);
Revised: One of the most representative and successful models is what we called deep neural network (LeCun et al. (2015);

Original: Amodei et al.
Revised: Amodei et al.

Original: ; Graves et al.
Revised: ; Graves et al.

Original: ; He et al.
Revised: ; He et al.

Original: ; Silver et al.
Revised: ; Silver et al.

Original: (2017)), which has achieved great empirical success in various application areas (Wu et al.
Revised: (2017)), which has achieved great empirical success in various application areas (Wu et al.

Original: ; Krizhevsky et al.
Revised: ; Krizhevsky et al.

Original: (2017); Silver et al.
Revised: (2017); Silver et al.

Original: ; Halla et al.
Revised: ; Halla et al.

Original: Meanwhile, deep neural networks are large in scale and have an optimization landscape that is in general non-smooth and non-convex (Wu et al., 2019; Brutzkus & Globerson, 2017).
Revised: Meanwhile, deep neural networks are large in scale and have an optimization landscape that is in general non-smooth and non-convex (Wu et al., 2019; Brutzkus & Globerson, 2017).

Original: Training such a model should have been concerning.
Revised: Training such a model should have been concerning.

Original: However, people could usually acquire very good results just through using first-order methods such as stochastic gradient descent (SGD).
Revised: However, people could usually acquire very good results just through using first-order methods such as stochastic gradient descent (SGD).

Original: A large theoretical gap persists in understanding this process.
Revised: A large theoretical gap persists in understanding this process.

Original: Two main questions arise.
Revised: Two main questions arise.

Original: Due to the over-parametrization and the highly complex loss landscape of deep neural networks, optimizing the deep networks to the global optimum is likely NP-hard (Brutzkus & Globerson, 2017; Blum & Rivest, 1992).
Revised: Due to the over-parametrization and the highly complex loss landscape of deep neural networks, optimizing the deep networks to the global optimum is likely NP-hard (Brutzkus & Globerson, 2017; Blum & Rivest, 1992).

Original: Nevertheless, in practice, simple first-order methods, which does not have a convergence guarantee in the non-smooth and non-convex case (Liu et al., 2022a;b), are capable of finding a global optimum.
Revised: Nevertheless, in practice, simple first-order methods, which does not have a convergence guarantee in the non-smooth and non-convex case (Liu et al., 2022a;b), are capable of finding a global optimum.

Original: This happens even more often on the training data (Zhang et al., 2021; Brutzkus & Globerson, 2017; Wu et al., 2019).
Revised: This happens even more often on the training data (Zhang et al., 2021; Brutzkus & Globerson, 2017; Wu et al., 2019).

Original: It has been an open problem (Goodfellow et al., 2014) that, in this case, does SGD provably find the global optimum?
Revised: It has been an open problem (Goodfellow et al., 2014) that, in this case, does SGD provably find the global optimum?

Original: Does the result generalize to more general model structures beyond neural networks?
Revised: Does the result generalize to more general model structures beyond neural networks?

Original: In general, over-parametrized models offer many global optimums.
Revised: In general, over-parametrized models offer many global optimums.

Original: These global optimums have the same training loss of zero, and meanwhile drastically different test performance (Wu et al., 2018;
Revised: These global optimums have the same training loss of zero, and meanwhile drastically different test performance (Wu et al., 2018;

Original: Feng & Tu, 2021).
Revised: Feng & Tu, 2021).

Original: Interestingly, studies find that SGD tends to converge to those generalizable ones (Zhang et al., 2021).
Revised: Interestingly, studies find that SGD tends to converge to those generalizable ones (Zhang et al., 2021).

Original: In fact, it is observed empirically that SGD could usually find flat minima, which subsequently enjoys better generalization (Kramers, 1940; Dziugaite & Roy, 2017; Arpit et al., 2017; Kleinberg et al., 2018; Hochreiter & Schmidhuber, 1997; 1994).
Revised: In fact, it is observed empirically that SGD could usually find flat minima, which subsequently enjoys better generalization (Kramers, 1940; Dziugaite & Roy, 2017; Arpit et al., 2017; Kleinberg et al., 2018; Hochreiter & Schmidhuber, 1997; 1994).

Original: Why and how does SGD find a flat global minimum?
Revised: Why and how does SGD find a flat global minimum?

Original: The empirical finding has yet to be theoretically validated.
Revised: The empirical finding has yet to be theoretically validated.

Original: Related Works For the first question, in recent years, there have been a number of theoretical results that target to explain this phenomenon.
Revised: Related Works For the first question, in recent years, there have been a number of theoretical results that target to explain this phenomenon.

Original: Many of them focus on concrete neural network models, like two-layer networks with linear active function (Bartlett et al., 2018; Hardt & Ma, 2016).
Revised: Many of them focus on concrete neural network models, like two-layer networks with linear active function (Bartlett et al., 2018; Hardt & Ma, 2016).

Original: Several works need the inputs to be random Gaussian variables (Ge et al., 2018; Tian, 2017; Du et al., 2017; Zhong et al., 2017).
Revised: Several works need the inputs to be random Gaussian variables (Ge et al., 2018; Tian, 2017; Du et al., 2017; Zhong et al., 2017).

Original: Authors in Wu et al.
Revised: Authors in Wu et al.

Original: Allen-Zhu et al.
Revised: Allen-Zhu et al.

Original: (2019) consider the non-smooth case, but its techniques is depending on the structure of the network.
Revised: (2019) consider the non-smooth case, but its techniques is depending on the structure of the network.

Original: They prove when the number of nodes is enough large, the objective is “almost convex” and “semi-smooth”.
Revised: They prove when the number of nodes is enough large, the objective is “almost convex” and “semi-smooth”.

Original: The techniques unfortunately do not generalize to more general models.
Revised: The techniques unfortunately do not generalize to more general models.

Original: Another commonly used technique is to ignore the non-smoothness and apply the chain rule anyway on the non-smooth points (Bartlett et al., 2018).
Revised: Another commonly used technique is to ignore the non-smoothness and apply the chain rule anyway on the non-smooth points (Bartlett et al., 2018).

Original: The derivation does provide some intuitions but they do not offer any rigorous guarantees, as the chain rule does not hold (Liu et al., 2022a;b).
Revised: The derivation does provide some intuitions but they do not offer any rigorous guarantees, as the chain rule does not hold (Liu et al., 2022a;b).

Original: Even with these kinds of restrictions, existing works (Ge et al., 2018; Tian, 2017; Du et al., 2017; Bartlett et al., 2018; Vaswani et al., 2019; Chizat & Bach, 2018) only manage to find a high probability convergence result to the global optimum.
Revised: Even with these kinds of restrictions, existing works (Ge et al., 2018; Tian, 2017; Du et al., 2017; Bartlett et al., 2018; Vaswani et al., 2019; Chizat & Bach, 2018) only manage to find a high probability convergence result to the global optimum.

Original: The difference between this probability and 1 could depend on the structure of the model, like the number of nodes in the neural network, which raises further concerns on the tightness of the probability bound.
Revised: The difference between this probability and 1 could depend on the structure of the model, like the number of nodes in the neural network, which raises further concerns on the tightness of the probability bound.

Original: It is currently lacking to analyze SGD for general models to obtain an almost surely convergence to the global optimum.
Revised: It is currently lacking to analyze SGD for general models to obtain an almost surely convergence to the global optimum.

Original: For the second question, most works investigate the flat minima in a qualitative way.
Revised: For the second question, most works investigate the flat minima in a qualitative way.

Original: A recent work is by Xie et al.
Revised: A recent work is by Xie et al.

Original: (2020), which views the SGD process as a stochastic differential equation (SDE), and uses SDE to describe the process of the iteration escaping from the sharp minimum.
Revised: (2020), which views the SGD process as a stochastic differential equation (SDE), and uses SDE to describe the process of the iteration escaping from the sharp minimum.

Original: Similar techniques are also used in the works by Wu et al.
Revised: Similar techniques are also used in the works by Wu et al.

Original: ; Feng & Tu (2021).
Revised: ; Feng & Tu (2021).

Original: Unfortunately, SGD can be viewed as an SDE only when the learning rate is sufficiently small, and for a normal learning rate trajectories formed by SGD and SDE could be arbitrarily different.
Revised: Unfortunately, SGD can be viewed as an SDE only when the learning rate is sufficiently small, and for a normal learning rate trajectories formed by SGD and SDE could be arbitrarily different.

Original: Another technique used to study this problem is to use the linear stability (Wu et al., 2018; Feng & Tu, 2021), which considers a linear system near a global minimum.
Revised: Another technique used to study this problem is to use the linear stability (Wu et al., 2018; Feng & Tu, 2021), which considers a linear system near a global minimum.

Original: The behavior of SGD near some global minimum can then be characterized by the linear system of this global minimum.
Revised: The behavior of SGD near some global minimum can then be characterized by the linear system of this global minimum.

Original: However, different from a deterministic system where the property near one point can be quantitative determine by the linearized system of this point, a stochastic system property near one point is determined by all points in R d .
Revised: However, different from a deterministic system where the property near one point can be quantitative determine by the linearized system of this point, a stochastic system property near one point is determined by all points in R d .

Original: Using this linearized function to fully represent SGD near some global minimum is thus not a rigorous argument.
Revised: Using this linearized function to fully represent SGD near some global minimum is thus not a rigorous argument.

Original: Contributions
Revised: Contributions

Original: Under several mild assumptions about the non-smooth and non-convex loss function, we provide the first proof that from an arbitrary initialization SGD could make the iteration converge to the global optimum almost surely, i.e., P ( θ n converges to a global optimum ) = 1 .
Revised: Under several mild assumptions about the non-smooth and non-convex loss function, we provide the first proof that from an arbitrary initialization SGD could make the iteration converge to the global optimum almost surely, i.e., P ( θ n converges to a global optimum ) = 1 .

Original: Under the same set of assumptions and the same setting of SGD, we prove that if the learning rate is larger than a threshold, which depends on the sharpness of a global minimum, the probability which the iteration converges to this global optimum is strictly 0 .
Revised: Under the same set of assumptions and the same setting of SGD, we prove that if the learning rate is larger than a threshold, which depends on the sharpness of a global minimum, the probability which the iteration converges to this global optimum is strictly 0 .

Original: With similar assumptions and the same setting, we acquire the asymptotic convergence rate of the iteration converging to the global optimum.
Revised: With similar assumptions and the same setting, we acquire the asymptotic convergence rate of the iteration converging to the global optimum.

Original: By this result, we know that SGD achieves an arbitrary accuracy in polynomial time.
Revised: By this result, we know that SGD achieves an arbitrary accuracy in polynomial time.

Original: Technical Insight
Revised: Technical Insight

Original: The basic intuition is as follows.
Revised: The basic intuition is as follows.

Original: We first understand the SGD as a Markov chain with the continuous state space.
Revised: We first understand the SGD as a Markov chain with the continuous state space.

Original: Then we aim to prove that the global optimum is the only absorbing state of this Markov chain.
Revised: Then we aim to prove that the global optimum is the only absorbing state of this Markov chain.

Original: Concretely, due to the property of the sampling noise, this noise enjoys 0 variance when the optimization variable θ reaches the global optimum (Claim 2.1), i.e., E ξ n ∥ ˜ ∇ g ( θ, ξ n ) − ˜ ∇ g ( θ ) ∥ 2 = 0 (notations are defined in the next section), which guarantees that once θ n reaches the global optimum, it will not escape from the optimum.
Revised: Concretely, due to the property of the sampling noise, this noise enjoys 0 variance when the optimization variable θ reaches the global optimum (Claim 2.1), i.e., E ξ n ∥ ˜ ∇ g ( θ, ξ n ) − ˜ ∇ g ( θ ) ∥ 2 = 0 (notations are defined in the next section), which guarantees that once θ n reaches the global optimum, it will not escape from the optimum.

Original: Meanwhile, in other local optimums, the positive variance makes θ n jump out to this local optimum.
Revised: Meanwhile, in other local optimums, the positive variance makes θ n jump out to this local optimum.

Original: Otherwise, as this Markov chain is a continuous state space
Revised: Otherwise, as this Markov chain is a continuous state space

Original: Markov chain, an absorbing state with the measure 0 cannot become the real absorbing state (the probability of the θ n reaching this absorbing state in every epoch is 0).
Revised: Markov chain, an absorbing state with the measure 0 cannot become the real absorbing state (the probability of the θ n reaching this absorbing state in every epoch is 0).

Original: Based on this, we need this absorbing state to have a flat-enough neighborhood (Assumption 2.2 in the new version), which deduces that θ n that fall on this neighborhood tend to move closer to this absorbing state.
Revised: Based on this, we need this absorbing state to have a flat-enough neighborhood (Assumption 2.2 in the new version), which deduces that θ n that fall on this neighborhood tend to move closer to this absorbing state.

Original: Combining this absorbing state and this neighborhood statement, we can prove the distribution of θ n will concentrate on the global optimum when as the iteration goes.
Revised: Combining this absorbing state and this neighborhood statement, we can prove the distribution of θ n will concentrate on the global optimum when as the iteration goes.

Original: Finally, this distribution will degenerate to the global optimum, that is, θ n will converge to the global optimum.
Revised: Finally, this distribution will degenerate to the global optimum, that is, θ n will converge to the global optimum.

Original: This neighborhood is the key insight of proving the convergence of SGD.
Revised: This neighborhood is the key insight of proving the convergence of SGD.

Original: The neighborhood cannot be very sharp (have at most quadratic growth), which is the reason we made Assumption 2.2, item 1.
Revised: The neighborhood cannot be very sharp (have at most quadratic growth), which is the reason we made Assumption 2.2, item 1.

Original: It is actually reflected in Equation (8).
Revised: It is actually reflected in Equation (8).

Original: A flat enough neighborhood can make the coefficient of the third term of (8) negative, which in turn makes the R ( θ n ) (the Lyapunov function) to decrease with high probability ( θ n close to global optimum).
Revised: A flat enough neighborhood can make the coefficient of the third term of (8) negative, which in turn makes the R ( θ n ) (the Lyapunov function) to decrease with high probability ( θ n close to global optimum).

Original: Otherwise, if the neighborhood is sharp, this coefficient will become positive, which makes R ( θ n ) increasing ( θ n away from global optimum).
Revised: Otherwise, if the neighborhood is sharp, this coefficient will become positive, which makes R ( θ n ) increasing ( θ n away from global optimum).

Original:  P ROBLEM F ORMULATION
Revised:  P ROBLEM F ORMULATION

Original: We investigate SGD under the over-parametrization setting, under a few mild assumptions on the objective function.
Revised: We investigate SGD under the over-parametrization setting, under a few mild assumptions on the objective function.

Original: The setting and the assumptions, as well as some preliminaries that are relevant to the results, are provided in Section 2.1.
Revised: The setting and the assumptions, as well as some preliminaries that are relevant to the results, are provided in Section 2.1.

Original: We then present the sampling schemes in Section 2.2.
Revised: We then present the sampling schemes in Section 2.2.

Original: 2. O PTIMIZATION UNDER O VER - PARAMETRIZATION
Revised: 2. O PTIMIZATION UNDER O VER - PARAMETRIZATION

Original: In this paper, given a dataset D = { ( x i , y i ) } , x i , y i ∈ R d , we consider a model ˆ y i
Revised: In this paper, given a dataset D = { ( x i , y i ) } , x i , y i ∈ R d , we consider a model ˆ y i

Original: =
Revised: =

Original: f
Revised: f

Original: ( θ, x i ) , and the mean-square error (MSE) loss, i.e.,
Revised: ( θ, x i ) , and the mean-square error (MSE) loss, i.e.,

Original: [Equation]
Revised: [Equation]

Original: The goal of an optimization method, like SGD, is to obtain an optimum θ ∈ J ∗ , where J ∗ = arg min θ ∈ R d g ( θ ) .
Revised: The goal of an optimization method, like SGD, is to obtain an optimum θ ∈ J ∗ , where J ∗ = arg min θ ∈ R d g ( θ ) .

Original: In the over-parametrization setting, this optimum is zero.
Revised: In the over-parametrization setting, this optimum is zero.

Original: To handle the non-smoothness, we recall the definition of Clarke subdifferential (Clarke, 1990), which is an important tool to design and operate SGD algorithms.
Revised: To handle the non-smoothness, we recall the definition of Clarke subdifferential (Clarke, 1990), which is an important tool to design and operate SGD algorithms.

Original: Definition 1 (Clarke subdifferential (Clarke, 1990)) .
Revised: Definition 1 (Clarke subdifferential (Clarke, 1990)) .

Original: Let ¯ x ∈ Ω be given.
Revised: Let ¯ x ∈ Ω be given.

Original: The Clarke subdifferential of f at ¯ x is defined by
Revised: The Clarke subdifferential of f at ¯ x is defined by

Original: [Equation]
Revised: [Equation]

Original: where co represents the convex hull.
Revised: where co represents the convex hull.

Original: If f is furthermore smooth, it holds that ∂f ( x ) = {∇ f ( x ) } .
Revised: If f is furthermore smooth, it holds that ∂f ( x ) = {∇ f ( x ) } .

Original: We use ˜ ∇ f ( x ) to denote an arbitrary element in ∂f ( x ) , and for convenient, we call ˜ ∇ as subgradient.
Revised: We use ˜ ∇ f ( x ) to denote an arbitrary element in ∂f ( x ) , and for convenient, we call ˜ ∇ as subgradient.

Original: The Clarke subdifferential does not enjoy the chain rule and several techniques involved in regular gradient cannot be reused in our case.
Revised: The Clarke subdifferential does not enjoy the chain rule and several techniques involved in regular gradient cannot be reused in our case.

Original: We provide a counterexample to illustrate this in Claim A.1.
Revised: We provide a counterexample to illustrate this in Claim A.1.

Original: This property and a few assumptions to eliminate pathological cases are described in the below assumption.
Revised: This property and a few assumptions to eliminate pathological cases are described in the below assumption.

Original: Assumption 2.1.
Revised: Assumption 2.1.

Original: The loss function g ( θ ) satisfies the following conditions:
Revised: The loss function g ( θ ) satisfies the following conditions:

Original: g ( θ ) is continuous and smooth almost everywhere; 2.
Revised: g ( θ ) is continuous and smooth almost everywhere; 2.

Original: The global optimum value of g ( θ ) is 0 ; 3.
Revised: The global optimum value of g ( θ ) is 0 ; 3.

Original: The set of global optimum points J ∗ is composed of countably connected components J i , i.e., J ∗ =
Revised: The set of global optimum points J ∗ is composed of countably connected components J i , i.e., J ∗ =

Original: (cid:83) + ∞ i =
Revised: (cid:83) + ∞ i =

Original: 1 J i ( J i ∩ J j = ∅ ) ; 4.
Revised: 1 J i ( J i ∩ J j = ∅ ) ; 4.

Original: There is a scalar c > 0 , such that whenever g is smooth on θ 1 , θ 2 then for any data point ( x i , y i ) ,
Revised: There is a scalar c > 0 , such that whenever g is smooth on θ 1 , θ 2 then for any data point ( x i , y i ) ,

Original: [Equation]
Revised: [Equation]

Original: This assumption describes the overall structure of the loss function g ( θ ) .
Revised: This assumption describes the overall structure of the loss function g ( θ ) .

Original: All 4 items in this Assumption are quite mild and are commonly used in optimization and learning.
Revised: All 4 items in this Assumption are quite mild and are commonly used in optimization and learning.

Original: Items 1 and 2 are true under the MSE loss and the over-parametrization setting.
Revised: Items 1 and 2 are true under the MSE loss and the over-parametrization setting.

Original: Item 3 describes that the optimum is composed of countably many connected components and this item holds for almost all functions unless one delicately constructs a pathological counterexample Jin et al.
Revised: Item 3 describes that the optimum is composed of countably many connected components and this item holds for almost all functions unless one delicately constructs a pathological counterexample Jin et al.

Original: In this paper, to make the presentation clear, we continue with the countably many points assumption
Revised: In this paper, to make the presentation clear, we continue with the countably many points assumption

Original: J ∗ =
Revised: J ∗ =

Original: (cid:83) + ∞ i =1 { θ ∗ i } to avoid the tedious arguments on continuum of optimums.
Revised: (cid:83) + ∞ i =1 { θ ∗ i } to avoid the tedious arguments on continuum of optimums.

Original: Item 4 can be seen as a non-smooth extension of the traditional L -smooth condition, i.e., ∥∇ g ( x ) − ∇ g ( y )
Revised: Item 4 can be seen as a non-smooth extension of the traditional L -smooth condition, i.e., ∥∇ g ( x ) − ∇ g ( y )

Original: ∥
Revised: ∥

Original: ≤ L ∥ x − y ∥ .
Revised: ≤ L ∥ x − y ∥ .

Original: It can be satisfied by many non-smooth functions, like ReLU and leaky-ReLU.
Revised: It can be satisfied by many non-smooth functions, like ReLU and leaky-ReLU.

Original: Similar to the regular gradient, the subgradient is also zero at the optimum.
Revised: Similar to the regular gradient, the subgradient is also zero at the optimum.

Original: Claim 2.1.
Revised: Claim 2.1.

Original: For the MSE loss function (1), if the global optimum is 0 , i.e., min θ ∈ R d g ( θ ) = 0 .
Revised: For the MSE loss function (1), if the global optimum is 0 , i.e., min θ ∈ R d g ( θ ) = 0 .

Original: Then the subgradient at the optimum points J ∗ is 0 .
Revised: Then the subgradient at the optimum points J ∗ is 0 .

Original: Proof.
Revised: Proof.

Original: For any θ 0 ∈ { θ | g is smooth at θ } , we can get that
Revised: For any θ 0 ∈ { θ | g is smooth at θ } , we can get that

Original: [Equation]
Revised: [Equation]

Original: Then for any θ ∗ ∈ J ∗ , we have
Revised: Then for any θ ∗ ∈ J ∗ , we have

Original: [Equation]
Revised: [Equation]

Original: where g is smooth at θ 0 .
Revised: where g is smooth at θ 0 .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: This concludes that ˜ ∇ g ( θ ∗ ) = 0 .
Revised: This concludes that ˜ ∇ g ( θ ∗ ) = 0 .

Original: Notice that despite that g is non-smooth in general, in our setting, it is smooth on the optimum as described in the above claim.
Revised: Notice that despite that g is non-smooth in general, in our setting, it is smooth on the optimum as described in the above claim.

Original: This distinguishes our setting from the line of literature on non-smooth optimization.
Revised: This distinguishes our setting from the line of literature on non-smooth optimization.

Original: To make a global convergence, we need at least one θ ∗ ∈ J ∗ to be not very “sharp”.
Revised: To make a global convergence, we need at least one θ ∗ ∈ J ∗ to be not very “sharp”.

Original: That is, at the δ θ ∗ − neighboring of θ ∗ the loss function holds L-smooth condition with the coefficient β θ ∗ and an assumption as follow:
Revised: That is, at the δ θ ∗ − neighboring of θ ∗ the loss function holds L-smooth condition with the coefficient β θ ∗ and an assumption as follow:

Original: Assumption 2.2.
Revised: Assumption 2.2.

Original: There exist θ ∗ ∈ J ∗ , r θ ∗ ≥ 1 , δ > 0 , a neighboring area
Revised: There exist θ ∗ ∈ J ∗ , r θ ∗ ≥ 1 , δ > 0 , a neighboring area

Original: U ( θ ∗ , δ θ ∗ ) of θ ∗ , such that for those θ ∈ U ( θ ∗ , δ θ ∗ ) that ˜ ∇ g ( θ ) holds
Revised: U ( θ ∗ , δ θ ∗ ) of θ ∗ , such that for those θ ∈ U ( θ ∗ , δ θ ∗ ) that ˜ ∇ g ( θ ) holds

Original: For any mini-batch C i , g C i ( θ ) holds the local one point L-smooth condition,
Revised: For any mini-batch C i , g C i ( θ ) holds the local one point L-smooth condition,

Original: i
Revised: i

Original: e. ∥
Revised: e. ∥

Original: ˜ ∇ g ( θ ) ∥ < β θ ∗ ∥ θ − θ ∗ ∥ ( ∀ θ ∈ U ( θ ∗ , δ θ ∗ )) .
Revised: ˜ ∇ g ( θ ) ∥ < β θ ∗ ∥ θ − θ ∗ ∥ ( ∀ θ ∈ U ( θ ∗ , δ θ ∗ )) .

Original: The loss function holds ˜ ∇ g ( θ ) T ( θ − θ ∗ ) >
Revised: The loss function holds ˜ ∇ g ( θ ) T ( θ − θ ∗ ) >

Original: α θ ∗ ∥ θ − θ ∗ ∥ r θ ∗ +1 ( ∀ θ ∈ U ( θ ∗ , δ θ ∗ ))) , for some constant α θ ∗ > 0 .
Revised: α θ ∗ ∥ θ − θ ∗ ∥ r θ ∗ +1 ( ∀ θ ∈ U ( θ ∗ , δ θ ∗ ))) , for some constant α θ ∗ > 0 .

Original: The first item of this assumption is very mild.
Revised: The first item of this assumption is very mild.

Original: Due to Claim 2.1, we know g ( θ ) is smooth in θ ∗ , that is, lim θ → θ ∗ ˜ ∇ g ( θ ) =
Revised: Due to Claim 2.1, we know g ( θ ) is smooth in θ ∗ , that is, lim θ → θ ∗ ˜ ∇ g ( θ ) =

Original: ∇ g
Revised: ∇ g

Original: θ ∗ ) = 0 .
Revised: θ ∗ ) = 0 .

Original: Then item 1 is just to bound the speed of subgradient tend to 0 is not slower than a linear function (not too sharp as O ( (cid:112)
Revised: Then item 1 is just to bound the speed of subgradient tend to 0 is not slower than a linear function (not too sharp as O ( (cid:112)

Original: ∥
Revised: ∥

Original: θ − θ ∗ ∥ ) or O ( ∥ θ − θ ∗ ∥ 0 . 9 ) ).
Revised: θ − θ ∗ ∥ ) or O ( ∥ θ − θ ∗ ∥ 0 . 9 ) ).

Original: The second item of this assumption is very close to the local Kurdyka-Lojasiewicz condition, i.e. ∥∇ g ( θ )
Revised: The second item of this assumption is very close to the local Kurdyka-Lojasiewicz condition, i.e. ∥∇ g ( θ )

Original: ∥ 2 r ≥ g
Revised: ∥ 2 r ≥ g

Original: ( θ ) − g ( θ ∗ ) ( r ≥ 1)( θ ∈ U ( θ ∗ , δ θ ∗ )) which is a typically mild condition used to substitute the local Polyak-Łojasiewicz condition (item 2 and the local Kurdyka-Lojasiewicz condition are totally equivalent for a unary function).
Revised: ( θ ) − g ( θ ∗ ) ( r ≥ 1)( θ ∈ U ( θ ∗ , δ θ ∗ )) which is a typically mild condition used to substitute the local Polyak-Łojasiewicz condition (item 2 and the local Kurdyka-Lojasiewicz condition are totally equivalent for an unary function).

Original: This assumption is milder than several assumptions used in the previous works.
Revised: This assumption is milder than several assumptions used in the previous works.

Original: It can be seen as the loss function has an r θ ∗ + 1 -order Taylor expansion on θ ∗ .
Revised: It can be seen as the loss function has an r θ ∗ + 1 -order Taylor expansion on θ ∗ .

Original: Compared with the one point strongly convexity used in Li & Yuan (2017); Kleinberg et al.
Revised: Compared with the one point strongly convexity used in Li & Yuan (2017); Kleinberg et al.

Original: (2018), the positive Hessian matrix and local Polyak-Łojasiewicz condition in global optimum used in Wu et al.
Revised: (2018), the positive Hessian matrix and local Polyak-Łojasiewicz condition in global optimum used in Wu et al.

Original: ; Jin et al.
Revised: ; Jin et al.

Original: (2022), our assumption is much milder.
Revised: (2022), our assumption is much milder.

Original: 2. T WO TYPES OF NOISE OF SGD
Revised: 2. T WO TYPES OF NOISE OF SGD

Original: In the rest of this section we describe two types of SGD algorithms, by different sampling noise.
Revised: In the rest of this section we describe two types of SGD algorithms, by different sampling noise.

Original: The first type is with the traditional sampling noise while the second type is SGD with the sampling noise with global stable guarantee.
Revised: The first type is with the traditional sampling noise while the second type is SGD with the sampling noise with global stable guarantee.

Original: They involve slightly different assumptions and the analysis of
Revised: They involve slightly different assumptions and the analysis of

Original: SGD also varies by the type of noise.
Revised: SGD also varies by the type of noise.

Original: Nevertheless, they conclude similar results as we will present in the next section.
Revised: Nevertheless, they conclude similar results as we will present in the next section.

Original: 2.2. R EGULAR SAMPLING NOISE
Revised: 2.2. R EGULAR SAMPLING NOISE

Original: We start with the iterations of an (regular) SGD algorithm, that
Revised: We start with the iterations of an (regular) SGD algorithm, that

Original: [Equation]
Revised: [Equation]

Original: where { ξ n } represents the sampling noise.
Revised: where { ξ n } represents the sampling noise.

Original: That is, we have the noised sampling
Revised: That is, we have the noised sampling

Original: [Equation]
Revised: [Equation]

Original: where C i is a randomly selected mini-batch from the original data set.
Revised: where C i is a randomly selected mini-batch from the original data set.

Original: The next statement assumes that the subdgradient can be sampled without the sampling error being too large.
Revised: The next statement assumes that the subdgradient can be sampled without the sampling error being too large.

Original: It is necessary for an algorithm to use the gradient:
Revised: It is necessary for an algorithm to use the gradient:

Original: Assumption 2.3.
Revised: Assumption 2.3.

Original: Let ξ n be the sampling noise involved in the n -th iteration of SGD and ˜ ∇ g ( θ, ξ n ) be the noised sampling of the subgradient.
Revised: Let ξ n be the sampling noise involved in the n -th iteration of SGD and ˜ ∇ g ( θ, ξ n ) be the noised sampling of the subgradient.

Original: For any θ ∈ R d , it holds
Revised: For any θ ∈ R d , it holds

Original: [Equation]
Revised: [Equation]

Original:  where M 0 > 0 and a> 0 are constants decided by g .
Revised: and [Equation] where M 0 ≥ 0 is a constants decided by g .

Original: Meanwhile,  lim inf θ →∞ ∥
Revised: Meanwhile, we need lim inf θ →∞ ∥

Original: ˜ ∇ g ( θ 
Revised: ˜ ∇ g ( θ ) ∥ >

Original: [Equation]
Revised: [Equation]

Original:  ∥ >
Revised: Then we explain why we request M 0 ≥ 1 . Due to for any binary number pair ( M 0 < 1 , a ) , the bound it forms is not a sharp bound. We can always construct a sharp bound 1 ·∥ ˜ ∇ g ( θ ) ∥ 2 + a − (1 −

Original: [Equation]
Revised: M 0 ) ∥

Original:  First of this assumption is milder than the widely used bounded variance assumption ,i.e.,
Revised: ˜ ∇ g ( θ ) ∥ 2 First of this assumption is milder than the widely used bounded variance assumption ,

Original:  E ξ n (cid:13)(cid:13) ˜ ∇ g ( θ, ξ n ) − ˜ ∇ g ( θ ) (cid:13)(cid:13) 2 ≤ a (Li & Yuan, 2017; Kleinberg et al., 2018).
Revised: i.e., E ξ n (cid:13)(cid:13) ˜ ∇ g ( θ, ξ n ) − ˜ ∇ g ( θ ) (cid:13)(cid:13) 2 ≤ a (Li & Yuan, 2017; Kleinberg et al., 2018).

Original: Second part is to combine the { θ n } tend to ∞ .
Revised: Second part is to combine the { θ n } tend to ∞ .

Original: For example, for a very simple loss functions g ( θ ) = 13 (cid:0)
Revised: For example, for a very simple loss functions g ( θ ) = 13 (cid:0)

Original: ∥
Revised: ∥

Original: θ − θ 1 ∥ 2 + ∥ θ − θ 2 ∥ 2 + ∥ θ − θ 3 ∥ 2 (cid:1) , It hold our Assumption 2.3 but not hold bounded variance assumption.
Revised: θ − θ 1 ∥ 2 + ∥ θ − θ 2 ∥ 2 + ∥ θ − θ 3 ∥ 2 (cid:1) , It hold our Assumption 2.3 but not hold bounded variance assumption.

Original: Meanwhile, this sampling immediately implies the below bound.
Revised: Meanwhile, this sampling immediately implies the below bound.

Original: Claim 2.2.
Revised: Claim 2.2.

Original: For any bounded set Q that include J ∗ , it holds
Revised: For any bounded set Q that include J ∗ , it holds

Original: [Equation]
Revised: [Equation]

Original: where G Q is a constant decided by Q .
Revised: where G Q is a constant decided by Q .

Original: Proof.
Revised: Proof.

Original: For any smooth point in Q , the mini-batch gradient norm satisfies
Revised: For any smooth point in Q , the mini-batch gradient norm satisfies

Original: [Equation]
Revised: [Equation]

Original: where N 0 is the size of the mini-batch.
Revised: where N 0 is the size of the mini-batch.

Original: Define
Revised: Define

Original: [Equation]
Revised: [Equation]

Original: Through Assumption 2.
Revised: N

Original: [Equation]
Revised: [Equation]

Original:  we know that h ( θ ) is bounded on smooth points.
Revised: Through Assumption 2.1, we know that h ( θ ) is bounded on smooth points.

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: For the non-smooth point θ [Equation] we can prove for any sequence θ [-
-] → θ ( g is smooth at θ 0 ), through
Revised: For the non-smooth point θ , we can prove for any sequence θ [-
-]0 → θ ( g is smooth at θ 0 ), through

Original: Equation (4) [Equation] there is
Revised: Equation (4), there is

Original: [Equation]
Revised: [Equation]

Original: Recall the following fact:
Revised: Recall the following fact:

Original: If ∥ a 1 ∥ 2 < s 0 , ∥ a 2 ∥ 2 < s 0 , . .
Revised: If ∥ a 1 ∥ 2 < s 0 , ∥ a 2 ∥ 2 < s 0 , . .

Original: , ∥ a n ∥ 2 < s 0 , the norm of their any convex combination 
Revised: , ∥ a n ∥ 2 < s 0 , the norm of their any convex combination ∥ ¯ a ∥ [Equation] =

Original: [Equation]
Revised: [Equation]

Original: Then we obtain
Revised: Then we obtain

Original: [Equation]
Revised: [Equation]

Original: This concludes that
Revised: This concludes that

Original: [Equation]
Revised: [Equation]

Original: We could observe that the noise variance E ξ n ∥ ˜ ∇ g ( θ [Equation] − ˜ ∇ g [Equation]
Revised: We could observe that the noise variance E ξ n ∥ ˜ ∇ g ( θ, ξ n ) − ˜ ∇ g ( θ ) ∥ 2 = 0 at the global optimum (Claim 2.1).

Original: = 0 at the global optimum (Claim 2.1).

Original: Intuitively, the zero variance makes the θ n stable in the global optimum, while for a local minimum or a saddle point the variance is nonzero in general.
Revised: Intuitively, the zero variance makes the θ n stable in the global optimum, while for a local minimum or a saddle point the variance is nonzero in general.

Original: This is intuitively how SGD escapes from local minimum and saddle points.
Revised: This is intuitively how SGD escapes from local minimum and saddle points.

Original: We have to notice that the global optimum is a subset of the set where the noise variance equals 0 .
Revised: We have to notice that the global optimum is a subset of the set where the noise variance equals 0 .

Original: It is easy to prove that
Revised: It is easy to prove that

Original: [Equation]
Revised: [Equation]

Original: where J ∗∗ is equivalent to
Revised: where J ∗∗ is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Our techniques will eventually prove that the SGD with regular sampling noise converges to J ∗∗ .
Revised: Our techniques will eventually prove that the SGD with regular sampling noise converges to J ∗∗ .

Original: This could be different than J ∗ in theory, but intuitively, for the over-parameter model and a large amount of data the model f ( θ, x ) is complex enough to make sure that other stationary points are sensitive to the mini-batch batch selection.
Revised: This could be different than J ∗ in theory, but intuitively, for the over-parameter model and a large amount of data the model f ( θ, x ) is complex enough to make sure that other stationary points are sensitive to the mini-batch batch selection.

Original: As such making a point, that is not the global optimum, stationary to all batches simultaneously is almost impossible, i.e., J ∗∗ /J ∗ = ∅ .
Revised: As such making a point, that is not the global optimum, stationary to all batches simultaneously is almost impossible, i.e., J ∗∗ /J ∗ = ∅ .

Original: Nevertheless, in order to insure the rigor of the theory, we make an additional assumption only for the regular sampling noise.
Revised: Nevertheless, in order to insure the rigor of the theory, we make an additional assumption only for the regular sampling noise.

Original: This assumption is lifted in the sampling noise with global stable guarantee.
Revised: This assumption is lifted in the sampling noise with global stable guarantee.

Original: Additional assumption for regular sampling noise For the sampling noise { ξ n } , points that are stationary to all mini-batches must be in J ∗ , i.e., J ∗ =
Revised: Additional assumption for regular sampling noise For the sampling noise { ξ n } , points that are stationary to all mini-batches must be in J ∗ , i.e., J ∗ =

Original: J ∗∗ .
Revised: J ∗∗ .

Original: Meanwhile, for every mini-batch loss function g C i , the stationary point set of g C i is countable.
Revised: Meanwhile, for every mini-batch loss function g C i , the stationary point set of g C i is countable.

Original: If one slightly modifies SGD by adding an additional Gaussian noise, we will prove that such sampling noise will enjoy a global stable guarantee.
Revised: If one slightly modifies SGD by adding an additional Gaussian noise, we will prove that such sampling noise will enjoy a global stable guarantee.

Original: With this variant of SGD, the above assumption could be lifted.
Revised: With this variant of SGD, the above assumption could be lifted.

Original: We now present our proposed variant of SGD.
Revised: We now present our proposed variant of SGD.

Original: 2.2. S AMPLING NOISE WITH GLOBAL STABLE GUARANTEE
Revised: 2.2. S AMPLING NOISE WITH GLOBAL STABLE GUARANTEE

Original: The sampling noise we propose in this section is the regular noise in SGD plus an extra Gaussian noise, as
Revised: The sampling noise we propose in this section is the regular noise in SGD plus an extra Gaussian noise, as

Original: [Equation]
Revised: [Equation]

Original: where { ξ n } again represents the sampling noise, K 0 is a constant to prevent the noise from approaching infinity, {N n } represents a mutually independent standard Gaussian noise, { τ n } is a mutually independent Bernoulli variable, i.e., P ( τ n = 0)
Revised: where { ξ n } again represents the sampling noise, K 0 is a constant to prevent the noise from approaching infinity, {N n } represents a mutually independent standard Gaussian noise, { τ n } is a mutually independent Bernoulli variable, i.e., P ( τ n = 0)

Original: =
Revised: =

Original: p 0 , P ( τ n = 1) =
Revised: p 0 , P ( τ n = 1) =

Original: 1 − p 0 , and { τ n } , { ξ n } , {N n } are also mutually independent.
Revised: 1 − p 0 , and { τ n } , { ξ n } , {N n } are also mutually independent.

Original: The coefficient min { g ( θ n ) , K 0 } is to make sure the algorithm hold a positive noise variance E ξ n ,τ n , N n ∥ v n ∥ 2 > 0 in non-optimal stationary points.
Revised: The coefficient min { g ( θ n ) , K 0 } is to make sure the algorithm hold a positive noise variance E ξ n ,τ n , N n ∥ v n ∥ 2 > 0 in non-optimal stationary points.

Original: We use { τ n } to reduce the scale of the problem, making the scale of the new noise equal to the scale of the mini-batch gradient ˜ ∇ g ( θ n , ξ n ) and as the original sampling noise.
Revised: We use { τ n } to reduce the scale of the problem, making the scale of the new noise equal to the scale of the mini-batch gradient ˜ ∇ g ( θ n , ξ n ) and as the original sampling noise.

Original: For example, if the batch size is 100 and the scale of the original data set is 10000 , then we can set p 

Revised: For example, if the batch size is 100 and the scale of the original data set is 10000 , then we can set p

Original: = 1 − 0 .
Revised: = 1 − 0 .

Original: 01 , which makes the average scale of the noise min { g ( θ n ) , K 0 } τ n N n (cid:1) also 100 .
Revised: 01 , which makes the average scale of the noise min { g ( θ n ) , K 0 } τ n N n (cid:1) also 100 .

Original: The tail term (cid:112) min { g ( θ n ) , K 0 } τ n N n (cid:1) guarantees that this algorithm has a positive variance in R d /J ∗ .
Revised: The tail term (cid:112) min { g ( θ n ) , K 0 } τ n N n (cid:1) guarantees that this algorithm has a positive variance in R d /J ∗ .

Original:  M AIN RESULTS
Revised:  M AIN RESULTS

Original: Our first main result states that SGD must converge to a global optimum with probability 1 .
Revised: Our first main result states that SGD must converge to a global optimum with probability 1 .

Original: This is a large improvement from previous results with only 1 − δ probability, where δ depends on the model.
Revised: This is a large improvement from previous results with only 1 − δ probability, where δ depends on the model.

Original: Our theorem answers the question raised in the introduction, affirmatively, that SGD could indeed obtain a global optimum even in this non-smooth non-convex over-parameter setting.
Revised: Our theorem answers the question raised in the introduction, affirmatively, that SGD could indeed obtain a global optimum even in this non-smooth non-convex over-parameter setting.

Original: The next two theorems discuss the cases of r θ ∗ > 1 (higher than second-order local structure) and r θ ∗ =(second-order local structure) respectively.
Revised: The next two theorems discuss the cases of r θ ∗ > 1 (higher than second-order local structure) and r θ ∗ =(second-order local structure) respectively.

Original: Theorem 3.1. Consider the SGD iteration in Equation (5), or alternatively Equation (2) with J ∗ =
Revised: Theorem 3.1. Consider the SGD iteration in Equation (5), or alternatively Equation (2) with J ∗ =

Original: J ∗∗ , and the MSE loss function (1).
Revised: J ∗∗ , and the MSE loss function (1).

Original: If Assumptions 2.1, 2.3 hold, and Assumption 2.2 holds with r θ ∗ > 1 , then for any 0 < ϵ 0 < min { 1 / 2 cM 0 , a/ 4 c ( K 0 (1 − p 0 )+ a ) } , and for any initialization θ 1 ∈ R d , { θ n } converges to the set J ∗ almost surely, i.e.,
Revised: If Assumptions 2.1, 2.3 hold, and Assumption 2.2 holds with r θ ∗ > 1 , then for any 0 < ϵ 0 < min { 1 / 2 cM 0 , 1 / 4 cK 0 (1 − p 0 ) } , and for any initialization θ 1 ∈ R d , { θ n } converges to the set J ∗ almost surely, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where d ( x, J ∗ ) =
Revised: where d ( x, J ∗ ) =

Original: inf y {
Revised: inf y {

Original: ∥ x − y ∥ , y ∈
Revised: ∥ x − y ∥ , y ∈

Original: J ∗ } denotes the distance between point x and set J ∗ .
Revised: J ∗ } denotes the distance between point x and set J ∗ .

Original: Meanwhile the value of the loss function converges to 0 almost surely, i.e.,
Revised: Meanwhile the value of the loss function converges to 0 almost surely, i.e.,

Original: [Equation]
Revised: [Equation]

Original: For each main result, we provide a proof sketch to illustrate our idea in deriving the result.
Revised: For each main result, we provide a proof sketch to illustrate our idea in deriving the result.

Original: A rigorous argument is deferred to the appendix.
Revised: A rigorous argument is deferred to the appendix.

Original: Proof sketch.
Revised: Proof sketch.

Original: Our proof mainly relies on two techniques.
Revised: Our proof mainly relies on two techniques.

Original: The first technique is the Lyapunov method.
Revised: The first technique is the Lyapunov method.

Original: It transfers the convergence of a high dimension vector θ n to a one dimensional Lyapunov function R ( θ n ) .
Revised: It transfers the convergence of a high dimension vector θ n to a one dimensional Lyapunov function R ( θ n ) .

Original: The second technique is to use the idea of Markov process.
Revised: The second technique is to use the idea of Markov process.

Original: We sketch these two steps and an additional step as follows.
Revised: We sketch these two steps and an additional step as follows.

Original: Step 1 :
Revised: Step 1 :

Original: In this step, we aim to prove that there exists at least one bounded set S 0 such that there is no limit point of { θ n } is in it almost surely.
Revised: In this step, we aim to prove that there exists at least one bounded set S 0 such that there is no limit point of { θ n } is in it almost surely.

Original: Through the Borel–Cantelli Lemma, it amounts to proving
Revised: Through the Borel–Cantelli Lemma, it amounts to proving

Original: [Equation]
Revised: [Equation]

Original: In order to prove Equation (6), we use the Lyapunov method, constructing a Lyapunov function
Revised: In order to prove Equation (6), we use the Lyapunov method, constructing a Lyapunov function

Original: R ( θ ) which holds a unique zero R ( θ ∗ )
Revised: R ( θ ) which holds a unique zero R ( θ ∗ )

Original: = 0 and an open set ˆ S 0 which include θ ∗ (exact forms of
Revised: = 0 and an open set ˆ S 0 which include θ ∗ (exact forms of

Original: R ( θ ) and ˆ S 0 are provided in the appendix).
Revised: R ( θ ) and ˆ S 0 are provided in the appendix).

Original: We assign I n as the characteristic function of the event { θ n ∈ ˆ S 0 } .
Revised: We assign I n as the characteristic function of the event { θ n ∈ ˆ S 0 } .

Original: Then we obtain the inequality
Revised: Then we obtain the inequality

Original: [Equation]
Revised: [Equation]

Original: where u n is defined in (12) with (cid:80) + ∞ n =1 E ( u n ) < + ∞ .
Revised: where u n is defined in (12) with (cid:80) + ∞ n =1 E ( u n ) < + ∞ .

Original: Summing up Equation (6) yields
Revised: Summing up Equation (6) yields

Original: [Equation]
Revised: [Equation]

Original: Subsequently we could construct S 0 :
Revised: Subsequently we could construct S 0 :

Original: = ˆ S 0 /U ( θ ∗ , δ ′ 0 ) , for some small enough δ ′ 0 , to make (cid:80) + ∞ 
Revised: = ˆ S 0 /U ( θ ∗ , δ ′ 0 ) , for some small enough δ ′ 0 , to make (cid:80) + ∞ n =

Original: [Equation]
Revised: [Equation]

Original: R
Revised: E

Original: [Equation]
Revised: [Equation]

Original: θ n ) (cid:1) < + ∞ .
Revised: < + ∞ .

Original: Then, as whenever θ n ∈ ˆ S 

Revised: Then, as whenever θ n ∈ ˆ S

Original: /U ( θ ∗ , δ ′ 0 )
Revised: /U ( θ ∗ , δ ′ 0 )

Original: we have R 2 r r +1 ( θ ) >
Revised: we have R 2 r r +1 ( θ ) >

Original: ˜ ϵ , we have
Revised: ˜ ϵ , we have

Original: [Equation]
Revised: [Equation]

Original: As such we conclude Equation (6), and through the Borel–Cantelli Lemma, we know that there is no limit point in S 0 almost surely.
Revised: As such we conclude Equation (6), and through the Borel–Cantelli Lemma, we know that there is no limit point in S 0 almost surely.

Original: Step 2 :
Revised: Step 2 :

Original: In this step, we aim to prove that for any bounded set S that has no intersection with J ∗ , there is no limit point in it.
Revised: In this step, we aim to prove that for any bounded set S that has no intersection with J ∗ , there is no limit point in it.

Original: The way we prove it is different for the two types of noise (2) and (5).
Revised: The way we prove it is different for the two types of noise (2) and (5).

Original: Handling the sampling noise with global stable guarantee (5) is relatively simple.
Revised: Handling the sampling noise with global stable guarantee (5) is relatively simple.

Original: The Gaussian noise of (5) guarantees that it forms an irreducible Markov process.
Revised: The Gaussian noise of (5) guarantees that it forms an irreducible Markov process.

Original: Then using the property of the irreducible
Revised: Then using the property of the irreducible

Original: Markov process directly will prove the statement.
Revised: Markov process directly will prove the statement.

Original: For (2), the situation becomes complicated where an argument of the regular sampling noise does not deduce an irreducible Markov process.
Revised: For (2), the situation becomes complicated where an argument of the regular sampling noise does not deduce an irreducible Markov process.

Original: We prove it using a delicate argument.
Revised: We prove it using a delicate argument.

Original: We first prove that a max positive bounded invariant set D must hold its boundary set ∂D
Revised: We first prove that a max positive bounded invariant set D must hold its boundary set ∂D

Original: ∩ J ∗ ̸ = ∅ , and every trajectory started from this set must almost surely converge to some global optimum.
Revised: ∩ J ∗ ̸ = ∅ , and every trajectory started from this set must almost surely converge to some global optimum.

Original: Here a set is max positive invariant if any trajectories started in S 0 will not escape S 0 and for any points θ ′′ / ∈ J ∗ ∪ D, θ ′′ ∪ D is not a positive invariant set.
Revised: Here a set is max positive invariant if any trajectories started in S 0 will not escape S 0 and for any points θ ′′ / ∈ J ∗ ∪ D, θ ′′ ∪ D is not a positive invariant set.

Original: That means, for any point either almost every trajectory started with it converges to J ∗ , or it holds a positive probability transfer to S 0 .
Revised: That means, for any point either almost every trajectory started with it converges to J ∗ , or it holds a positive probability transfer to S 0 .

Original: For the first situation, this statement is satisfied.
Revised: For the first situation, this statement is satisfied.

Original: For the second situation, we can make a small enough positive measure set, such that for any θ ∈ S, there exists a δ ′ 0 , and some large enough k , P ( θ n + k ∈ S 

Revised: For the second situation, we can make a small enough positive measure set, such that for any θ ∈ S, there exists a δ ′ 0 , and some large enough k , P ( θ n + k ∈ S

Original: | θ n = θ ) >
Revised: | θ n = θ ) >

Original: υ .
Revised: υ .

Original: Then we can get as desired
Revised: Then we can get as desired

Original: [Equation]
Revised: [Equation]

Original: Step 3 :
Revised: Step 3 :

Original: In the previous step we actually proved that almost surely either θ n → J ∗ or θ n → + ∞ .
Revised: In the previous step we actually proved that almost surely either θ n → J ∗ or θ n → + ∞ .

Original: Through the Kolmogorov 0 - 1 law, we know { θ n converges } is a tail event.
Revised: Through the Kolmogorov 0 - 1 law, we know { θ n converges } is a tail event.

Original: As such, P ( θ n → J ∗ ) ∈ { 0 , 1 } .
Revised: As such, P ( θ n → J ∗ ) ∈ { 0 , 1 } .

Original: Meanwhile as P ( θ n → ∞ ) = 1 is impossible, P ( θ n → J ∗ ) could only take 1 .
Revised: Meanwhile as P ( θ n → ∞ ) = 1 is impossible, P ( θ n → J ∗ ) could only take 1 .

Original: In step 3, we suspect that P ( θ n → ∞ ) = 1 is indeed impossible, even without the assumption lim inf θ →∞ ∥
Revised: In step 3, we suspect that P ( θ n → ∞ ) = 1 is indeed impossible, even without the assumption lim inf θ →∞ ∥

Original: ˜ ∇ g ( θ ) ∥ > 2 c max {√ M 0 , a/ 2 cM 0 } .
Revised: ˜ ∇ g ( θ ) ∥ > 2 c max {√ M 0 , a/ 2 cM 0 } .

Original: In fact, as long as θ n converges to J ∗ for any initialization θ 1 in some neighboring domain of the optimum, it converges for all initialization.
Revised: In fact, as long as θ n converges to J ∗ for any initialization θ 1 in some neighboring domain of the optimum, it converges for all initialization.

Original: This is because for every initialization it either converges to the optimum or it has a positive probability to transfer to an arbitrary set with a positive measure.
Revised: This is because for every initialization it either converges to the optimum or it has a positive probability to transfer to an arbitrary set with a positive measure.

Original: As the neighboring domain could be arbitrarily small, it is likely to exist.
Revised: As the neighboring domain could be arbitrarily small, it is likely to exist.

Original: Theorem 3.2.
Revised: Theorem 3.2.

Original: Consider the SGD iteration in Equation (5), or alternatively Equation (2) with J ∗ =
Revised: Consider the SGD iteration in Equation (5), or alternatively Equation (2) with J ∗ =

Original: J ∗∗ , and the MSE loss function (1).
Revised: J ∗∗ , and the MSE loss function (1).

Original: If Assumptions 2.1, 2.3 hold, and Assumption 2.2 holds with r θ ∗ = 1 , then for any 0 < ϵ 0 < min { 1 / 2 cM 0 , α θ ∗ / 2(2 − p 0 ) β 2 θ ∗ , a/ 4 c ( K 0 (1 − p 0 )+ a ) } , where normal sampling noise 2 can be seen as p 0 = 0 , and for any θ 1 ∈ R d , θ n converges to J ∗ almost surely, i.e.,
Revised: If Assumptions 2.1, 2.3 hold, and Assumption 2.2 holds with r θ ∗ = 1 , then for any 0 < ϵ 0 < min { 1 / 2 cM 0 , α θ ∗ / 2(2 − p 0 ) β 2 θ ∗ , 1 / 4 cK 0 (1 − p 0 ) } , where normal sampling noise 2 can be seen as p 0 = 0 , and for any θ 1 ∈ R d , θ n converges to J ∗ almost surely, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where d ( x, J ∗ ) =
Revised: where d ( x, J ∗ ) =

Original: inf y {
Revised: inf y {

Original: ∥ x − y ∥ , y ∈
Revised: ∥ x − y ∥ , y ∈

Original: J ∗ } denotes the distance between point x and set J ∗ .
Revised: J ∗ } denotes the distance between point x and set J ∗ .

Original: Meanwhile the value of the loss function converges to 0 almost surely, i.e.,
Revised: Meanwhile the value of the loss function converges to 0 almost surely, i.e.,

Original: [Equation]
Revised: [Equation]

Original: Proof sketch.
Revised: Proof sketch.

Original: This proof will be similar to the proof of Theorem 3.1.
Revised: This proof will be similar to the proof of Theorem 3.1.

Original: The difference is when r θ ∗ =
Revised: The difference is when r θ ∗ =

Original: the convergence towards a global optimum with second-order local structure is conditional on the selection of the initial learning rate ϵ 0 .
Revised: the convergence towards a global optimum with second-order local structure is conditional on the selection of the initial learning rate ϵ 0 .

Original: The reason for this is the inequality
Revised: The reason for this is the inequality

Original: [Equation]
Revised: [Equation]

Original: holds only when the coefficient α θ ∗ ϵ 0 − 2(2 − p 0 )
Revised: holds only when the coefficient α θ ∗ ϵ 0 − 2(2 − p 0 )

Original: ϵ 20 β 2 θ ∗ > 0 .
Revised: ϵ 20 β 2 θ ∗ > 0 .

Original: By setting ϵ 0 as the theorem the inequality and other arguments remain valid.
Revised: By setting ϵ 0 as the theorem the inequality and other arguments remain valid.

Original: This proof also agrees with our intuition that SGD converges to a sharper global optimum not as easy as a flat one ( r θ ∗ > 1 ).
Revised: This proof also agrees with our intuition that SGD converges to a sharper global optimum not as easy as a flat one ( r θ ∗ > 1 ).

Original: Recall the second question raised in SGD was conjecturing if SGD tends to choose the flat minima (and so as to enjoy a better generalization).
Revised: Recall the second question raised in SGD was conjecturing if SGD tends to choose the flat minima (and so as to enjoy a better generalization).

Original: In the end of the above proof we find that SGD converges to a sharper global optimum not as easy as a flat one.
Revised: In the end of the above proof we find that SGD converges to a sharper global optimum not as easy as a flat one.

Original: This observation is through positive results only, though.
Revised: This observation is through positive results only, though.

Original: We wonder if the converse is also true, that is, if a global minimum is not flat, then
Revised: We wonder if the converse is also true, that is, if a global minimum is not flat, then

Original: SGD is unlikely to converge to that.
Revised: SGD is unlikely to converge to that.

Original: In the below theorem we answer the converse affirmatively.
Revised: In the below theorem we answer the converse affirmatively.

Original: Is is proved that if ϵ 0 is large enough, then the iteration will almost surely not converge to this optimum.
Revised: Is is proved that if ϵ 0 is large enough, then the iteration will almost surely not converge to this optimum.

Original: Theorem 3.3.
Revised: Theorem 3.3.

Original: Consider the SGD iteration in Equation (5), or alternatively Equation (2) with J ∗ =
Revised: Consider the SGD iteration in Equation (5), or alternatively Equation (2) with J ∗ =

Original: J ∗∗ , and the MSE loss function (1).
Revised: J ∗∗ , and the MSE loss function (1).

Original: If Assumptions 2.1, 2.3 hold, and Assumption 2.2 holds with r θ ∗ = 1 , then for any θ 1 ∈ R d , if ϵ 0 > β θ ∗ / 2(2 − p 0 )
Revised: If Assumptions 2.1, 2.3 hold, and Assumption 2.2 holds with r θ ∗ = 1 , then for any θ 1 ∈ R d , if ϵ 0 > β θ ∗ / 2(2 − p 0 )

Original: α 2 θ ∗ , where normal sampling noise 2 can be seen as p 0 = 0 , the probability that θ n converges to θ ∗ is 0 , i.e.,
Revised: α 2 θ ∗ , where normal sampling noise 2 can be seen as p 0 = 0 , the probability that θ n converges to θ ∗ is 0 , i.e.,

Original: [Equation]
Revised: [Equation]

Original: Proof sketch.
Revised: Proof sketch.

Original: The main idea is to prove that if the iteration always stays in a neighboring domain of θ ∗ , then the probability that this iteration converges to θ ∗ is zero.
Revised: The main idea is to prove that if the iteration always stays in a neighboring domain of θ ∗ , then the probability that this iteration converges to θ ∗ is zero.

Original: The Lyapunov method is helpful in this case.
Revised: The Lyapunov method is helpful in this case.

Original: Step 1 :
Revised: Step 1 :

Original: In this step, we aim to acquire a reverse inequality of (7).
Revised: In this step, we aim to acquire a reverse inequality of (7).

Original: We first construct a Lyapunov function R ( θ ) and a domain S 1 of θ ∗ , and an event A
Revised: We first construct a Lyapunov function R ( θ ) and a domain S 1 of θ ∗ , and an event A

Original: i,n = { θ n 0 ∈ S 1 , n 0 ∈
Revised: i,n = { θ n 0 ∈ S 1 , n 0 ∈

Original: [ i, n ] } as well its characteristic function
Revised: [ i, n ] } as well its characteristic function

Original: I i,n .
Revised: I i,n .

Original: Then we can acquire an inequality
Revised: Then we can acquire an inequality

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ζ n is defined by equation 33.
Revised: where ζ n is defined by equation 33.

Original: Notice that if (cid:0) 2(2 − p 0 )
Revised: Notice that if (cid:0) 2(2 − p 0 )

Original: ϵ 20 α 2 θ ∗ − ϵ 0 β θ ∗ (cid:1) > 0 , then this inequality will be a variant of diffusion process.
Revised: ϵ 20 α 2 θ ∗ − ϵ 0 β θ ∗ (cid:1) > 0 , then this inequality will be a variant of diffusion process.

Original: Step 2 :
Revised: Step 2 :

Original: In this step, we aim to prove when n approaches infinity, the iteration will transform a fixed part of itself out of S 1 .
Revised: In this step, we aim to prove when n approaches infinity, the iteration will transform a fixed part of itself out of S 1 .

Original: Through (9), we get
Revised: Through (9), we get

Original: [Equation]
Revised: [Equation]

Original: We know if
Revised: We know if

Original: [Equation]
Revised: [Equation]

Original: then E (cid:0)
Revised: then E (cid:0)

Original: I i,n +1 R ( θ n +1 ) (cid:1) will diverge to infinity, which is impossible to happen.
Revised: I i,n +1 R ( θ n +1 ) (cid:1) will diverge to infinity, which is impossible to happen.

Original: As such, it must hold
Revised: As such, it must hold

Original: [Equation]
Revised: [Equation]

Original: Step 3 :
Revised: Step 3 :

Original: In this step, we will show that if E ( I i, + ∞ ̸ = 0) > 0 , then I i,n R ( θ n ) will not converge to 0 almost surely.
Revised: In this step, we will show that if E ( I i, + ∞ ̸ = 0) > 0 , then I i,n R ( θ n ) will not converge to 0 almost surely.

Original: We prove it by contradiction and assume P (cid:0) lim n → + ∞ I
Revised: We prove it by contradiction and assume P (cid:0) lim n → + ∞ I

Original: i,n R ( θ n ) = 0 (cid:1) = 1 .
Revised: i,n R ( θ n ) = 0 (cid:1) = 1 .

Original: That means for any ϵ ′ 0 > 0 , P (cid:16)
Revised: That means for any ϵ ′ 0 > 0 , P (cid:16)

Original: I i,n R ( θ n ) >
Revised: I i,n R ( θ n ) >

Original: ϵ ′ 0 (cid:17) → 0 , which concludes P (cid:16)
Revised: ϵ ′ 0 (cid:17) → 0 , which concludes P (cid:16)

Original: I
Revised: I

Original: i,n R ( θ n ) ≤ ϵ ′ 0 (cid:17) → 1 .
Revised: i,n R ( θ n ) ≤ ϵ ′ 0 (cid:17) → 1 .

Original: Then
Revised: Then

Original: E [Equation] n + [Equation]
Revised: E (cid:0) R ( θ n +1 )

Original:  I i [Equation] n − I i,n +1 ) (cid:1) 
Revised: ( I i,n − I i,n +1 ) (cid:1) E

Original: [Equation]
Revised: [Equation]

Original: i
Revised: I i,n R (

Original: [Equation]
Revised: [Equation]

Original:  →[Equation]
Revised: n ) (cid:1) →

Original: [Equation]
Revised: k ′ ϵ ′ 0 .

Original: This forms a contradiction.
Revised: This forms a contradiction.

Original: Step 4 :
Revised: Step 4 :

Original: In this final step, we will prove P (cid:0) lim n → + ∞ θ n = θ ∗ (cid:1) = 0 .
Revised: In this final step, we will prove P (cid:0) lim n → + ∞ θ n = θ ∗ (cid:1) = 0 .

Original: We inspect the event { θ n → θ ∗ } .
Revised: We inspect the event { θ n → θ ∗ } .

Original: If E ( I i, + ∞ ̸ = 0) > 0 , then due to lim n → + ∞ I i,n g ( θ n ) − I i, + ∞ g ( θ n )
Revised: If E ( I i, + ∞ ̸ = 0) > 0 , then due to lim n → + ∞ I i,n g ( θ n ) − I i, + ∞ g ( θ n )

Original: = 

Revised: =

Original: a.s. , we could get
Revised: a.s. , we could get

Original: P (cid:0) lim n → + ∞ I i, + ∞ R ( θ n ) = 0 (cid:1) = 0 .
Revised: P (cid:0) lim n → + ∞ I i, + ∞ R ( θ n ) = 0 (cid:1) = 0 .

Original: Then,
Revised: Then,

Original: P (cid:0) { θ n → θ ∗ } ∩
Revised: P (cid:0) { θ n → θ ∗ } ∩

Original: A i, + ∞ (cid:1) =
Revised: A i, + ∞ (cid:1) =

Original: P (cid:0) lim n → + ∞ I i, + ∞ R ( θ n ) = 0 (cid:1) = 0 .
Revised: P (cid:0) lim n → + ∞ I i, + ∞ R ( θ n ) = 0 (cid:1) = 0 .

Original: Otherwise if E ( I i, + ∞ ̸ = 0) = 0 , we have
Revised: Otherwise if E ( I i, + ∞ ̸ = 0) = 0 , we have

Original: P (cid:0) { θ n → θ ∗ } ∩ A i, + ∞ (cid:1) ≤ E ( I i, + ∞ ̸ = 0) = 0 .
Revised: [Equation]

Original: Absolutely, we have{ θ n → θ ∗ } ⊂ (cid:40) + ∞ (cid:91) i =
Revised: Absolutely, we have

Original: A i, + ∞ (cid:41) .
Revised: [Equation]

Original: Subsequently we have
Revised: Subsequently we have

Original: P (cid:0) θ n → θ ∗ (cid:1) = P (cid:32) { θ n → θ ∗ } (cid:92) (cid:40) + ∞ (cid:91) m = A i, + ∞ (cid:41)(cid:33) = P (cid:32) + ∞ (cid:91) i = { θ n → θ ∗ } (cid:92) A i, + ∞ (cid:33) ≤ + ∞ (cid:88) i = P (cid:16) { θ n → θ ∗ } (cid:92) A i, + ∞ (cid:17) = 0 .
Revised: [Equation]

Original: As we have shown the asymptotic convergence of SGD, the natural question is how fast it converges.
Revised: As we have shown the asymptotic convergence of SGD, the natural question is how fast it converges.

Original: To provide the convergence rate, we will need a slightly stronger version of Assumption 2.2.
Revised: To provide the convergence rate, we will need a slightly stronger version of Assumption 2.2.

Original: We need, instead of just one θ ∗ , all θ ∗ , to satisfy the order r + 1 expansion.
Revised: We need, instead of just one θ ∗ , all θ ∗ , to satisfy the order r + 1 expansion.

Original: In this case, the supremum of the expansion order, among all optimum points, is denoted as ˆ r = max θ ∗ ∈ J ∗∞ r θ ∗ , where J ∗∞ :=
Revised: In this case, the supremum of the expansion order, among all optimum points, is denoted as ˆ r = max θ ∗ ∈ J ∗∞ r θ ∗ , where J ∗∞ :=

Original: { θ ∗ ∈
Revised: { θ ∗ ∈

Original: J ∗ | P ( θ n → θ ∗ ) > 0 } .
Revised: J ∗ | P ( θ n → θ ∗ ) > 0 } .

Original: Our next theorem provides the convergence rate of SGD.
Revised: Our next theorem provides the convergence rate of SGD.

Original: Theorem 3.4.
Revised: Theorem 3.4.

Original: Consider the SGD iteration in Equation (5), or alternatively Equation (2) with J ∗ =
Revised: Consider the SGD iteration in Equation (5), or alternatively Equation (2) with J ∗ =

Original: J ∗∗ , and the MSE loss function (1).
Revised: J ∗∗ , and the MSE loss function (1).

Original: If Assumptions 2.1, 2.3 hold, and the variant of Assumption 2. described immediately preceding this statement holds with order
Revised: If Assumptions 2.1, 2.3 hold, and the variant of Assumption 2. described immediately preceding this statement holds with order

Original: ˆ r + 1 , then for any θ 1 ∈ R d , θ n has an asymptotic convergence rate as
Revised: ˆ r + 1 , then for any θ 1 ∈ R d , θ n has an asymptotic convergence rate as

Original: [Equation]
Revised: [Equation]

Original: where p 0 < 1 is a constant decided by the learning rate ϵ 0 .
Revised: where p 0 < 1 is a constant decided by the learning rate ϵ 0 .

Original: Proof sketch.
Revised: Proof sketch.

Original: The proof of this theorem is based on the proof of Theorem 3.1.
Revised: The proof of this theorem is based on the proof of Theorem 3.1.

Original: We asymptotically bound of martingale difference (Lemma A.1) and with the bound apply the martingale convergence theorem.
Revised: We asymptotically bound of martingale difference (Lemma A.1) and with the bound apply the martingale convergence theorem.

Original: The asymptotic convergence rate follows.
Revised: The asymptotic convergence rate follows.

Original: As an immediate consequence of the convergence rate, the SGD algorithm could obtain an arbitrary accuracy in polynomial time.
Revised: As an immediate consequence of the convergence rate, the SGD algorithm could obtain an arbitrary accuracy in polynomial time.

Original: This validates the efficiency of SGD.
Revised: This validates the efficiency of SGD.

Original: Corollary 3.1.
Revised: Corollary 3.1.

Original: Consider the same setting as Theorem 3.4.
Revised: Consider the same setting as Theorem 3.4.

Original: For any θ 1 ∈ R d , the computational time for g ( θ n ) to reach an η accuracy is
Revised: For any θ 1 ∈ R d , the computational time for g ( θ n ) to reach an η accuracy is

Original: [Equation]
Revised: [Equation]

Original: O (cid:0) N

Original: [Equation]

Original: d

Original: [Equation]

Original: η

Original: [Equation]

Original: 12 (cid:1)

Original: [Equation]

Original: .s

Original: [Equation]

Original: , if

Original: [Equation]

Original: r > 1 , where N 0 is the mini-batch size.
Revised: where N 0 is the mini-batch size.

Original: A A PPENDIX A. C OUNTER - EXAMPLE
Revised: A A PPENDIX A. C OUNTER - EXAMPLE

Original: Claim A.1.
Revised: Claim A.1.

Original: The chain rule does not hold the Clarke subdifferential.
Revised: The chain rule does not hold the Clarke subdifferential.

Original: Proof.
Revised: Proof.

Original: For a composite nonsmooth function, the chain rule may not hold at the nonsmooth point Liu et al.
Revised: For a composite nonsmooth function, the chain rule may not hold at the nonsmooth point Liu et al.

Original: (2022b).
Revised: (2022b).

Original: We introduce an example as follows.
Revised: We introduce an example as follows.

Original: Consider
Revised: Consider

Original: [Equation]
Revised: [Equation]

Original: Let w ∗ 2 = 1 , b ∗ 1 = 0 , w ∗ 1 = 0 , b ∗ 2 = 0 , one can easily see that the SGD method will get stuck at ( w ∗ 1 , w ∗ 2 , b ∗ 1 , b ∗ 2 ) , and
Revised: Let w ∗ 2 = 1 , b ∗ 1 = 0 , w ∗ 1 = 0 , b ∗ 2 = 0 , one can easily see that the SGD method will get stuck at ( w ∗ 1 , w ∗ 2 , b ∗ 1 , b ∗ 2 ) , and

Original: [Equation]
Revised: [Equation]

Original: some small positive number ϵ
Revised: some small positive number ϵ

Original: [Equation]
Revised: [Equation]

Original: Then, observe that ( w ∗ 1 , w ∗ 2 , b ∗ 1 , b ∗ 2 ) is neither a local minimizer of equation 10.
Revised: Then, observe that ( w ∗ 1 , w ∗ 2 , b ∗ 1 , b ∗ 2 ) is neither a local minimizer of equation 10.

Original: Moreover, one can see that (1 , 2 , − 1 , − 1) is a global minimizer of equation 10, at which the function value is 0.
Revised: Moreover, one can see that (1 , 2 , − 1 , − 1) is a global minimizer of equation 10, at which the function value is 0.

Original: [Equation]
Revised: [Equation]

Original: EMMAS
Revised: EMMAS

Original: Lemma A.1.
Revised: Lemma A.1.

Original: (Theorem 4.2.13, Lei et al.
Revised: (Theorem 4.2.13, Lei et al.

Original: Consider a Martingale difference column { X n , F n } that satisfies sup n E ( ∥ X n +1 ∥ 2 |F n ) <
Revised: Consider a Martingale difference column { X n , F n } that satisfies sup n E ( ∥ X n +1 ∥ 2 |F n ) <

Original: + ∞ almost surely.
Revised: + ∞ almost surely.

Original: Then it holds that
Revised: Then it holds that

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: β k X k
Revised: β k X k

Original: = O (cid:16)(cid:112) S n
Revised: = O (cid:16)(cid:112) S n

Original: ln 12 + η ( S n + e ) (cid:17) almost surely , ∀ η > 0 , where S n = (cid:80) nk
Revised: ln 12 + η ( S n + e ) (cid:17) almost surely , ∀ η > 0 , where S n = (cid:80) nk

Original: =

Revised: =

Original: β 2 k .
Revised: β 2 k .

Original: Lemma A.2.
Revised: Lemma A.2.

Original: (Lemma 6 in Jin et al.
Revised: (Lemma 6 in Jin et al.

Original: Suppose that { X n } ∈ R d is a non-negative sequence of random variables, then (cid:80) ∞ n =

Revised: Suppose that { X n } ∈ R d is a non-negative sequence of random variables, then (cid:80) ∞ n =

Original: X n < + ∞ holds almost surely if (cid:80) ∞ n =0 E (cid:0) X n (cid:1) <
Revised: X n < + ∞ holds almost surely if (cid:80) ∞ n =0 E (cid:0) X n (cid:1) <

Original: + ∞ .
Revised: + ∞ .

Original: Lemma A.3.
Revised: Lemma A.3.

Original: (Wang et al., 2019)
Revised: (Wang et al., 2019)

Original: Suppose that { X n } ∈ R d is an L 2 martingale difference sequence, and ( X n , F n ) is an adaptive process.
Revised: Suppose that { X n } ∈ R d is an L 2 martingale difference sequence, and ( X n , F n ) is an adaptive process.

Original: Then it holds almost surely that (cid:80) ∞ k
Revised: Then it holds almost surely that (cid:80) ∞ k

Original: =

Revised: =

Original: X k < + ∞
Revised: X k < + ∞

Original: if
Revised: if

Original: [Equation]
Revised: [Equation]

Original: happens almost surely.
Revised: happens almost surely.

Original: A. P ROOF OF L EMMA A.4.
Revised: A. P ROOF OF L EMMA A.4.

Original: Lemma A.4.
Revised: Lemma A.4.

Original: Consider the SGD updates specified in equation 2 (or equation 2 with J ∗ =
Revised: Consider the SGD updates specified in equation 2 (or equation 2 with J ∗ =

Original: J ∗∗ ) and the MSE loss function equation 1.
Revised: J ∗∗ ) and the MSE loss function equation 1.

Original: If Assumptions 2.1, 2.3 hold, then for any ϵ 0 < min { 1 / 2 cM 0 , a/ 4 c ( K 0 (1 − p 0)+ a ) } , where normal sampling noise 2 can be seen as p [-
-] = 0 .
Revised: If Assumptions 2.1, 2.3 hold, then for any ϵ 0 < min { 1 / 2 cM 0 , 1 / 4 cK 0 (1 − p 0 ) } , where normal sampling noise 2 can be seen as p [-
-]0 = 0 .

Original: Then for any θ 1 ∈ R d , the probability of θ n diverge to the infinity is less than 1 ,
Revised: Then for any θ 1 ∈ R d , the probability of θ n diverge to the infinity is less than 1 ,

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e., P ( θ n → ∞ ) < 1 .
Revised: e., P ( θ n → ∞ ) < 1 .

Original: Proof.
Revised: Proof.

Original: We prove this Lemma by contradiction.
Revised: We prove this Lemma by contradiction.

Original: We first assume P ( θ n → ∞ ) = 1 , which means θ n → ∞ almost surely.
Revised: We first assume P ( θ n → ∞ ) = 1 , which means θ n → ∞ almost surely.

Original: By the Lagrange’s mean value theorem, we have
Revised: By the Lagrange’s mean value theorem, we have

Original: [Equation]
Revised: [Equation]

Original: where ζ n is a point between θ n and θ n +1 .
Revised: where ζ n is a point between θ n and θ n +1 .

Original: If ζ n is a non-smooth point
Revised: If ζ n is a non-smooth point

Original: [Equation]
Revised: [Equation]

Original: then we can find at least one point in the set of ˜ ∇ g ( θ ζ n ) .
Revised: then we can find at least one point in the set of ˜ ∇ g ( θ ζ n ) .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Revised: Through Assumption 2.3, we know that it hold E ξ n ∥ ˜ ∇ g ( θ n , ξ n ) ∥ 2 ≤ M 0 ∥

Revised: ˜ ∇ g ( θ n ) ∥ 2 when θ n → ∞ .

Original: Taking expectation over the sampling noise, we have
Revised: Then we take an expectation over the sampling noise, we have

Original: [Equation]
Revised: [Equation]

Original: Since 12 ϵ 0 − M 0 ϵ 20 > 0 , and ∥ ˜ ∇ g ( θ n ) ∥ > max { 4 c √ M 0 , a } when θ n → ∞ , we can get P ( ∥ ˜ ∇ g ( θ n ) ∥ > 2 cϵ 0 M 0  )
Revised: Since 12 ϵ 0 − cM 0 ϵ 20 > 0 , and ∥ ˜ ∇ g ( θ n ) ∥ > max { 4 c √ M 0 , 4 c (cid:112) (1 − p 0 ) K 0 } when θ n → ∞ , we can get P ( ∥ ˜ ∇ g ( θ n ) ∥ > max { 4 c √ M 0 , 4 c (cid:112) (1 − p 0 ) K 0 } )

Original: → 1 .
Revised: → 1 .

Original: This implies
Revised: This implies

Original: [Equation]
Revised: [Equation]

Revised: g

Revised: [Equation]

Revised: ∥

Revised: [Equation]

Original: With this, we have
Revised: With this, we have

Original: [Equation]
Revised: [Equation]

Original:  which is impossible.
Revised: k = E ∥ ˜ ∇ g ( θ n ) ∥ 2 → −∞ , which is impossible.

Original: We thus conclude that { θ n } can not tend to infinity almost surely, i.e., P ( θ n → ∞ ) < 1 .
Revised: We thus conclude that { θ n } can not tend to infinity almost surely, i.e., P ( θ n → ∞ ) < 1 .

Original: A. P ROOF OF T HEOREM 3.1.
Revised: A. P ROOF OF T HEOREM 3.1.

Original: Proof.
Revised: Proof.

Original: For convenience, we abbreviate r θ ∗ :
Revised: For convenience, we abbreviate r θ ∗ :

Original: = r .
Revised: = r .

Original: Then we let l 0 :
Revised: Then we let l 0 :

Original: = min (cid:40)(cid:18) 
Revised: = min (cid:40)(cid:18) β θ ∗ +

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  and construct a function
Revised: , and construct a function

Original: [Equation]
Revised: [Equation]

Original: where ˆ k ( ∥ θ − θ ∗ ∥ ) is the smooth connection between ∥ θ − θ ∗ ∥ ( ∥ θ − θ ∗ ∥ > ¯ K 0 ) and ∥ θ − θ ∗ ∥ r +1 ( ∥ θ − θ ∗ ∥ ≤ max { 1 , δ θ ∗ } ) .
Revised: where ˆ k ( ∥ θ − θ ∗ ∥ ) is the smooth connection between ∥ θ − θ ∗ ∥ ( ∥ θ − θ ∗ ∥ > ¯ K 0 ) and ∥ θ − θ ∗ ∥ r +1 ( ∥ θ − θ ∗ ∥ ≤ max { 1 , δ θ ∗ } ) .

Original: Then through choosing feasible ˆ k ( θ − θ ∗ ) and ˆ K 0 , we can ensure that the Hessian matrix of R ( θ ) is bounded in R d .
Revised: Then through choosing feasible ˆ k ( θ − θ ∗ ) and ˆ K 0 , we can ensure that the Hessian matrix of R ( θ ) is bounded in R d .

Original: Let the upper bound of the Hessian matrix be r ( r + 1) , i.e., x T H θθ x ≤ r ( r + 1)
Revised: Let the upper bound of the Hessian matrix be r ( r + 1) , i.e., x T H θθ x ≤ r ( r + 1)

Original: ∥ x ∥ 2 ( ∀ x ∈ R d , θ ∈ R d ) .
Revised: ∥ x ∥ [Equation] ∀ x ∈ R d , θ ∈ R d ) .

Original: Next, we construct a set
Revised: Next, we construct a set

Original: [Equation]
Revised: [Equation]

Original: We also define event A ( l 0 ) n
Revised: We also define event A ( l 0 ) n

Original: = { θ n ∈ S
Revised: = { θ n ∈ S

Original: ( l 0 ) } and the characteristic function I ( l 0 ) n .
Revised: ( l 0 ) } and the characteristic function I ( l 0 ) n .

Original: Through the
Revised: Through the

Original: Lagrange’s mean value theorem, we obtain
Revised: Lagrange’s mean value theorem, we obtain

Original: [Equation]
Revised: [Equation]

Original: where θ ζ n ∈ [ θ n +1 , θ n ] .
Revised: where θ ζ n ∈ [ θ n +1 , θ n ] .

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: , and thus
Revised: and thus

Original: [Equation]
Revised: [Equation]

Original: Hence, for any θ ∈ { θ |∥ θ − θ n ∥ ≤ max { 1 , δ θ ∗ }} we have
Revised: Hence, for any θ ∈ { θ |∥ θ − θ n ∥ ≤ max { 1 , δ θ ∗ }} we have

Original: [Equation]
Revised: [Equation]

Original: Moreover, if ∥ θ ξ n − θ n ∥ < max { 1 , δ θ ∗ } , we also have 
Revised: Moreover, if ∥ θ ξ n − θ n ∥ < max { 1 , δ θ ∗ } , we also have ∥∇ R [Equation] θ [Equation]

Revised: n

Original: [Equation]
Revised: [Equation]

Revised: ∇ R

Revised: [Equation]

Revised: θ n

Revised: [Equation]

Revised: ≤ r

Revised: [Equation]

Revised: r + 1) ∥ θ

Revised: [Equation]

Original: +1 − θ n ∥ r , and if ∥ θ ξ n − θ n ∥ ≥ max { l 0 , 1 } , we have
Revised: +1 − θ n ∥ r , and if ∥ θ ξ n − θ n ∥ ≥ max { l 0 , 1 } , we have

Original: [Equation]
Revised: [Equation]

Original: With this, we have
Revised: With this, we have

Original: [Equation]
Revised: [Equation]

Original: Taking expectation of equation
Revised: Taking expectation of equation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Revised: [Equation]

Revised: T ˜ ∇ g ( θ n , ξ n ) (cid:1) + I ( l 0 ) n ϵ 0 E (cid:0) ∇ R ( θ n ) T (cid:112) min { g ( θ n ) , K 0 } τ n N n (cid:1)(cid:12)(cid:12) F n (cid:1)(cid:17) =

Revised: ϵ 0 E (cid:16) I ( l 0 ) n

Revised: E

Original: [Equation]
Revised: [Equation]

Original: Define ˆ S to be the set of θ ′ , such that g ( θ ′ ) is not smooth.
Revised: Define ˆ S to be the set of θ ′ , such that g ( θ ′ ) is not smooth.

Original: Then with Assumption 2.1, we have
Revised: Then with Assumption 2.1, we have

Original: E θ n ∈ ˆ S ( h ( θ n ))
Revised: E θ n ∈ ˆ S ( h ( θ n ))

Original: = 0 , where h is an arbitrary measurable function.
Revised: = 0 , where h is an arbitrary measurable function.

Original: Hence, when θ n ∈ R d / ˆ S ,
Revised: Hence, when θ n ∈ R d / ˆ S ,

Original: [Equation]
Revised: [Equation]

Original: Therefore [Equation] we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: and through Assumption 2.2, we get
Revised: and through Assumption 2.2, we get

Original: [Equation]
Revised: [Equation]

Original: where G ( r θ ∗ )is defined in Claim 2.1, and results of equation 2 can be seen the situation which p 0 = 0 .
Revised: where G ( r θ ∗ )is defined in Claim 2. [Equation] and results of equation 2 can be seen the situation which p 0 = 0 .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Due to θ n ∈ S ( l 0 ) , we know
Revised: Due to θ n ∈ S ( l 0 ) [Equation] we know

Original: [Equation]
Revised: [Equation]

Original: That means
Revised: That means

Original: [Equation]
Revised: [Equation]

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: For the term E (cid:0) ( I ( l 0 ) n [Equation] +1 ) (cid:1) , we observe that
Revised: For the term E (cid:0) ( I ( l 0 ) n − I ( l 0 ) n +1 ) R ( θ n +1 ) (cid:1) , we observe that

Original: E
Revised: E

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 12) and
Revised: 12) and

Original: [Equation]
Revised: [Equation]

Original: Taking these into equation 
Revised: Taking these into equation 12, we obtain

Original: [Equation]
Revised: [Equation]

Original: we obtain

Original: [Equation]
Revised: [Equation]

Original: Taking equation 13 into equation
Revised: Taking equation 13 into equation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Summing equation 14 over n , we have
Revised: Summing equation 14 over n , we have

Original: [Equation]
Revised: [Equation]

Original: Rearranging the equation, we have
Revised: Rearranging the equation, we have

Original: [Equation]
Revised: [Equation]

Original: Next we construct a subset of S ( l 0 ) as
Revised: Next we construct a subset of S ( l 0 ) as

Original: [Equation]
Revised: [Equation]

Original: Define event
Revised: Define event

Original: A

Original: [Equation]
Revised: [Equation]

Original: and the characteristic function be I ( δ 0 ,l 0 ) n .
Revised: and the characteristic function be I ( δ 0 ,l 0 ) n .

Original: Obviously, we have [Equation] < +
Revised: Obviously, we have n (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Let r 0 :
Revised: Let r 0 :

Original: =
Revised: =

Original: inf θ ∈ S ( δ 0 ,l 0 ) R 2 r r
Revised: inf θ ∈ S ( δ 0 ,l 0 ) R 2 r r

Original: +1 ( θ ) > 0 , we have rn (cid:88) k =
Revised: +1 ( θ ) > 0 , we have r

Original: [Equation]
Revised: [Equation]

Revised: that is

Original: [Equation]
Revised: [Equation]

Original: α θ ∗ ϵ< + ∞ , that is

Original: [Equation]
Revised: [Equation]

Original: Then we can obtain
Revised: Then we can obtain

Original: [Equation]
Revised: [Equation]

Original: Note that equation 17 means the set S ( δ 0 ,l 0 ) has no limit point of { θ n } almost surely.
Revised: Note that equation 17 means the set S ( δ 0 ,l 0 ) has no limit point of { θ n } almost surely.

Original: Then if we use the SGD update rule equation 5 Since the noise is Gaussian, any θ ∈ R d /J ∗ and for any k > 0 , there is P ( θ n + k ∈ S ( δ 0 ,l 0 ) | θ n = θ )
Revised: Then if we use the SGD update rule equation 5 Since the noise is Gaussian, any θ ∈ R d /J ∗ and for any k > 0 , there is P ( θ n + k ∈ S ( δ 0 ,l 0 ) | θ n = θ )

Original: =
Revised: =

Original: ˆ δ 0 > 0 .
Revised: ˆ δ 0 > 0 .

Original: If we use SGD update rule equation 2, for any max positive invariant set D/J ∗ , we know that there must exist a boundary set ∂D .
Revised: If we use SGD update rule equation 2, for any max positive invariant set D/J ∗ , we know that there must exist a boundary set ∂D .

Original: Moreover, ∀ θ ′ ∈ ∂D , if θ ′ ∈ R d /D , then for any mini-batch C i , we have ˜ ∇ g C i ( θ ′ )
Revised: Moreover, ∀ θ ′ ∈ ∂D , if θ ′ ∈ R d /D , then for any mini-batch C i , we have ˜ ∇ g C i ( θ ′ )

Original: = 0 .
Revised: = 0 .

Original: Otherwise we can find a sequence { θ ′′ → θ ′ , /θ ′′ ∈ D } , making the trajectories started from θ ′′ close to the trajectory started from θ ′ .
Revised: Otherwise we can find a sequence { θ ′′ → θ ′ , /θ ′′ ∈ D } , making the trajectories started from θ ′′ close to the trajectory started from θ ′ .

Original: It forms a contradiction.
Revised: It forms a contradiction.

Original: Then due to J ∗∗ = J ∗ , we know θ
Revised: Then due to J ∗∗ = J ∗ , we know θ

Original: ′ ∈
Revised: ′ ∈

Original: J ∗ .
Revised: J ∗ .

Original: That means D ∩ J ∗ ̸ = ∅ .
Revised: That means D ∩ J ∗ ̸ = ∅ .

Original: If θ ′ ∈ D, we can conclude all trajectories started from θ ′ are a subset of ∂D.
Revised: If θ ′ ∈ D, we can conclude all trajectories started from θ ′ are a subset of ∂D.

Original: On the other hand, we can conclude ∂g is a close set.
Revised: On the other hand, we can conclude ∂g is a close set.

Original: Through Heine ˘ Borel theorem , it exists a finite open cover (cid:83) Mn = 1 O n ⊃ ∂D , and every O n holding an arbitrary small diameter.
Revised: Through Heine ˘ Borel theorem , it exists a finite open cover (cid:83) M n =1 O n ⊃ ∂D , and every O n holding an arbitrary small diameter.

Original: We let θ ′ ∈
Revised: We let θ ′ ∈

Original: O 1 .
Revised: O 1 .

Original: Then we assign T n as the lone time interval of one trajectory started from θ ′ and back to T n .
Revised: Then we assign T n as the lone time interval of one trajectory started from θ ′ and back to T n .

Original: If T n → + ∞ , that means this trajectory must stay a infinity time in some O k , that means exists a global optimum in O k .
Revised: If T n → + ∞ , that means this trajectory must stay a infinity time in some O k , that means exists a global optimum in O k .

Original: Naturally, the trajectory will converge to this global optimum.
Revised: Naturally, the trajectory will converge to this global optimum.

Original: If T n is bounded, that means the trajectory will enter into O 1 infinite times.
Revised: If T n is bounded, that means the trajectory will enter into O 1 infinite times.

Original: Due to a mass of different mini-batch and the enough small diameter and f ( θ ) := P ( θ n + k ∈ R d /D | θ n = θ )
Revised: Due to a mass of different mini-batch and the enough small diameter and f ( θ ) := P ( θ n + k ∈ R d /D | θ n = θ )

Original: =
Revised: =

Original: ˆ δ 0 > 0 is a continuous function, We get P ( θ n + k ∈ R d /D | θ n ∈
Revised: ˆ δ 0 > 0 is a continuous function, We get P ( θ n + k ∈ R d /D | θ n ∈

Original: O 1 )
Revised: O 1 )

Original: =
Revised: =

Original: ˆ δ 0 > 0 , it is contradiction about D is a positive invariant set.
Revised: ˆ δ 0 > 0 , it is contradiction about D is a positive invariant set.

Original: That means for any θ ∈ R /J ∗ , either trajectories started from it will converge to some global optimum, either it has a positive probability to make sure it transfers to S ( δ 0 ,l 0 ) after k steps.
Revised: That means for any θ ∈ R /J ∗ , either trajectories started from it will converge to some global optimum, either it has a positive probability to make sure it transfers to S ( δ 0 ,l 0 ) after k steps.

Original: Then for any bounded set ˆ S 0 which has no intersection with J ∗ , we first get rid of those points which will converge to J ∗ .
Revised: Then for any bounded set ˆ S 0 which has no intersection with J ∗ , we first get rid of those points which will converge to J ∗ .

Original: We know that f ( θ ) :
Revised: We know that f ( θ ) :

Original: = P ( θ n + k ∈ S ( δ 0 ,l 0 ) | θ n = θ )
Revised: = P ( θ n + k ∈ S ( δ 0 ,l 0 ) | θ n = θ )

Original: =
Revised: =

Original: ˆ δ 0 > 0 is a continuous function.
Revised: ˆ δ 0 > 0 is a continuous function.

Original: Then we can get for any bounded closed set ˆ S 0 which satisfied ˆ S 

Revised: Then we can get for any bounded closed set ˆ S 0 which satisfied ˆ S

Original: ∩ J ∗ = ∅ , there is min θ ∈ ˆ S 0 P ( θ n + k ∈ S ( δ 0 ,l 0 )
Revised: ∩ J ∗ = ∅ , there is min θ ∈ ˆ S 0 P ( θ n + k ∈ S ( [Equation]

Original:  | θ n = θ )
Revised: 0 ,l 0 ) | θ n = θ )

Original: =
Revised: =

Original: ˆ δ 1 > 0 .
Revised: ˆ δ 1 > 0 .

Original: Then we aim to prove there is no limit point in ˆ S 0 almost surely by contradiction.
Revised: Then we aim to prove there is no limit point in ˆ S 0 almost surely by contradiction.

Original: We assume
Revised: We assume

Original: [Equation]
Revised: [Equation]

Original: Then, 
Revised: Then, +

Original: [Equation]
Revised: [Equation]

Original: Note that this is in contradiction with equation 16 and thus (cid:80) + ∞ n =1 P (cid:0) θ n ∈ ˆ S 0 (cid:1) <
Revised: Note that this is in contradiction with equation 16 and thus (cid:80) + ∞ n =1 P (cid:0) θ n ∈ ˆ S 0 (cid:1) <

Original: + ∞ .
Revised: + ∞ .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Combining equation 20 with equation 16, we can see that for any bounded set which does not include
Revised: Combining equation 20 with equation 16, we can see that for any bounded set which does not include

Original: J ∗ =
Revised: J ∗ =

Original: { θ | g ( θ ) = 0 } has no limit point almost surely.
Revised: { θ | g ( θ ) = 0 } has no limit point almost surely.

Original: This implies θ n → J ∗ or θ n → ∞ .
Revised: This implies θ n → J ∗ or θ n → ∞ .

Original: Since {{ θ n } is convergence } is a tail event.
Revised: Since {{ θ n } is convergence } is a tail event.

Original: Then by the zero-one law, we know P ( { θ n } is convergence )
Revised: Then by the zero-one law, we know P ( { θ n } is convergence )

Original: = 0 or 1 .
Revised: = 0 or 1 .

Original: That means { θ n } either converges to J ∗ almost surely, or diverges to infinity almost surely.
Revised: That means { θ n } either converges to J ∗ almost surely, or diverges to infinity almost surely.

Original: Through Lemma A.4, we know P ( θ n → ∞ ) < 1 , thus { θ n } can only converge to J ∗ almost surely.
Revised: Through Lemma A.4, we know P ( θ n → ∞ ) < 1 , thus { θ n } can only converge to J ∗ almost surely.

Original: A. P ROOF OF T HEOREM 3. 
Revised: A. P ROOF OF T HEOREM 3.

Original: Proof.
Revised: Proof.

Original: We define R ( θ ) =
Revised: We define R ( θ ) =

Original: ∥ θ − θ ∗ ∥ 2 , and a set
Revised: ∥ θ − θ ∗ ∥ 2 , and a set

Revised: S (

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: {+= { θ (cid:12)(cid:12) 
+}
Revised: {+= { θ (cid:12)(cid:12) 
+}

Revised: ≤ ∥

Revised: θ − θ ∗ ∥ <

Revised: l

Revised: [Equation]

Revised: =

Revised: δ θ ∗ } .

Original: We also define an event A ( l 0 ) n
Revised: We also define an event A ( l 0 ) n

Original: = { θ n ∈ S
Revised: = { θ n ∈ S

Original: ( l 0 ) } and the characteristic function I ( l 0 ) n .
Revised: ( l 0 ) } and the characteristic function I ( l 0 ) n .

Original: By Lagrange’s mean value theorem, we have
Revised: By Lagrange’s mean value theorem, we have

Original: [Equation]
Revised: [Equation]

Original: where θ ζ n ∈ [ θ n +1 , θ n ] .
Revised: where θ ζ n ∈ [ θ n +1 , θ n ] .

Original: Note that ∇ R ( θ ζ n ) = ∇ R ( θ n ) + ∇ R ( θ ζ n ) − ∇ R ( θ n ) , we haveI
Revised: Note that ∇ R ( θ ζ n ) = ∇ R ( θ n ) + ∇ R ( θ ζ n ) − ∇ R ( θ n ) , we have

Original: [Equation]
Revised: [Equation]

Original: n ∥ .

Original: Moreover, we also have
Revised: Moreover, we also have

Original: [Equation]
Revised: [Equation]

Original: Taking expectation of equation
Revised: Taking expectation of equation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: We define ˆ S = { θ ′ | ˜ ∇ g ( θ ) is not continue at θ ′ } .
Revised: We define ˆ S = { θ ′ | ˜ ∇ g ( θ ) is not continue at θ ′ } .

Original: Then through Assumption 2.1, and note that E θ n ∈ ˆ S ( h ( θ n ))
Revised: Then through Assumption 2.1, and note that E θ n ∈ ˆ S ( h ( θ n ))

Original: = 0 , where h is an arbitrary measurable function, we have that the following when θ n ∈ R d / ˆ S .
Revised: = 0 , where h is an arbitrary measurable function, we have that the following when θ n ∈ R d / ˆ S .

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: and through Assumption 2.
Revised: and through Assumption 2.

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: , where the situation of equation 2 can be seen as p 0 = 0 .
Revised: , where the situation of equation 2 can be seen as p 0 = 0 .

Original: Then we have
Revised: Then we have

Original: E (cid:0) I ( l 0 ) n +1 R ( θ n +1 ) (cid:1) − E (cid:0) I ( l 0 ) n R ( θ n ) (cid:1) ≤ − c θ ∗ ϵ 0 E (cid:0) I ( l 0 ) n R ( θ n ) (cid:1) + 2(2 − p 0 )
Revised: E (cid:0) I ( l 0 ) n +1 R ( θ n +1 ) (cid:1) − E (cid:0) I ( l 0 ) n R ( θ n ) (cid:1) ≤ − c θ ∗ ϵ 0 E (cid:0) I ( l 0 ) n R ( θ n ) (cid:1) + 2(2 − p 0 )

Original: ϵ 20 β 2 θ ∗ E (cid:0) I
Revised: ϵ 20 β 2 θ ∗ E (cid:0) I

Original: ( l 0 ) n R ( θ n ) (cid:1) − E (cid:0) ( I ( l 0 ) n − I ( l 0 ) n +1 )
Revised: ( l 0 ) n R ( θ n ) (cid:1) − E (cid:0) ( I ( l 0 ) n − I ( l 0 ) n +1 )

Original: R ( θ n +1 ) (cid:1) , and
Revised: R ( θ n +1 ) (cid:1) , and

Original: E (cid:0) I ( l 0 ) n +1 R ( θ n +1 ) (cid:1) − E (cid:0) I ( l 0 ) n R ( θ n ) (cid:1) ≤ − (cid:0) α θ ∗ ϵ 0 − 2(2 − p 0 )
Revised: E (cid:0) I ( l 0 ) n +1 R ( θ n +1 ) (cid:1) − E (cid:0) I ( l 0 ) n R ( θ n ) (cid:1) ≤ − (cid:0) α θ ∗ ϵ 0 − 2(2 − p 0 )

Original: ϵ 20 β 2 θ ∗ (cid:1) E (cid:0)
Revised: ϵ 20 β 2 θ ∗ (cid:1) E (cid:0)

Original: I
Revised: I

Original: ( l 0 ) n R ( θ n ) (cid:1) − E (cid:0) ( I ( l 0 ) n − I ( l 0 ) n +1 )
Revised: ( l 0 ) n R ( θ n ) (cid:1) − E (cid:0) ( I ( l 0 ) n − I ( l 0 ) n +1 )

Original: R ( θ n +1 ) (cid:1) .
Revised: R ( θ n +1 ) (cid:1) .

Original: For the term E (cid:0) ( I ( l 0 ) n − I ( l 0 ) n +1 ) R ( θ n +1 ) (cid:1) , we first observe that
Revised: For the term E (cid:0) ( I ( l 0 ) n − I ( l 0 ) n +1 ) R ( θ n +1 ) (cid:1) , we first observe that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Taking these into equation
Revised: Taking these into equation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Substituting equation 24 into equation 23, we get
Revised: Substituting equation 24 into equation 23, we get

Original: [Equation]
Revised: [Equation]

Original: Summing equation 25 over n , we have
Revised: Summing equation 25 over n , we have

Original: [Equation]
Revised: [Equation]

Original: As ϵ 0 < α θ ∗ / 2(2 − p 0 )
Revised: As ϵ 0 < α θ ∗ / 2(2 − p 0 )

Original: β 2 θ ∗ , we have
Revised: β 2 θ ∗ , we have

Original: [Equation]
Revised: [Equation]

Original: Next, we construct a subset of S ( l 0 ) as
Revised: Next, we construct a subset of S ( l 0 ) as

Original: [Equation]
Revised: [Equation]

Original: We also define A ( δ 0 ,l 0 ) n = { θ n ∈ S
Revised: We also define A ( δ 0 ,l 0 ) n = { θ n ∈ S

Original: ( δ 0 ,l 0 ) } and the characteristic function be I ( δ 0 ,l 0 ) n .
Revised: ( δ 0 ,l 0 ) } and the characteristic function be I ( δ 0 ,l 0 ) n .

Original: Notice that, we have
Revised: Notice that, we have

Original: [Equation]
Revised: [Equation]

Original: Denote r 0 :
Revised: Denote r 0 :

Original: = inf θ ∈ S ( δ 0 ,l 0 ) R ( θ ) > 0 , then r
Revised: = inf θ ∈ S ( δ 0 ,l 0 ) R ( θ ) > 0 , then r

Original: [Equation]
Revised: [Equation]

Original: that is + ∞ (cid:88) k
Revised: that is + ∞ (cid:88) k

Original: =
Revised: =

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: With this
Revised: With this

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: We remark thatequation 28 implies the set S ( δ 0 ,l 0 ) has no limit point of { θ n } almost surely.
Revised: We remark thatequation 28 implies the set S ( δ 0 ,l 0 ) has no limit point of { θ n } almost surely.

Original: Then if we use the SGD update rule equation 5, as the noise is Gaussian, for any θ ∈ R d /J ∗ and any k > 0 , there is P ( θ n + k ∈ S ( δ 0 ,l 0 ) | θ n = θ )
Revised: Then if we use the SGD update rule equation 5, as the noise is Gaussian, for any θ ∈ R d /J ∗ and any k > 0 , there is P ( θ n + k ∈ S ( δ 0 ,l 0 ) | θ n = θ )

Original: =
Revised: =

Original: ˆ δ 0 > 0 .
Revised: ˆ δ 0 > 0 .

Original: If we use SGD update rule equation 2, for any max positive invariant set D/J ∗ , we know that there must exist a boundary set ∂D .
Revised: If we use SGD update rule equation 2, for any max positive invariant set D/J ∗ , we know that there must exist a boundary set ∂D .

Original: Moreover, ∀ θ ′ ∈ ∂D , if θ ′ ∈ R d /D , then for any mini-batch C i , we have ˜ ∇ g C i ( θ ′ )
Revised: Moreover, ∀ θ ′ ∈ ∂D , if θ ′ ∈ R d /D , then for any mini-batch C i , we have ˜ ∇ g C i ( θ ′ )

Original: = 0 .
Revised: = 0 .

Original: Otherwise we can find a sequence { θ ′′ → θ ′ , /θ ′′ ∈ D } , making the trajectories started from θ ′′ close to the trajectory started from θ ′ .
Revised: Otherwise we can find a sequence { θ ′′ → θ ′ , /θ ′′ ∈ D } , making the trajectories started from θ ′′ close to the trajectory started from θ ′ .

Original: It forms a contradiction.
Revised: It forms a contradiction.

Original: Then due to J ∗∗ = J ∗ , we know θ
Revised: Then due to J ∗∗ = J ∗ , we know θ

Original: ′ ∈
Revised: ′ ∈

Original: J ∗ .
Revised: J ∗ .

Original: That means D ∩ J ∗ ̸ = ∅ .
Revised: That means D ∩ J ∗ ̸ = ∅ .

Original: If θ ′ ∈ D, we can conclude all trajectories started from θ ′ are a subset of ∂D.
Revised: If θ ′ ∈ D, we can conclude all trajectories started from θ ′ are a subset of ∂D.

Original: On the other hand, we can conclude ∂g is a close set.
Revised: On the other hand, we can conclude ∂g is a close set.

Original: Through Heine ˘ Borel theorem , it exists a finite open cover (cid:83)
Revised: Through Heine ˘ Borel theorem , it exists a finite open cover (cid:83)

Original: Mn =
Revised: Mn =

Original: 1 O n ⊃ ∂D , and every
Revised: 1 O n ⊃ ∂D , and every

Original: O n holding an arbitrary small diameter.
Revised: O n holding an arbitrary small diameter.

Original: We let θ ′ ∈
Revised: We let θ ′ ∈

Original: O 1 .
Revised: O 1 .

Original: Then we assign T n as the lone time interval of one trajectory started from θ ′ and back to T n .
Revised: Then we assign T n as the lone time interval of one trajectory started from θ ′ and back to T n .

Original: If T n → + ∞ , that means this trajectory must stay a infinity time in some O k , that means exists a global optimum in O k .
Revised: If T n → + ∞ , that means this trajectory must stay a infinity time in some O k , that means exists a global optimum in O k .

Original: Naturally, the trajectory will converge to this global optimum.
Revised: Naturally, the trajectory will converge to this global optimum.

Original: If T n is bounded, that means the trajectory will enter into O 1 infinite times.
Revised: If T n is bounded, that means the trajectory will enter into O 1 infinite times.

Original: Due to a mass of different mini-batch and the enough small diameter and f ( θ ) := P ( θ n + k ∈ R d /D | θ n = θ )
Revised: Due to a mass of different mini-batch and the enough small diameter and f ( θ ) := P ( θ n + k ∈ R d /D | θ n = θ )

Original: =
Revised: =

Original: ˆ δ 0 > 0 is a continuous function, We get P ( θ n + k ∈
Revised: ˆ δ 0 > 0 is a continuous function, We get P ( θ n + k ∈

Original: R d /D | θ n ∈ O 1 )
Revised: R d /D | θ n ∈ O 1 )

Original: =
Revised: =

Original: ˆ δ 0 > 0 , it is contradiction about D is a positive invariant set.
Revised: ˆ δ 0 > 0 , it is contradiction about D is a positive invariant set.

Original: That means for any θ ∈ R /J ∗ , either trajectories started from it will converge to some global optimum, either it has a positive probability to make sure it transfers to S ( δ 0 ,l 0 ) after k steps.
Revised: That means for any θ ∈ R /J ∗ , either trajectories started from it will converge to some global optimum, either it has a positive probability to make sure it transfers to S ( δ 0 ,l 0 ) after k steps.

Original: Then for any bounded set ˆ S 0 which has no intersection with J ∗ , we first get rid of those points which will converge to J ∗ .
Revised: Then for any bounded set ˆ S 0 which has no intersection with J ∗ , we first get rid of those points which will converge to J ∗ .

Original: We know that f ( θ ) :
Revised: We know that f ( θ ) :

Original: = P ( θ n + k ∈ S ( δ 0 ,l 0 ) | θ n = θ )
Revised: = P ( θ n + k ∈ S ( δ 0 ,l 0 ) | θ n = θ )

Original: =
Revised: =

Original: ˆ δ 0 > 0 is a continuous function.
Revised: ˆ δ 0 > 0 is a continuous function.

Original: Then we can get for any bounded closed set ˆ S 0 which satisfied ˆ S 

Revised: Then we can get for any bounded closed set ˆ S 0 which satisfied ˆ S

Original: ∩ J ∗ = ∅ , there is min θ ∈
Revised: ∩ J ∗ = ∅ , there is min θ ∈

Original: ˆ S 0 P ( θ n + k ∈
Revised: ˆ S 0 P ( θ n + k ∈

Original: S ( δ 0 ,l 0 ) | θ n = θ ) =
Revised: S ( δ 0 ,l 0 ) | θ n = θ ) =

Original: ˆ δ 1 > 0 .
Revised: ˆ δ 1 > 0 .

Original: Then we aim to prove there is no limit point in ˆ S 0 almost surely by contradiction.
Revised: Then we aim to prove there is no limit point in ˆ S 0 almost surely by contradiction.

Original: We assume
Revised: We assume

Original: [Equation]
Revised: [Equation]

Original: Then we can get +
Revised: Then we can get +

Original: [Equation]
Revised: [Equation]

Original: This is contradiction with equation 27, which implies
Revised: This is contradiction with equation 27, which implies

Original: [Equation]
Revised: [Equation]

Original: Hence, we can obtain
Revised: Hence, we can obtain

Original: [Equation]
Revised: [Equation]

Original: Combining equation 29 with equation 27, for any bounded set which does not include J ∗
Revised: Combining equation 29 with equation 27, for any bounded set which does not include J ∗

Original: = { θ | g ( θ ) = 0 } , we can say that it has no limit point almost surely.
Revised: = { θ | g ( θ ) = 0 } , we can say that it has no limit point almost surely.

Original: That means θ n → J ∗ or θ n → ∞ almost surely.
Revised: That means θ n → J ∗ or θ n → ∞ almost surely.

Original: We know the event { θ n is convergence } is a tail event.
Revised: We know the event { θ n is convergence } is a tail event.

Original: By zero-one law, we have
Revised: By zero-one law, we have

Original: P ( { θ n } is convergence )
Revised: P ( { θ n } is convergence )

Original: = 0 or 1 .
Revised: = 0 or 1 .

Original: That means { g ( θ n ) } either converges to J ∗ almost surely, or diverges to infinity almost surely.
Revised: That means { g ( θ n ) } either converges to J ∗ almost surely, or diverges to infinity almost surely.

Original: Through Lemma A.4, we know P ( θ n → ∞ ) < 1 .
Revised: Through Lemma A.4, we know P ( θ n → ∞ ) < 1 .

Original: That proves { θ n } can only converge to J ∗ almost surely.
Revised: That proves { θ n } can only converge to J ∗ almost surely.

Original: A. P ROOF OF T HEOREM 3. 
Revised: A. P ROOF OF T HEOREM 3.

Original: First we construct a function R ( θ ) =
Revised: First we construct a function R ( θ ) =

Original: ∥ θ − θ ∗ ∥ 2 .
Revised: ∥ θ − θ ∗ ∥ 2 .

Original: We can get that
Revised: We can get that

Original: [Equation]
Revised: [Equation]

Original: For the term 2( θ n − θ ∗ ) T (cid:0) ϵ 

Revised: For the term 2( θ n − θ ∗ ) T (cid:0) ϵ

Original: ˜ ∇ g ( θ n , ξ n ) + ϵ 0 (cid:112) min { g ( θ n ) , K 0 } τ n N n (cid:1) , we use the following transformation:
Revised: ˜ ∇ g ( θ n , ξ n ) + ϵ 0 (cid:112) min { g ( θ n ) , K 0 } τ n N n (cid:1) , we use the following transformation:

Original: [Equation]
Revised: [Equation]

Original: For the term (cid:13)(cid:13)
Revised: For the term (cid:13)(cid:13)

Original: ϵ 0 ˜ ∇ g ( θ n , ξ n ) + ϵ 0 (cid:112) min { g ( θ n ) , K 0 } τ n N n (cid:13)(cid:13) 2 , we can obtain (cid:13)(cid:13)
Revised: ϵ 0 ˜ ∇ g ( θ n , ξ n ) + ϵ 0 (cid:112) min { g ( θ n ) , K 0 } τ n N n (cid:13)(cid:13) 2 , we can obtain (cid:13)(cid:13)

Original: ϵ 0 ˜ ∇ g ( θ n , ξ n ) + ϵ 0 (cid:112) min { g ( θ n ) , K 0 } τ n N n (cid:13)(cid:13)= ϵ 20 (cid:13)(cid:13)
Revised: ϵ 0 ˜ ∇ g ( θ n , ξ n ) + ϵ 0 (cid:112) min { g ( θ n ) , K 0 } τ n N n (cid:13)(cid:13)= ϵ 20 (cid:13)(cid:13)

Original: ˜ ∇ g ( θ n , ξ n ) (cid:13) (cid:13) 

Revised: ˜ ∇ g ( θ n , ξ n ) (cid:13) (cid:13)

Original: + 2 ϵ 20 τ n (cid:112) min { g ( θ n ) , K 0 } ˜
Revised: + 2 ϵ 20 τ n (cid:112) min { g ( θ n ) , K 0 } ˜

Original: [Equation]
Revised: [Equation]

Original: g ( θ n , ξ n ) T N n + ϵ 20 τ 2 n N 2 n min { g ( θ n ) , K 0 } =
Revised: g ( θ n , ξ n ) T N n + ϵ 20 τ 2 n N 2 n min { g ( θ n ) , K 0 } =

Original: ϵ 20 E (cid:16)(cid:13)(cid:13)
Revised: ϵ 20 E (cid:16)(cid:13)(cid:13)

Original: ˜ ∇ g ( θ n , ξ n ) (cid:13)(cid:13) 2 (cid:12)(cid:12) F n (cid:17) + ϵ 20 p 

Revised: ˜ ∇ g ( θ n , ξ n ) (cid:13)(cid:13) 2 (cid:12)(cid:12) F n (cid:17) + ϵ 20 p

Original: min
Revised: min

Original: g ( θ n ) , K 0 } + ϵ 20 (cid:13)(cid:13)
Revised: g ( θ n ) , K 0 } + ϵ 20 (cid:13)(cid:13)

Original: ˜ ∇ g ( θ n , ξ n ) (cid:13)(cid:13)− ϵ 20 E (cid:16)(cid:13)(cid:13)
Revised: ˜ ∇ g ( θ n , ξ n ) (cid:13)(cid:13)− ϵ 20 E (cid:16)(cid:13)(cid:13)

Original: ˜ ∇ g ( θ n , ξ n ) (cid:13)(cid:13) 2 (cid:12)(cid:12) F n (cid:17) + ϵ 20 τ 2 n N 2 n min { g ( θ n ) , K 0 } − ϵ 20 p 0 min { g ( θ n ) , K 0 } + 2 ϵ 20 τ n (cid:112) min { g ( θ n ) , K 0 } ˜ ∇ g ( θ n , ξ n ) T N n ≥ ϵ 20 (cid:13)(cid:13)
Revised: ˜ ∇ g ( θ n , ξ n ) (cid:13)(cid:13) 2 (cid:12)(cid:12) F n (cid:17) + ϵ 20 τ 2 n N 2 n min { g ( θ n ) , K 0 } − ϵ 20 p 0 min { g ( θ n ) , K 0 } + 2 ϵ 20 τ n (cid:112) min { g ( θ n ) , K 0 } ˜ ∇ g ( θ n , ξ n ) T N n ≥ ϵ 20 (cid:13)(cid:13)

Original: ˜ ∇ g ( θ n ) (cid:13)(cid:13) 2 + ϵ 20 (cid:13)(cid:13)
Revised: ˜ ∇ g ( θ n ) (cid:13)(cid:13) 2 + ϵ 20 (cid:13)(cid:13)

Original: ˜ ∇ g ( θ n , ξ n ) (cid:13)(cid:13) 2 − ϵ 20 E (cid:16)(cid:13)(cid:13)
Revised: ˜ ∇ g ( θ n , ξ n ) (cid:13)(cid:13) 2 − ϵ 20 E (cid:16)(cid:13)(cid:13)

Original: ˜ ∇ g ( θ n , ξ n ) (cid:13)(cid:13) 2 (cid:12)(cid:12) F n (cid:17) + 2 ϵ 20 τ n (cid:112) min { g ( θ n ) , K 0 } ˜ ∇ g ( θ n , ξ n ) T N n .
Revised: ˜ ∇ g ( θ n , ξ n ) (cid:13)(cid:13) 2 (cid:12)(cid:12) F n (cid:17) + 2 ϵ 20 τ n (cid:112) min { g ( θ n ) , K 0 } ˜ ∇ g ( θ n , ξ n ) T N n .

Original: Then we construct a set
Revised: Then we construct a set

Original: S (ˆ l 0 ) =
Revised: S (ˆ l 0 ) =

Original: { θ |∥ θ − θ ∗ ∥ < l 0 :
Revised: { θ |∥ θ − θ ∗ ∥ < l 0 :

Original: =
Revised: =

Original: δ θ ∗ } / { θ ∗ } .
Revised: δ θ ∗ } / { θ ∗ } .

Original: We also define event A i,n = { θ n 0 ∈ S
Revised: We also define event A i,n = { θ n 0 ∈ S

Original: (ˆ l 0 ) , n 0 ∈
Revised: (ˆ l 0 ) , n 0 ∈

Original: [ i, n ] } , and its characteristic function as I i,n .
Revised: [ i, n ] } , and its characteristic function as I i,n .

Original: We substitute equation 32 and equation 31 into equation 30, and multiple I i,n , getting
Revised: We substitute equation 32 and equation 31 into equation 30, and multiple I i,n , getting

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: is a Martingale difference.
Revised: is a Martingale difference.

Original: Denote ˆ p 0 :
Revised: Denote ˆ p 0 :

Original: = (cid:0) R
Revised: = (cid:0) R

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: +1 ) − R
Revised: +1 ) − R

Original: [Equation]
Revised: [Equation]

Original: θ n ) (cid:1) ≥
Revised: θ n ) (cid:1) ≥

Original: [Equation]
Revised: [Equation]

Original: 2 −
Revised: 2 −

Original: [Equation]
Revised: [Equation]

Original: ϵ
Revised: ϵ

Original: [Equation]
Revised: [Equation]

Original: ∗ − ϵ
Revised: ∗ − ϵ

Original: [Equation]
Revised: [Equation]

Original: β θ ∗ (cid:1) , we have
Revised: β θ ∗ (cid:1) , we have

Original: I i
Revised: I i

Original: [Equation]
Revised: [Equation]

Original: Then,
Revised: Then,

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: Assuming lim sup n → + ∞
Revised: Assuming lim sup n → + ∞

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Note that this contradicted the E (cid:0)
Revised: Note that this contradicted the E (cid:0)

Original: I i,n +1 R ( θ n +1 ) (cid:1) ≤
Revised: I i,n +1 R ( θ n +1 ) (cid:1) ≤

Original: ˆ l 0 .
Revised: ˆ l 0 .

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: Define an event
Revised: Define an event

Original: A i, + ∞ := { θ n 0 ∈ S
Revised: A i, + ∞ := { θ n 0 ∈ S

Original: (ˆ l 0 ) , n 0 ≥ i } , and its characteristic function as I i, + ∞ .
Revised: (ˆ l 0 ) , n 0 ≥ i } , and its characteristic function as I i, + ∞ .

Original: We next prove P (cid:16) lim n → + ∞ I
Revised: We next prove P (cid:16) lim n → + ∞ I

Original: i, + ∞ R ( θ n ) = 0 (cid:17) = 0 .
Revised: i, + ∞ R ( θ n ) = 0 (cid:17) = 0 .

Original: We assume P (cid:16) lim n → + ∞
Revised: We assume P (cid:16) lim n → + ∞

Original: I
Revised: I

Original: i, + ∞ R ( θ n ) = 0 (cid:17) = 1 , and we can get P (cid:16) lim n → + ∞ I
Revised: i, + ∞ R ( θ n ) = 0 (cid:17) = 1 , and we can get P (cid:16) lim n → + ∞ I

Original: i,n R ( θ n ) = 0 (cid:17) = 1 .
Revised: i,n R ( θ n ) = 0 (cid:17) = 1 .

Original: That means for any ϵ ′ 0 > 0 , P (cid:16)
Revised: That means for any ϵ ′ 0 > 0 , P (cid:16)

Original: I i,n R ( θ n ) >
Revised: I i,n R ( θ n ) >

Original: ϵ ′ 0 (cid:17) → 0 , concluding P (cid:16)
Revised: ϵ ′ 0 (cid:17) → 0 , concluding P (cid:16)

Original: I i,n R ( θ n ) ≤ ϵ ′ 0 (cid:17) → 1 .
Revised: I i,n R ( θ n ) ≤ ϵ ′ 0 (cid:17) → 1 .

Original: Then we get
Revised: Then we get

Original: [Equation]
Revised: [Equation]

Original: Note that this contradicted equation 34, which implies P (cid:16) lim n → + ∞ I
Revised: Note that this contradicted equation 34, which implies P (cid:16) lim n → + ∞ I

Original: i, + ∞ R ( θ n ) = 0 (cid:17) = 0 .
Revised: i, + ∞ R ( θ n ) = 0 (cid:17) = 0 .

Original: Through inspecting the event { θ n → θ ∗ } , we can get { θ n → θ ∗ } ⊂ (cid:40) + ∞ (cid:91) i =
Revised: Through inspecting the event { θ n → θ ∗ } , we can get { θ n → θ ∗ } ⊂ (cid:40) + ∞ (cid:91) i =

Original: A i, + ∞ (cid:41) .
Revised: A i, + ∞ (cid:41) .

Original: That means
Revised: That means

Original: [Equation]
Revised: [Equation]

Original: A. P ROOF OF T HEOREM 3.4.
Revised: A. P ROOF OF T HEOREM 3.4.

Original: First we order J ∗ ∞ as { θ ∗ i } .
Revised: First we order J ∗ ∞ as { θ ∗ i } .

Original: Then Assumption 2.2 implies that ∀ θ ∗ ∈
Revised: Then Assumption 2.2 implies that ∀ θ ∗ ∈

Original: J ∗ , there is ∥ ˜ ∇ g ( θ ) ∥ > 0 ( g is smooth at θ and θ ∈ U ( θ ∗ , δ θ ∗ ) /
Revised: J ∗ , there is ∥ ˜ ∇ g ( θ ) ∥ > 0 ( g is smooth at θ and θ ∈ U ( θ ∗ , δ θ ∗ ) /

Original: { θ ∗ } ).
Revised: { θ ∗ } ).

Original: That means for any θ ∗ i ̸
Revised: That means for any θ ∗ i ̸

Original: = θ ∗ j ∈ J ∗ , there is ∥ θ ∗ i − θ ∗ j ∥ ≥ inf θ i ̸
Revised: = θ ∗ j ∈ J ∗ , there is ∥ θ ∗ i − θ ∗ j ∥ ≥ inf θ i ̸

Original: = θ j ∥ θ ∗ i − θ ∗ j ∥ :
Revised: = θ j ∥ θ ∗ i − θ ∗ j ∥ :

Original: =
Revised: =

Original: ˆ δ 0 ̸ = 0 and U ( θ ∗ i , δ θ ∗ i )
Revised: ˆ δ 0 ̸ = 0 and U ( θ ∗ i , δ θ ∗ i )

Original: ∩ U ( θ ∗ j , δ θ ∗ j ) = ∅ .
Revised: ∩ U ( θ ∗ j , δ θ ∗ j ) = ∅ .

Original: Furthermore, it means that there are at most infinite { θ ∗ i } .
Revised: Furthermore, it means that there are at most infinite { θ ∗ i } .

Original: We assign this number as m .
Revised: We assign this number as m .

Original: Due lim inf θ → + ∞ ∥
Revised: Due lim inf θ → + ∞ ∥

Original: ˜ ∇ g ∥ > 0 , we know { δ θ ∗ i } is bounded.
Revised: ˜ ∇ g ∥ > 0 , we know { δ θ ∗ i } is bounded.

Original: Then we construct a function ¯ R ( θ ) as follow:
Revised: Then we construct a function ¯ R ( θ ) as follow:

Original: [Equation]
Revised: [Equation]

Original: Then we try to prove that there exists a function ˆ R ( θ ) satisfies:
Revised: Then we try to prove that there exists a function ˆ R ( θ ) satisfies:

Original: For any θ ∈ R d , there exist H θθ such that θ T H θθ ( ˆ R ) θ ≤ (cid:0) max
Revised: For any θ ∈ R d , there exist H θθ such that θ T H θθ ( ˆ R ) θ ≤ (cid:0) max

Original: θ ∗ i
Revised: θ ∗ i

Original: [Equation]
Revised: [Equation]

Original: J ∗∞ r
Revised: J ∗∞ r

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: ( r
Revised: ( r

Original: θ ∗ i
Revised: θ ∗ i

Original: + 1) (cid:1)
Revised: + 1) (cid:1)

Original: ∥ θ ∥
Revised: ∥ θ ∥

Original: ˆ R ( θ ) =
Revised: ˆ R ( θ ) =

Original: ∥ θ − θ ∗ i ∥
Revised: ∥ θ − θ ∗ i ∥

Original: r θ ∗ i +1 , when θ near the θ ∗ i .
Revised: r θ ∗ i +1 , when θ near the θ ∗ i .

Original: ˆ R ( θ ) is bounded.
Revised: ˆ R ( θ ) is bounded.

Original: We define indicator functions
Revised: We define indicator functions

Original: [Equation]
Revised: [Equation]

Original: where r i is an undetermined coefficient.
Revised: where r i is an undetermined coefficient.

Original: Clearly, function ˆ I ( r i )
Revised: Clearly, function ˆ I ( r i )

Original: θ ∗ i ¯ R θ ∗
Revised: θ ∗ i ¯ R θ ∗

Original: i ( θ ) can be seen as an unary function f θ ∗ i ( x ) =
Revised: i ( θ ) can be seen as an unary function f θ ∗ i ( x ) =

Original: x r θ ∗ i +1 , (0 < x < r i ) about the independent variable ∥ θ − θ ∗ i ∥ .
Revised: x r θ ∗ i +1 , (0 < x < r i ) about the independent variable ∥ θ − θ ∗ i ∥ .

Original: Then for any ¯ δ 0 > 0 , we can always find
Revised: Then for any ¯ δ 0 > 0 , we can always find

Original: [Equation]
Revised: [Equation]

Original: to ensure there is a smooth connection (a parabola) between f θ ∗ i ( x ) and h θ ∗ i ( x ) .
Revised: to ensure there is a smooth connection (a parabola) between f θ ∗ i ( x ) and h θ ∗ i ( x ) .

Original: Denote this entirety after adding the smooth connection between f θ ∗ i ( x ) and h θ ∗ i ( x ) as j θ ∗ i ( x ) , j θ ∗ i ( x ) satisfied j ′′ ( x ) < 1 and the connection point on h θ ∗ i ( x ) is ˆ r i ( r i ) :
Revised: Denote this entirety after adding the smooth connection between f θ ∗ i ( x ) and h θ ∗ i ( x ) as j θ ∗ i ( x ) , j θ ∗ i ( x ) satisfied j ′′ ( x ) < 1 and the connection point on h θ ∗ i ( x ) is ˆ r i ( r i ) :

Original: = r i + ( r θ ∗ i + 1) r r θ ∗ i
Revised: = r i + ( r θ ∗ i + 1) r r θ ∗ i

Original: i .
Revised: i .

Original: Then let h θ ∗ i ( x ) be an arbitrary constant value M , for different r θ ∗ i , we can always get an inverse solution
Revised: Then let h θ ∗ i ( x ) be an arbitrary constant value M , for different r θ ∗ i , we can always get an inverse solution

Original: r i := h − 1 θ ∗ i ( M ) .
Revised: r i := h − 1 θ ∗ i ( M ) .

Original: Take K 0 :
Revised: Take K 0 :

Original: =
Revised: =

Original: min θ ∗ i ∈
Revised: min θ ∗ i ∈

Original: J ∗∞ { ˆ I δ θ ∗ i θ ∗ i ¯ R θ ∗
Revised: J ∗∞ { ˆ I δ θ ∗ i θ ∗ i ¯ R θ ∗

Original: i ( θ ) , 1 } , there must exists K 0 < K 0 , such that sets { U ( θ ∗ i , ˆ r
Revised: i ( θ ) , 1 } , there must exists K 0 < K 0 , such that sets { U ( θ ∗ i , ˆ r

Original: i ( h
Revised: i ( h

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: is what we need.
Revised: is what we need.

Original: We next discuss this problem case by case according to the value of ˆ r .
Revised: We next discuss this problem case by case according to the value of ˆ r .

Original: The first case is ˆ r = 1 (from here to equation 38), we define an event
Revised: The first case is ˆ r = 1 (from here to equation 38), we define an event

Original: [Equation]
Revised: [Equation]

Original: and the characteristic function be I (ˆ l 0 ) n,θ i .
Revised: and the characteristic function be I (ˆ l 0 ) n,θ i .

Original: Then we can get that
Revised: Then we can get that

Original: [Equation]
Revised: [Equation]

Original: where { ˆ ζ n } is a Martingale difference sequence defined as
Revised: where { ˆ ζ n } is a Martingale difference sequence defined as

Original: [Equation]
Revised: [Equation]

Original: where k 0 , ˆ k 1 are two constants.
Revised: where k 0 , ˆ k 1 are two constants.

Original: We also define I ( − ˆ l 0 ) n :
Revised: We also define I ( − ˆ l 0 ) n :

Original: [Equation]
Revised: [Equation]

Original: ,θ ∗ i , and obtain
Revised: ,θ ∗ i , and obtain

Original: [Equation]
Revised: [Equation]

Original: Through calculating the sum of equation 36, equation 37, we obtain
Revised: Through calculating the sum of equation 36, equation 37, we obtain

Original: [Equation]
Revised: [Equation]

Original: where ˆ ζ ′ n := (cid:80) mi
Revised: where ˆ ζ ′ n := (cid:80) mi

Original: =

Revised: =

Original: I (ˆ l 0 ) n,θ ∗ i ˆ ζ n .
Revised: I (ˆ l 0 ) n,θ ∗ i ˆ ζ n .

Original: Denote k ′ :
Revised: Denote k ′ :

Original: =
Revised: =

Original: k 

Revised: k

Original: ˆ k 1 ϵ 0 / 2 , we get
Revised: ˆ k 1 ϵ 0 / 2 , we get

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Through the upper martingale convergence theorem
Revised: Through the upper martingale convergence theorem

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: almost surely.
Revised: almost surely.

Original: By Theorem 3.1, we also (cid:80) + ∞ k =1 I ( − ˆ l 0 ) k < + ∞ almost surely, which means
Revised: By Theorem 3.1, we also (cid:80) + ∞ k =1 I ( − ˆ l 0 ) k < + ∞ almost surely, which means

Original: [Equation]
Revised: [Equation]

Original: almost surely.
Revised: almost surely.

Original: Denote p
Revised: Denote p

Original: [Equation]
Revised: [Equation]

Original: 1 , we have
Revised: 1 , we have

Original: [Equation]
Revised: [Equation]

Original: almost surely
Revised: almost surely

Original: [Equation]
Revised: [Equation]

Original: The second case is when ˆ r > 1 .
Revised: The second case is when ˆ r > 1 .

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: Then we construct a set
Revised: Then we construct a set

Original: [Equation]
Revised: [Equation]

Original: We also define event A (ˆ l 0 ) n,θ ∗ i
Revised: We also define event A (ˆ l 0 ) n,θ ∗ i

Original: = { θ n ∈ S
Revised: = { θ n ∈ S

Original: (ˆ l 0 ) }
Revised: (ˆ l 0 ) }

Original: and let the characteristic function be I (ˆ l 0 ) n,θ i .
Revised: and let the characteristic function be I (ˆ l 0 ) n,θ i .

Original: Then we can get that
Revised: Then we can get that

Original: [Equation]
Revised: [Equation]

Original: where { ˆ ζ n } is a Martingale difference sequence defined as
Revised: where { ˆ ζ n } is a Martingale difference sequence defined as

Original: [Equation]
Revised: [Equation]

Original: and k 0 , ˆ k 1 are two constants.
Revised: and k 0 , ˆ k 1 are two constants.

Original: Define I ( − ˆ l 0 ) n
Revised: Define I ( − ˆ l 0 ) n

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: where ˆ a 0 is a constant.
Revised: where ˆ a 0 is a constant.

Original: Through calculating the sum of equation 39 and equation 40, we get
Revised: Through calculating the sum of equation 39 and equation 40, we get

Original: [Equation]
Revised: [Equation]

Original: where ˆ ζ ′ n := (cid:80) mi
Revised: where ˆ ζ ′ n := (cid:80) mi

Original: =

Revised: =

Original: I (ˆ l 0 ) n,θ ∗ i ˆ ζ n .
Revised: I (ˆ l 0 ) n,θ ∗ i ˆ ζ n .

Original: We also have
Revised: We also have

Original: [Equation]
Revised: [Equation]

Original: This
Revised: This

Original: [Equation]
Revised: [Equation]

Original: Using the inequalities (1 + x ) r 0 ≥ 1 + r 0 x
Revised: Using the inequalities (1 + x ) r 0 ≥ 1 + r 0 x

Original: [Equation]
Revised: [Equation]

Original: ( r 0 < 0)
Revised: ( r 0 < 0)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Summing this over n , we have
Revised: Summing this over n , we have

Original: [Equation]
Revised: [Equation]

Original: Note that (cid:80) + ∞ k =

Revised: Note that (cid:80) + ∞ k =

Original: I
Revised: I

Original: [Equation]
Revised: [Equation]

Original: − ˆ
Revised: − ˆ

Original: [Equation]
Revised: [Equation]

Original: k < + ∞ almost surely, thus we have
Revised: k < + ∞ almost surely, thus we have

Original: [Equation]
Revised: [Equation]

Original: , almost surely .
Revised: , almost surely .

Original: Denote
Revised: Denote

Original: [Equation]
Revised: [Equation]

Original: Clearly, sup
Revised: Clearly, sup

Original: [Equation]
Revised: [Equation]

Original: almost surely .
Revised: almost surely .

Original: By Lemma A.1, we have
Revised: By Lemma A.1, we have

Original: [Equation]
Revised: [Equation]

Original: almost surely .
Revised: almost surely .

Original: Then, ˆ R
Revised: Then, ˆ R

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: Ω( n ) almost surely , which implies
Revised: Ω( n ) almost surely , which implies

Original: [Equation]
Revised: [Equation]

Original: almost surely .
Revised: almost surely .

Original: A. P ROOF OF C OROLLARY 3. 
Revised: A. P ROOF OF C OROLLARY 3.

Original: When the loss function g ( θ n ) attains the ε ′ accuracy, according to Theorem 3.4,the overall number of SGD iteration is
Revised: When the loss function g ( θ n ) attains the ε ′ accuracy, according to Theorem 3.4,the overall number of SGD iteration is

Original: [Equation]
Revised: [Equation]

Original: almost surely , if ˆ r =
Revised: almost surely , if ˆ r =

Original: [Equation]
Revised: [Equation]

Original: almost surely , if ˆ r > 1 .
Revised: almost surely , if ˆ r > 1 .

Original: Then we consider the computational time of a single step of SGD.
Revised: Then we consider the computational time of a single step of SGD.

Original: Generally, the main timeconsuming part of one step is computing the gradient of loss function on a batch of datasets, which can be decomposed into computing N 0 times of numerical differentiation, where the Nis the size of the dataset.
Revised: Generally, the main timeconsuming part of one step is computing the gradient of loss function on a batch of datasets, which can be decomposed into computing N 0 times of numerical differentiation, where the Nis the size of the dataset.

Original: We assume time consumed of computing a function value is O (cid:0) 1 (cid:1) .
Revised: We assume time consumed of computing a function value is O (cid:0) 1 (cid:1) .

Original: When a specific numerical differentiation scheme is given, such as ∂f ( θ
Revised: When a specific numerical differentiation scheme is given, such as ∂f ( θ

Original: [Equation]
Revised: [Equation]

Original: , it’s obviously the computation time of numerical gradient is O (cid:0) d (cid:1) .
Revised: , it’s obviously the computation time of numerical gradient is O (cid:0) d (cid:1) .

Original: In summary, the whole computation time is
Revised: In summary, the whole computation time is

Original: [Equation]
Revised: [Equation]

Original: almost surely , if ˆ r =
Revised: almost surely , if ˆ r =

Original: [Equation]
Revised: [Equation]

Original: almost surely , if ˆ r > 1 , which is bounded by a polynomial time.
Revised: almost surely , if ˆ r > 1 , which is bounded by a polynomial time.