Original: B ANDWIDTH - BASED S TEP -S IZES FOR N ON -C ONVEX S TOCHASTIC O PTIMIZATION 
Revised: B ANDWIDTH - BASED S TEP -S IZES FOR N ON -C ONVEX S TOCHASTIC O PTIMIZATION : N ON - MONOTONICITY , W ORST - CASE C ONVERGENCE AND P ERFORMANCE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Many popular learning-rate schedules for deep neural networks combine a decaying trend with local perturbations that attempt to escape saddle points and bad local minima.
Revised: Many popular learning-rate schedules for deep neural networks combine a decaying trend with local perturbations that attempt to escape saddle points and bad local minima.

Original: We derive convergence guarantees for bandwidth-based step-sizes, a general class of learning-rates that are allowed to vary in a banded region.
Revised: We derive convergence guarantees for bandwidth-based step-sizes, a general class of learning-rates that are allowed to vary in a banded region.

Original: This framework includes many popular cyclic and non-monotonic step-sizes for which no theoretical guarantees were previously known.
Revised: This framework includes many popular cyclic and non-monotonic step-sizes for which no theoretical guarantees were previously known.

Original: We provide worst-case guarantees for SGD on smooth non-convex problems under several bandwidth-based step sizes, including stagewise 1 / √ t and the popular step-decay (“constant and then drop by a constant”), which is also shown to be optimal.
Revised: We provide worst-case guarantees for SGD on smooth non-convex problems under several bandwidth-based step sizes, including stagewise 1 / √ t and the popular step-decay (“constant and then drop by a constant”), which is also shown to be optimal.

Original: Moreover, we show that its momentum variant converges as fast as SGD with the bandwidth-based step-decay step-size.
Revised: Moreover, we show that its momentum variant converges as fast as SGD with the bandwidth-based step-decay step-size.

Original: Finally, we propose novel step-size schemes in the bandwidth-based family and verify their efﬁciency on several deep neural network training tasks.
Revised: Finally, we propose novel step-size schemes in the bandwidth-based family and verify their efﬁciency on several deep neural network training tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Stochastic gradient methods including stochastic gradient descent (SGD) (Robbins and Monro, 1951) and its accelerated variants (e.g., SGD with momentum (Polyak, 1964; Sutskever et al., 2013)) have become the algorithmic workhorse in much of machine learning.
Revised: Stochastic gradient methods including stochastic gradient descent (SGD) (Robbins and Monro, 1951) and its accelerated variants (e.g., SGD with momentum (Polyak, 1964; Sutskever et al., 2013)) have become the algorithmic workhorse in much of machine learning.

Original: The step-size (learning rate) is the most important hyper-parameter for controlling the speed at which gradient-based methods converge to stationarity.
Revised: The step-size (learning rate) is the most important hyper-parameter for controlling the speed at which gradient-based methods converge to stationarity.

Original: For problems with multiple local minima, the step-size also affects which local optimum the optimization process converges to.
Revised: For problems with multiple local minima, the step-size also affects which local optimum the optimization process converges to.

Original: It therefore needs to be both well-designed and well-tuned to make SGD and its variants effective in practice.
Revised: It therefore needs to be both well-designed and well-tuned to make SGD and its variants effective in practice.

Original: In the deep learning literature, cyclical step-sizes (Loshchilov and Hutter, 2017; Smith, 2017) and non-monotonic schedules (Keskar and Saon, 2015; An et al., 2017; Seong et al., 2018) have attracted strong recent interest, with signiﬁcant beneﬁts for non-convex problems with poor local minima or saddle points (Seong et al., 2018).
Revised: In the deep learning literature, cyclical step-sizes (Loshchilov and Hutter, 2017; Smith, 2017) and non-monotonic schedules (Keskar and Saon, 2015; An et al., 2017; Seong et al., 2018; Loizou et al., 2021) have attracted strong recent interest, with signiﬁcant beneﬁts for non-convex problems with poor local minima or saddle points (Seong et al., 2018).

Original: Popular cyclical schedules include the cosine step-size (cosine with restart) (Loshchilov and Hutter, 2017) and the triangular policy (Smith, 2017), which have become the default choices in some deep learning libraries, e.g., PyTorch and TensorFlow (cf.
Revised: Popular cyclical schedules include the cosine step-size (cosine with restart) (Loshchilov and Hutter, 2017) and the triangular policy (Smith, 2017), which have become the default choices in some deep learning libraries, e.g., PyTorch and TensorFlow (cf.

Original: lr scheduler.
Revised: lr scheduler.

Original: CyclicLR and CosineAnnealingLR).
Revised: CyclicLR and CosineAnnealingLR).

Original: However, non-monotonic policies are much more complex to analyze than decaying ones, and theoretical results for these non-monotonic policies are scarce.
Revised: However, non-monotonic policies are much more complex to analyze than decaying ones, and theoretical results for these non-monotonic policies are scarce.

Original: This motivates us to focus on a bandwidth step-size framework, in which
Revised: This motivates us to focus on a bandwidth step-size framework, in which

Original: [Equation]
Revised: [Equation]

Original: for some boundary function δ ( t ) and positive constants m and M .
Revised: for some boundary function δ ( t ) and positive constants m and M .

Original: This framework allows for nonmonotonic step-sizes and covers most of the situations discussed above.
Revised: This framework allows for nonmonotonic step-sizes and covers most of the situations discussed above.

Original: In particular, it includes the cosine (Loshchilov and Hutter, 2017), triangular (Smith, 2017), sine wave (An et al., 2017) step-sizes as special cases.
Revised: In particular, it includes the cosine (Loshchilov and Hutter, 2017), triangular (Smith, 2017), sine wave (An et al., 2017) step-sizes as special cases.

Original: The framework provides a uniform convergence rate guarantee for all step-size policies which remain in the band (1).
Revised: The framework provides a uniform convergence rate guarantee for all step-size policies which remain in the band (1).

Original: This gives a lot of freedom to design novel step-sizes schedules with improved practical performance without loosing track of their theoretical convergence guarantee.
Revised: This gives a lot of freedom to design novel step-sizes schedules with improved practical performance without loosing track of their theoretical convergence guarantee.

Original: Note that the framework also allows adaptive step-size policies, as long as the step-size is guaranteed to belong to a band on the form (1).

Original: For example, the trust-region-ish algorithms by Curtis et al.

Original: (2019) uses a careful step normalization procedure to adapt the learning rate based on the gradient norm.

Original: It is not easy to derive a convergence rate guarantee for such a complex policy, but it can be captured by the bandwidth step-size framework and therefore inherits its theoretical convergence properties.

Original: In short, the bandwidth step-size framework is an alternative, and often easier, way to get the worst-case theoretical guarantees for complex step-size policies such as Curtis et al.

Original: (2019) and other alarm conditions.

Original: A more detailed discussion is given in Section D.3.

Original: The generic bandwidth framework has recently been proposed by Wang and Yuan (2021), but they only analyzed strongly convex problems.
Revised: The generic bandwidth framework has recently been proposed by Wang and Yuan (2021), but they only analyzed strongly convex problems.

Original: We believe that more signiﬁcant potential lies in the non-convex regime.
Revised: We believe that more signiﬁcant potential lies in the non-convex regime.

Original: For non-convex problems, non-monotonic step-sizes have distinct advantages, helping iterates to escape local minima and producing ﬁnal iterates of high quality.
Revised: For non-convex problems, non-monotonic step-sizes have distinct advantages, helping iterates to escape local minima and producing ﬁnal iterates of high quality.

Original: In the paper, we demonstrate this point on both a simple toy example and on large-scale neural network training tasks.
Revised: In the paper, we demonstrate this point on both a simple toy example and on large-scale neural network training tasks.

Original: Our main contribution is a sequence of non-asymptotic convergence results for the bandwidth stepsize on non-convex optimization problems, based on the popular “constant and then drop” step-size schedules (Krizhevsky et al., 2012;
Revised: Our main contribution is a sequence of non-asymptotic convergence results for the bandwidth stepsize on non-convex optimization problems, based on the popular “constant and then drop” step-size schedules (Krizhevsky et al., 2012;

Original: He et al., 2016; Hazan and Kale, 2014; Ge et al., 2019; Wang et al., 2021).
Revised: He et al., 2016; Hazan and Kale, 2014; Ge et al., 2019; Wang et al., 2021).

Original: This allows non-monotonic variations both within each (inner) stage and between stages.
Revised: This allows non-monotonic variations both within each (inner) stage and between stages.

Original: 1. C ONTRIBUTIONS
Revised: 1. C ONTRIBUTIONS

Original: Inspired by the strong potential of non-monotonic step-size schedules demonstrated above, we extend the bandwidth-based step-size framework to “constant and then drop” (multi-stage) proﬁles, where the bands stay constant throughout each stage and drops between stages.
Revised: Inspired by the strong potential of non-monotonic step-size schedules demonstrated above, we extend the bandwidth-based step-size framework to “constant and then drop” (multi-stage) proﬁles, where the bands stay constant throughout each stage and drops between stages.

Original: We provide convergence guarantees for both SGD and its momentum variant (SGDM) on non-convex problems.
Revised: We provide convergence guarantees for both SGD and its momentum variant (SGDM) on non-convex problems.

Original: Speciﬁcally,
Revised: Speciﬁcally,

Original: • We establish worst-case theoretical guarantees for SGD with bandwidth step-size on smooth nonconvex problems.
Revised: • We establish worst-case theoretical guarantees for SGD with bandwidth step-size on smooth nonconvex problems.

Original: We (i) derive an optimal rate for SGD under a bandwidth step-size with δ ( t ) = 1 / √ t ; (ii) and achieve optimal and near-optimal rates for step-decay (constant and then drop by a constant), improving the results by Wang et al.
Revised: We (i) derive an optimal rate for SGD under a bandwidth step-size with δ ( t ) = 1 / √ t ; (ii) and achieve optimal and near-optimal rates for step-decay (constant and then drop by a constant), improving the results by Wang et al.

Original: • We provide worst-case theoretical guarantees for SGDM with bandwidth-based step-decay step-size in the smooth nonconvex setting.
Revised: • We provide worst-case theoretical guarantees for SGDM with bandwidth-based step-decay step-size in the smooth non-convex setting.

Original: To the best of our knowledge, this is the ﬁrst results that provide optimal (Theorem 4.3) and near-optimal (Theorem 4.2) results for momentum with step-decay step-sizes.
Revised: To the best of our knowledge, these are the ﬁrst results that provide optimal (Theorem 4.3) and near-optimal (Theorem 4.2) results for momentum with step-decay step-sizes.

Original: Moreover, our results signiﬁcantly improve the convergence results from Liu et al.
Revised: Moreover, our results signiﬁcantly improve the convergence results from Liu et al.

Original: (2020) (see Remark 4.4).
Revised: (2020) (see Remark 4.4).

Original: • Our analysis results also provide state-of-art theoretical guarantees for cosine (Loshchilov and Hutter, 2017) and triangular (Smith, 2017) step-sizes if their boundary functions are within our bands.
Revised: • Our analysis results also provide state-of-the-art theoretical guarantees for cosine (Loshchilov and Hutter, 2017) and triangular (Smith, 2017) step-sizes if their boundary functions are within our bands.

Original: Especially, we improve the result of Li et al.
Revised: Especially, we improve the result of Li et al.

Original: (2021) for cosine step-size and achieve a state-of-art rate (see Remark 3.4).
Revised: (2021) for cosine step-size and achieve a state-of-art rate (see Remark 3.4).

Original: Moreover, our results ﬁrst provide the convergence guarantees for triangular step-size (Smith, 2017).
Revised: Moreover, our results ﬁrst provide the convergence guarantees for triangular step-size (Smith, 2017).

Original: • We propose novel, possibly non-monotonic, step-size schedules (e.g., step-decay with linearmode and cosine-mode) based on the bandwidth-based framework and demonstrate their efﬁcacy on several large-scale neural network training tasks.
Revised: • We propose novel, possibly non-monotonic, step-size schedules (e.g., step-decay with linearmode and cosine-mode) based on the bandwidth-based framework and demonstrate their efﬁcacy on several large-scale neural network training tasks.

Original: 1. R ELATED W ORK
Revised: 1. R ELATED W ORK

Original: This subsection reviews the theoretical development of the SGD algorithm and its momentum variant in the smooth non-convex setting, with a special focus on different step-size policies.
Revised: This subsection reviews the theoretical development of the SGD algorithm and its momentum variant in the smooth non-convex setting, with a special focus on different step-size policies.

Original: SGD for nonconvex problems The ﬁrst non-asymptotic convergence of SGD to a stationary point of a general smooth non-convex function was established in Ghadimi and Lan (2013).
Revised: SGD for nonconvex problems The ﬁrst non-asymptotic convergence of SGD to a stationary point of a general smooth non-convex function was established in Ghadimi and Lan (2013).

Original: The authors proved that a constant step-size O (1 / √ T ) attains a convergence rate of O (1 / √ T ) , where T is the iteration budget.
Revised: The authors proved that a constant step-size O (1 / √ T ) attains a convergence rate of O (1 / √ T ) , where T is the iteration budget.

Original: To the best of our knowledge, this rate is not improvable and was proven to be tight up to a constant without additional assumptions (Drori and Shamir, 2020).
Revised: To the best of our knowledge, this rate is not improvable and was proven to be tight up to a constant without additional assumptions (Drori and Shamir, 2020).

Original: For the 1 / √ t decay step-size, an O (ln T/ √ T ) rate can be easily obtained from (Ghadimi and Lan, 2013).
Revised: For the 1 / √ t decay step-size, an O (ln T/ √ T ) rate can be easily obtained from (Ghadimi and Lan, 2013).

Original: This rate can be improved to the optimal by selecting a random iterate using weights proportional to the inverse of the step-size (Wang et al., 2021).
Revised: This rate can be improved to the optimal by selecting a random iterate using weights proportional to the inverse of the step-size (Wang et al., 2021).

Original: The sampling rule in (Wang et al., 2021) depends on the step-size and is easily applicable to different step-size policies.
Revised: The sampling rule in (Wang et al., 2021) depends on the step-size and is easily applicable to different step-size policies.

Original: Thus, in this paper, we choose a similar sampling rule as (Wang et al., 2021) to favor the later iterates when selecting the output for SGD and its momentum variant.
Revised: Thus, in this paper, we choose a similar sampling rule as (Wang et al., 2021) to favor the later iterates when selecting the output for SGD and its momentum variant.

Original: Step-decay step-sizes Recently
Revised: Step-decay step-sizes Recently

Original: , the theoretical performance of step-decay or stagewise strategies has attracted an increasing attention due to their excellent practical performance (Yuan et al., 2019;
Revised: , the theoretical performance of step-decay or stagewise strategies has attracted an increasing attention due to their excellent practical performance (Yuan et al., 2019;

Original: Ge et al., 2019; Chen et al., 2019;
Revised: Ge et al., 2019; Chen et al., 2019;

Original: Li et al., 2021; Wang et al., 2021).
Revised: Li et al., 2021; Wang et al., 2021).

Original: For a class of least-squares problems, Ge et al.
Revised: For a class of least-squares problems, Ge et al.

Original: (2019) established a near-optimal O (ln T/T ) rate for the step-decay step-size (cut by 2 every T/ log 2 ( T ) iterates) and showed that step-decay can perform better than the polynomial decay step-size.
Revised: (2019) established a near-optimal O (ln T/T ) rate for the step-decay step-size (cut by 2 every T/ log 2 ( T ) iterates) and showed that step-decay can perform better than the polynomial decay step-size.

Original: Stochastic optimization methods with stagewise step-sizes decaying as 1 /t were analyzed in Chen et al.
Revised: Stochastic optimization methods with stagewise step-sizes decaying as 1 /t were analyzed in Chen et al.

Original: A near-optimal rate for the continuous version of step-decay, called exp-decay, as well as for cosine decay step-sizes under the Polyak-L´ojasiewicz (PL) condition and a general smooth assumption were established in Li et al.
Revised: A near-optimal rate for the continuous version of step-decay, called exp-decay, as well as for cosine decay step-sizes under the Polyak-L´ojasiewicz (PL) condition and a general smooth assumption were established in Li et al.

Original: However, in the smooth case, to achieve such results for exponential and cosine decay step-sizes, the initial step-size is required to be bounded by O (1 / √ T ) .
Revised: However, in the smooth case, to achieve such results for exponential and cosine decay step-sizes, the initial step-size is required to be bounded by O (1 / √ T ) .

Original: This is obviously impractical when the number of iterations T is large.
Revised: This is obviously impractical when the number of iterations T is large.

Original: Near-optimal rates (up to ln T ) of SGD with step-decay step-size in several general settings including strongly convex, convex and smooth (non-convex) problems were proved in Wang et al.
Revised: Near-optimal rates (up to ln T ) of SGD with step-decay step-size in several general settings including strongly convex, convex and smooth (non-convex) problems were proved in Wang et al.

Original: They also removed the restriction on the initial step-size for exponential decay step-sizes.
Revised: They also removed the restriction on the initial step-size for exponential decay step-sizes.

Original: Empirical evidences have been given in (Wang and Yuan, 2021) that bandwidth-based strategies can improve the performance of the step-decay step-size on some large scale neural network tasks.
Revised: Empirical evidences have been given in (Wang and Yuan, 2021) that bandwidth-based strategies can improve the performance of the step-decay step-size on some large scale neural network tasks.

Original: However, no theoretical guarantees for non-convex problems were given.
Revised: However, no theoretical guarantees for non-convex problems were given.

Original: SGD
Revised: SGD

Original: with momentum on nonconvex problems The momentum variant of SGD (SGDM) has been widely used in deep neural networks (Krizhevsky et al., 2012;
Revised: with momentum on nonconvex problems The momentum variant of SGD (SGDM) has been widely used in deep neural networks (Krizhevsky et al., 2012;

Original: Sutskever et al., 2013;
Revised: Sutskever et al., 2013;

Original: He et al., 2016;
Revised: He et al., 2016;

Original: Zagoruyko and Komodakis, 2016).
Revised: Zagoruyko and Komodakis, 2016).

Original: Due to its practical success on neural networks, its theoretical performance is now attracting a lot of interest, especially for nonconvex problems (Yan et al., 2018;
Revised: Due to its practical success on neural networks, its theoretical performance is now attracting a lot of interest, especially for nonconvex problems (Yan et al., 2018;

Original: Gadat et al., 2018; Chen et al., 2019; Gitman et al., 2019; Mai and Johansson, 2020; Liu et al., 2020;
Revised: Gadat et al., 2018; Chen et al., 2019; Gitman et al., 2019; Mai and Johansson, 2020; Liu et al., 2020;

Original: Defazio, 2020).
Revised: Defazio, 2020).

Original: Under the assumption of bounded gradients, Yan et al.
Revised: Under the assumption of bounded gradients, Yan et al.

Original: (2018) proposed a uniﬁed analysis framework for stochastic momentum methods and proved an optimal O (1 / √ T ) rate under constant step-sizes.
Revised: (2018) proposed a uniﬁed analysis framework for stochastic momentum methods and proved an optimal O (1 / √ T ) rate under constant step-sizes.

Original: A similar result for the Nesterov-accelerated variant was established in Ghadimi and Lan (2016).
Revised: A similar result for the Nesterov-accelerated variant was established in Ghadimi and Lan (2016).

Original: However, studies related to the multi-stage performance of SGD with momentum is lacking and far from being complete.
Revised: However, studies related to the multi-stage performance of SGD with momentum is lacking and far from being complete.

Original: Reference Chen et al.
Revised: Reference Chen et al.

Original: (2019) considers a momentum method with a stagewise step-size, but the method is a proximal point algorithm with extra averaging between stages, and not the widely used momentum SGD considered here.
Revised: (2019) considers a momentum method with a stagewise step-size, but the method is a proximal point algorithm with extra averaging between stages, and not the widely used momentum SGD considered here.

Original: More recently, Liu et al.
Revised: More recently, Liu et al.

Original: (2020) established the convergence for multi-stage SGDM and provided empirical evidence to show that multi-stage SGDM is faster.
Revised: (2020) established the convergence for multi-stage SGDM and provided empirical evidence to show that multi-stage SGDM is faster.

Original: However, their results require an inverse relationship between stage length and step-size which limits the initial stage length or step-size.
Revised: However, their results require an inverse relationship between stage length and step-size which limits the initial stage length or step-size.

Original: A detailed comparison with (Liu et al., 2020) will be given in Section 4 (see Remark 4.4).
Revised: A detailed comparison with (Liu et al., 2020) will be given in Section 4 (see Remark 4.4).

Original: Organization: The rest of this paper is organized as follows.
Revised: Organization: The rest of this paper is organized as follows.

Original: Notations and basic deﬁnitions are introduced in Section 2.
Revised: Notations and basic deﬁnitions are introduced in Section 2.

Original: Our novel theoretical results for SGD and its momentum variant (SGDM) under bandwidth-based step-sizes are introduced in Sections 3 and 4, respectively.
Revised: Our novel theoretical results for SGD and its momentum variant (SGDM) under bandwidth-based step-sizes are introduced in Sections 3 and 4, respectively.

Original: Numerical experiments are presented and reported in Section 5.
Revised: Numerical experiments are presented and reported in Section 5.

Original: Finally, Section 6 concludes the paper.
Revised: Finally, Section 6 concludes the paper.

Original:  P ROBLEM 
Revised:  P ROBLEM S ET -U P

Original: [Equation]

Original: We study the following, possibly non-convex, stochastic optimization problem
Revised: We study the following, possibly non-convex, stochastic optimization problem

Original: [Equation]
Revised: [Equation]

Original: where ξ is a random variable drawn from some (unknown) probability distribution Ξ and f ( x ; ξ ) is the instantaneous loss function over the variable x ∈ R d .
Revised: where ξ is a random variable drawn from some (unknown) probability distribution Ξ and f ( x ; ξ ) is the instantaneous loss function over the variable x ∈ R d .

Original: We consider stochastic gradient methods that generate iterates x t according to
Revised: We consider stochastic gradient methods that generate iterates x t according to

Original: [Equation]
Revised: [Equation]

Original: where η t is the step-size and d t the search direction (e.g., d t = ∇ f ( x t ; ξ ) for SGD).
Revised: where η t is the step-size and d t the search direction (e.g., d t = ∇ f ( x t ; ξ ) for SGD).

Original: We assume that there are constants m > 0 and M ≥ m , and two functions n ( t ) and δ ( t ) :
Revised: We assume that there are constants m > 0 and M ≥ m , and two functions n ( t ) and δ ( t ) :

Original: R → R such that such that
Revised: R → R such that such that

Original: [Equation]
Revised: [Equation]

Original: where n ( t ) ∈ [ m, M ] and δ ( t ) is monotonically decreasing function satisfying δ (1) = 1 .
Revised: where n ( t ) ∈ [ m, M ] and δ ( t ) is monotonically decreasing function satisfying δ (1) = 1 .

Original: Note that even though the boundary function δ ( t ) is monotonic, the step-size itself is not restricted to be.
Revised: Note that even though the boundary function δ ( t ) is monotonic, the step-size itself is not restricted to be.

Original: Throughout the paper, we make the following assumptions:
Revised: Throughout the paper, we make the following assumptions:

Original: Assumption 1.
Revised: Assumption 1.

Original: The loss function f satisﬁes (cid:107)∇ f ( x ) −∇ f ( y ) (cid:107) ≤ L (cid:107) x − y (cid:107) for every x, y ∈ dom ( f ) .
Revised: The loss function f satisﬁes (cid:107)∇ f ( [Equation] (cid:107) x [Equation]

Revised: y (cid:107) for every x, y ∈ dom ( f ) .

Original: Assumption 2.
Revised: Assumption 2.

Original: For any input vector x , the stochastic gradient oracle O returns a vector g such that(a) E
Revised: For any input vector x , the stochastic gradient oracle O returns a vector g such that

Original: [ (cid:107) g − ∇ f ( x ) (cid:107) 2 ] ≤ ρ (cid:107)∇ f ( x ) (cid:107) 2 + σ where ρ ≥ 0 and σ ≥ 0 ; (b) E [ (cid:107) g (cid:107) 2 ] ≤ G 2 .∇
Revised: [Equation] 2 ] ≤ ρ (cid:107)∇ f ( x ) (cid:107) 2 + σ where ρ ≥ 0 and σ ≥ 0 ; (b) E [ (cid:107) g (cid:107) 2 ] ≤ G 2 .

Original:  N ON - ASYMPTOTIC C ONVERGENCE OF SGD WITH B ANDWIDTH - BASED S TEP -S IZE
Revised:  N ON - ASYMPTOTIC C ONVERGENCE OF SGD WITH B ANDWIDTH - BASED S TEP -S IZE

Original: In this section, we provide the ﬁrst non-asymptotic convergence guarantees for SGD with bandwidthbased step-sizes on smooth non-convex problems.
Revised: In this section, we provide the ﬁrst non-asymptotic convergence guarantees for SGD with bandwidthbased step-sizes on smooth non-convex problems.

Original: The results consider a general family of bandwidthbased step-sizes which includes the classical multi-stage SGD as a special case.
Revised: The results consider a general family of bandwidthbased step-sizes which includes the classical multi-stage SGD as a special case.

Original: Algorithm 1 details our bandwidth-based version of the popular “constant and then drop” policy for
Revised: Algorithm 1 details our bandwidth-based version of the popular “constant and then drop” policy for

Original: SGD.
Revised: SGD.

Original: Here, the boundary function δ ( t ) is adjusted in an outer stage, and the length of each stage S t is allowed to vary.
Revised: Here, the boundary function δ ( t ) is adjusted in an outer stage, and the length of each stage S t is allowed to vary.

Original: Similar to Wang et al.
Revised: Similar to Wang et al.

Original: (2021), the output distribution depends on the inverse of δ ( t ) , hence puts more weight on the ﬁnal iterates.
Revised: (2021), the output distribution depends on the inverse of δ ( t ) , hence puts more weight on the ﬁnal iterates.

Original: By considering speciﬁc combinations of δ ( t ) and S t , this framework allows us to analyze several important multi-stage SGD algorithms, including those with constant, polynomial-decay and step-decay step-sizes.
Revised: By considering speciﬁc combinations of δ ( t ) and S t , this framework allows us to analyze several important multi-stage SGD algorithms, including those with constant, polynomial-decay and step-decay step-sizes.

Revised: For example, we consider the step-decay step-size by letting n ( t, i ) = m

Revised: and δ ( t ) = 1 /α t − 1 where m denotes its initial step-size and α > 1 .

Original: Many interesting results on polynomial-decay step-size (e.g., δ ( t ) = 1 / √ t , we called it 1 / √ t -band) are given in Appendix A.
Revised: Many interesting results on polynomial-decay step-size (e.g., δ ( t ) = 1 / √ t , we called it 1 / √ t -band) are given in Appendix A.

Original: 3. C ONVERGENCE U NDER B ANDWIDTH S TEP -D ECAY S TEP -S IZE
Revised: 3. C ONVERGENCE U NDER B ANDWIDTH S TEP -D ECAY S TEP -S IZE

Original: Another important step-size is Step-Decay (“constant and then drop by a constant”), which is popular and widely used in practice, e.g. for neural network training (Krizhevsky et al., 2012;
Revised: Another important step-size is Step-Decay (“constant and then drop by a constant”), which is popular and widely used in practice, e.g. for neural network training (Krizhevsky et al., 2012;

Original: He et al., 2016).
Revised: He et al., 2016).

Original: In this subsection, we analyze bandwidth step-sizes that include step-decay as a special case.
Revised: In this subsection, we analyze bandwidth step-sizes that include step-decay as a special case.

Original: For Step-Decay, the stage length S t is typically a hyper-parameter selected by experience.
Revised: For Step-Decay, the stage length S t is typically a hyper-parameter selected by experience.

Original: We ﬁrst analyze a bandwidth version of the algorithm analyzed in [Theorem 3.2](Wang et al., 2021), namely Algorithm 1 with N =  (log α T ) / 2 ( α > 1 ) outer loops, each with a constant length ofwith
Revised: We ﬁrst analyze a bandwidth version of the algorithm analyzed in [Theorem 3.2](Wang et al., 2021), namely Algorithm 1 with N = (cid:98) (log α T ) / 2 (cid:99) outer loops where α > 1 , each with a constant length of

Original: S t =  2 T/ log α T  .
Revised: S t = (cid:100) 2 T/ log α T (cid:101) .

Original: The logarithmic dependence of N on T leads to a small number of stages in practice, and was demonstrated to perform well in deep neural network tasks (Wang et al., 2021).
Revised: The logarithmic dependence of N on T leads to a small number of stages in practice, and was demonstrated to perform well in deep neural network tasks (Wang et al., 2021).

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Under Assumptions 1 and 2(a), and assume that there exists a constant ∆ 0 > 0 such that E [ f ( x t 1 ) − f ∗ ] ≤ ∆ 0 for each t ≥ 1 where f ∗ = min f ( x ) , if we run Algorithm 1 with T > 1 , η ti ≤ 1 / (( ρ + 1) L ) , N =  (log α T ) / 2  , S t =  2 T/ log α T  , and δ ( t ) 
Revised: Under Assumptions 1 and 2(a), and assume that there exists a constant ∆ 0 > 0 such that E [ f ( x t 1 ) − f ∗ ] ≤ ∆ 0 for each t ≥ 1 where f ∗ = min f ( x ) , if we run Algorithm 1 with T (cid:29) α 2 , η ti ≤ 1 / (( ρ + 1) L ) , N = (cid:98) (log α T ) / 2 (cid:99) , S t = (cid:100) 2 T/ log α T (cid:101) , and δ ( t ) =

Original: = 1 /α t − 1 for 1 ≤ t ≤ N , where α > 1 then
Revised: 1 /α t − 1 for 1 ≤ t ≤ N , where α > 1 then

Original: [Equation]
Revised: [Equation]

Original: Theorem 3.1 establishes a near-optimal (up to ln T ) rate for the step-decay bandwidth scheme which matches the result achieved at its boundaries i.e., η t i = mδ ( t ) or η t i =
Revised: Theorem 3.1 establishes a near-optimal (up to ln T ) rate for the step-decay bandwidth scheme which matches the result achieved at its boundaries i.e., η ti = mδ ( t ) or η ti =

Original: Mδ ( t ) (Wang et al., 2021).
Revised: Mδ ( t ) (Wang et al., 2021).

Original: As the next theorem shows, this guarantee can be improved by appropriate tuning of the stage length S t .
Revised: As the next theorem shows, this guarantee can be improved by appropriate tuning of the stage length S t .

Original: Remark 3.2.
Revised: Remark 3.2.

Original: (Justiﬁcation of uniformly bounds on the function values)
Revised: ( Justiﬁcation of uniformly bounds on the function values )

Original: In Theorem 3.1, we require that the expectation of the function value at each outer iterate E [ f ( x t 1 )] is uniformly upper bounded.
Revised: In Theorem 3.1, we require that the expectation of the function value at each outer iterate E [ f ( x t 1 )] is uniformly upper bounded.

Original: As shown by Shi et al.
Revised: As shown by Shi et al.

Original: (2020), the function values at the iterates of SGD can be controlled (bounded) by the initial state provided the step-size is bounded by 1 /L .
Revised: (2020), the function values at the iterates of SGD can be controlled (bounded) by the initial state provided the step-size is bounded by 1 /L .

Original: So the assumption is fair if the initial state is settled.
Revised: So the assumption is fair if the initial state is settled.

Original: Nevertheless, this assumption (or its stronger version that the objective function is bounded) is commonly used or implied in optimization (Hazan et al., 2015; Xu et al., 2019b; 2020; 2019a) and statistic machine learning (Vapnik, 1998; Cortes et al., 2019) , and it has never been violated in our numerical experiments.
Revised: Nevertheless, this assumption (or its stronger version that the objective function is bounded) is commonly used or implied in optimization (Hazan et al., 2015; Xu et al., 2019b; 2020; 2019a) and statistic machine learning (Vapnik, 1998; Cortes et al., 2019) , and it has never been violated in our numerical experiments.

Original: Theorem 3.3.
Revised: Theorem 3.3.

Original: Under Assumptions 1 and 2(a), and assume that there exists a constant ∆ 0 > 0 such that E [ f ( x t 1 ) − f ∗ ] ≤ ∆ 0 for each t ≥ 1 , if we run Algorithm 1 with T > 1 , η ti ≤ 1 / (( ρ + 1) L ) ,
Revised: Under Assumptions 1 and 2(a), and assume that there exists a constant ∆ 0 > 0 such that E [ f ( x t 1 ) − f ∗ ] ≤ ∆ 0 for each t ≥ 1 , if we run Algorithm 1 with T > α 2 , η ti ≤ 1 / (( ρ + 1) L ) ,

Original: S 0 = √ T , S t =  S 

Revised: S 0 = √ T , S t = (cid:100) S

Original: α ( t − 1)  and δ ( t ) 
Revised: α ( t − 1) (cid:101) and δ ( t ) =

Original: = 1 /α t − 1 where α > 1 , then
Revised: 1 /α t − 1 where α > 1 , then

Original: [Equation]
Revised: [Equation]

Original: Optimal rate for step-decay step-size The theorem shows that if the stage length S t increases exponentially, and the length of the ﬁrst stage is set appropriately, then we can achieve an optimal
Revised: Optimal rate for step-decay step-size The theorem shows that if the stage length S t increases exponentially, and the length of the ﬁrst stage is set appropriately, then we can achieve an optimal

Original: O (1 / √ T ) rate for the bandwidth step-decay step-size in the non-convex case.
Revised: O (1 / √ T ) rate for the bandwidth step-decay step-size in the non-convex case.

Original: If M = m , which means that the bandwidth scheme degenerates to the step-decay type step-size, Theorem 3.3 removes the logarithmic term present in the results of Wang et al. (2021).
Revised: If M = m , which means that the bandwidth scheme degenerates to the step-decay type step-size, Theorem 3.3 removes the logarithmic term present in the results of Wang et al. (2021).

Original: To the best of our knowledge, this is the ﬁrst result that demonstrates that vanilla SGD with step-decay step-sizes can achieve the optimal rate for general non-convex problems.
Revised: To the best of our knowledge, this is the ﬁrst result that demonstrates that vanilla SGD with step-decay step-sizes can achieve the optimal rate for general non-convex problems.

Original: The numerical performance of the two step-size schedules in
Revised: The numerical performance of the two step-size schedules in

Original: Theorems 3.1 and 3.3 are reported in Figure 4.
Revised: Theorems 3.1 and 3.3 are reported in Figure 4.

Original: Beneﬁts of Theorems 3.5 vs the references of Hazan and Kale (2014); Yuan et al.
Revised: Beneﬁts of Theorems 3.5 vs the references of Hazan and Kale (2014); Yuan et al.

Original: Another commonly used step-decay scheme in theory which halves the step-size after each stage and then doubles the length of each stage (e.g., (Hazan and Kale, 2014; Yuan et al., 2019)).
Revised: Another commonly used step-decay scheme in theory which halves the step-size after each stage and then doubles the length of each stage (e.g., (Hazan and Kale, 2014; Yuan et al., 2019)).

Original: In Hazan and
Revised: In Hazan and

Original: Kale (2014), which considers strongly convex problem, the initial stage is very short, S 1 = 4 , while the analysis in Yuan et al.
Revised: Kale (2014), which considers strongly convex problem, the initial stage is very short, S 1 = 4 , while the analysis in Yuan et al.

Original: (2019) for PL functions use an inverse relation between stage length and step-size, which means that a longer initial stage length requires a smaller stepsize.
Revised: (2019) for PL functions use an inverse relation between stage length and step-size, which means that a longer initial stage length requires a smaller stepsize.

Original: In contrast to these references, Theorem 3.3 considers a step-decay with a long ﬁrst stage, S 1 = √ T  , which allows  to beneﬁt from a large constant step-size for more iterations.
Revised: In contrast to these references, Theorem 3.3 considers a step-decay with a long ﬁrst stage, S 1 = (cid:100)√ T (cid:101) , which allows us to beneﬁt from a large constant step-size for more iterations.

Original: Remark 3.4.
Revised: Remark 3.4.

Original: ( Guarantees for cyclical step-sizes )
Revised: ( Guarantees for cyclical step-sizes )

Original: In (Loshchilov and Hutter, 2017), the authors decay the step-size with cosine annealing and use η t min <
Revised: In (Loshchilov and Hutter, 2017), the authors decay the step-size with cosine annealing and use η t min <

Original: η t max to control the range of the step-size.
Revised: η t max to control the range of the step-size.

Original: If mδ ( t ) ≤ η t min , η t max ≤ Mδ ( t ) , then our results provide convergence guarantees for their step-size.
Revised: If mδ ( t ) ≤ η t min , η t max ≤ Mδ ( t ) , then our results provide convergence guarantees for their step-size.

Original: To achieve a near-optimal rate, Li et al.
Revised: To achieve a near-optimal rate, Li et al.

Original: (2021, Theorem 4) need to use an initial step-size that is smaller than O (1 / √ T ) which is obviously impractical.
Revised: (2021, Theorem 4) need to use an initial step-size that is smaller than O (1 / √ T ) which is obviously impractical.

Original: In contrast, we allow the cosine step-size to start from a relatively large step-size and then gradually decay (see Theorem 3.1) and also improve the convergence rate to be optimal (Theorem 3.3).
Revised: In contrast, we allow the cosine step-size to start from a relatively large step-size and then gradually decay (see Theorem 3.1) and also improve the convergence rate to be optimal (Theorem 3.3).

Original: A triangular cyclical step-size is proposed by Smith (2017) which is varied around the two boundaries that drop by a constant after a few iterations.
Revised: A triangular cyclical step-size is proposed by Smith (2017) which is varied around the two boundaries that drop by a constant after a few iterations.

Original: Our analysis ﬁrst provides theoretical guarantees (e.g.,
Revised: Our analysis ﬁrst provides theoretical guarantees (e.g.,

Original: Theorems 3.1 and 3.3) also for this step-size.
Revised: Theorems 3.1 and 3.3) also for this step-size.

Original: The details are shown in Appendix D.2.
Revised: The details are shown in Appendix D.2.

Original: [Figure]
Revised: [Figure]

Original:  N ON - ASYMPTOTIC C ONVERGENCE OF SGDM U NDER B ANDWIDTH - BASED S TEP -S IZE
Revised:  N ON - ASYMPTOTIC C ONVERGENCE OF SGDM U NDER B ANDWIDTH - BASED S TEP -S IZE

Original: In this section, we establish the ﬁrst non-asymptotic convergence properties of SGD with momentum (SGDM) under the bandwidth-based step-size on smooth nonconvex problems.
Revised: In this section, we establish the ﬁrst non-asymptotic convergence properties of SGD with momentum (SGDM) under the bandwidth-based step-size on smooth nonconvex problems.

Original: In this scheme, the inner iterations in Step 6 of Algorithm 1 are essentially replaced by
Revised: In this scheme, the inner iterations in Step 6 of Algorithm 1 are essentially replaced by

Original: [Equation]
Revised: [Equation]

Original: for β ∈ (0 , 1)
Revised: for β ∈ (0 , 1)

Original: We refer to Algorithm 2 in Appendix C for a more detailed description.
Revised: We refer to Algorithm 2 in Appendix C for a more detailed description.

Original: As in many studies of momentum-methods (e.g. Ghadimi et al.
Revised: As in many studies of momentum-methods (e.g. Ghadimi et al.

Original: ; Yan et al.
Revised: ; Yan et al.

Original: Liu et al.
Revised: Liu et al.

Original: (2020); Mai and Johansson (2020)), we establish an iterate relationship on the form E [ W t +1 ] ≤ E [ W t ] − c 0 η E [ e t ] + c 1 η 2 , where W t is a Lyapunov function, e t is a performance measure (here, e t =
Revised: (2020); Mai and Johansson (2020)), we establish an iterate relationship of the form E [ W t +1 ] ≤ E [ W t ] − c 0 η E [ e t ] + c 1 η 2 , where W t is a Lyapunov function, e t is a performance measure (here, e t =

Original: (cid:107)∇ f ( · ) (cid:107) 2 ), η is the step-size and c 0 and c 1 are constants.
Revised: (cid:107)∇ f ( · ) (cid:107) 2 ), η is the step-size and c 0 and c 1 are constants.

Original: However, due to the time-dependent and possibly non-monotonic bandwidth-based step-size, we cannot use the Lyapunov functions suggested in Yan et al.
Revised: However, due to the time-dependent and possibly non-monotonic bandwidth-based step-size, we cannot use the Lyapunov functions suggested in Yan et al.

Original: Liu et al.
Revised: Liu et al.

Original: but rely on the following non-trivial construction:
Revised: but rely on the following non-trivial construction:

Original: Lemma 4.1.
Revised: Lemma 4.1.

Original: Suppose that Assumption 1 and Assumption 2(b) hold.
Revised: Suppose that Assumption 1 and Assumption 2(b) hold.

Original: Let z ti =
Revised: Let z ti =

Original: (1 − β ) − 1 ( x ti − βx ti − 1 ) and assume that there exists a constant ∆ 0 such that E [ f ( x ti ) − f ∗ ] ≤ ∆ 0 for t, i ≥ 1 and the step-size in each stage is monotonically decreasing.
Revised: (1 − β ) − 1 ( x ti − βx ti − 1 ) and assume that there exists a constant ∆ 0 such that E [ f ( x ti ) − f ∗ ] ≤ ∆ 0 for t, i ≥ 1 and the step-size in each stage is monotonically decreasing.

Original: Deﬁne the function W ti +
Revised: Deﬁne the function W ti +

Original: [Equation]
Revised: [Equation]

Original: where r = βL 2(1 − β 2 )(1 − β ) 2 .
Revised: where r = βL 2(1 − β 2 )(1 − β ) 2 .

Original: Then, if η ti ≤ 1 /L , for any t and i ≥ 2 , we haveE [
Revised: Then, if η ti ≤ 1 /L , for any t and i ≥ 2 , we have

Original: [Equation]
Revised: [Equation]

Original: where A 1 =β ∆ 0 1 − β + ∆ z + rG 2 L 2 , B 1 = r (1 − β )(2 − β )
Revised: where A 1 =

Original: + L 2(1 − β ) 2 , and ∆ z = ∆ 0 1 − β + βG 2 2(1 − β ) 2 L .
Revised: [Equation]

Original: Note that even though the step-size is assumed to be monotonically decreasing in each stage, it may be increased between stages, leading to a globally non-monotonic step-size.
Revised: Note that even though the step-size is assumed to be monotonically decreasing in each stage, it may be increased between stages, leading to a globally non-monotonic step-size.

Original: The proposed bandwidthbased step-sizes (e.g., step-decay with linear or cosine modes) in the numerical experiments and the cosine annealing policy proposed in (Loshchilov and Hutter, 2017) all satisfy this condition.
Revised: The proposed bandwidthbased step-sizes (e.g., step-decay with linear or cosine modes) in the numerical experiments and the cosine annealing policy proposed in (Loshchilov and Hutter, 2017) all satisfy this condition.

Original: Note that, unlike (Mai and Johansson, 2020; Liu et al., 2020), the momentum parameter β does not rely on the step-size, but can be chosen freely in the interval (0 , 1) .
Revised: Note that, unlike (Mai and Johansson, 2020; Liu et al., 2020), the momentum parameter β does not rely on the step-size, but can be chosen freely in the interval (0 , 1) .

Original: In particular, our analysis supports the common choice of β = 0 .
Revised: In particular, our analysis supports the common choice of β = 0 .

Original: 9 used as default in many deep learning libraries (Krizhevsky et al., 2012;
Revised: 9 used as default in many deep learning libraries (Krizhevsky et al., 2012;

Original: He et al., 2016).
Revised: He et al., 2016).

Original: Similar to Remark 3.2, the function value of the iterates for momentum can also be controlled (bounded) by the initial state given η ti ≤ 1 /L ; see (Shi, 2021).
Revised: Similar to Remark 3.2, the function value of the iterates for momentum can also be controlled (bounded) by the initial state given η ti ≤ 1 /L ; see (Shi, 2021).

Original: Therefore, we believe our assumptions are reasonable.
Revised: Therefore, we believe our assumptions are reasonable.

Original: If we restrict the analysis to a single stage, N = 1 , the lemma allows to recover the optimal rate for
Revised: If we restrict the analysis to a single stage, N = 1 , the lemma allows to recover the optimal rate for

Original: SGDM under the step-size η ti = η 0 / √ T
Revised: SGDM under the step-size η ti = η 0 / √ T

Original: (Yan et al., 2018; Mai and Johansson, 2020; Liu et al., 2020;
Revised: (Yan et al., 2018; Mai and Johansson, 2020; Liu et al., 2020;

Original: Defazio, 2020) and to prove, for the ﬁrst time, an optimal O (1 / √ T ) rate for SGDM under the 1 / √ i stepsize.
Revised: Defazio, 2020) and to prove, for the ﬁrst time, an optimal O (1 / √ T ) rate for SGDM under the 1 / √ i stepsize.

Original: These results are formalized in Appendix D.1.
Revised: These results are formalized in Appendix D.1.

Original: 4. C ONVERGENCE OF SGDM FOR B ANDWIDTH S TEP -D ECAY S TEP -S IZE
Revised: 4. C ONVERGENCE OF SGDM FOR B ANDWIDTH S TEP -D ECAY S TEP -S IZE

Original: We now show the convergence complexity of SGDM with the bandwidth step-decay step-size.
Revised: We now show the convergence complexity of SGDM with the bandwidth step-decay step-size.

Original: Here step-decay means that the bandwidth limits are divided by a constant after some iterations.
Revised: Here step-decay means that the bandwidth limits are divided by a constant after some iterations.

Original: We ﬁrst consider the total number of iterations T to be given, the stage length S t to be constant, and the number of stages N as a hyper-parameter.
Revised: We ﬁrst consider the total number of iterations T to be given, the stage length S t to be constant, and the number of stages N as a hyper-parameter.

Original: Theorem 4.2.
Revised: Theorem 4.2.

Original: Assume the same setting as Lemma 4.1.
Revised: Assume the same setting as Lemma 4.1.

Original: If given the total number of iterations T ≥ 1 ,
Revised: If given the total number of iterations T ≥ 1 ,

Original: N ≥ 1 , S t = S =  T/N  , δ ( t ) = 1 /α t − 1 for each 1 ≤ t ≤ N and α > 1 , then
Revised: N ≥ 1 , S t = S = (cid:100) T/N (cid:101) , δ ( t ) = 1 /α t − 1 for each 1 ≤ t ≤ N and α > 1 , then

Original: [Equation]
Revised: [Equation]

Original: where C 0 =
Revised: where C 0 =

Original: r
Revised: r

Original: ( G 2 L + 2∆ 0 ) , C 

Revised: ( G 2 L + 2∆ 0 ) , C

Original: =
Revised: =

Original: A 1 + ∆ z + ∆ 0 1 − β , and C 2 = C 0 + A 2 G 2 , A 2 = 1 + β 2(1 − β ) 2 , and W 11 , A 1 , B 1 , ∆ z , and r are deﬁned in Lemma 4.1.
Revised: A 1 + ∆ z + ∆ 0 1 − β , and C 2 = C 0 + A 2 G 2 , A 2 = 1 + β 2(1 − β ) 2 , and W 11 , A 1 , B 1 , ∆ z , and r are deﬁned in Lemma 4.1.

Original: Furthermore, if N =  (log α T ) / 2  and
Revised: Furthermore, if N = (cid:98) (log α T ) / 2 (cid:99) and

Original: S t =  2 T/ log α T  for each 1 ≤ t ≤ N where α > 1 , we have
Revised: S t = (cid:100) 2 T/ log α T (cid:101) for each 1 ≤ t ≤ N where α > 1 , we have

Original: [Equation]
Revised: [Equation]

Original: When N = 1 , m ≤ η ti ≤ M and the bound (7) reduces to E [ (cid:107)∇ f (ˆ x T ) (cid:107) 2 ] ≤ O ( 1 T + 1 mT + M ) .
Revised: When N = 1 , m ≤ η ti ≤ M and the bound (7) reduces to E [ (cid:107)∇ f (ˆ x T ) (cid:107) 2 ] ≤ O ( 1 T + 1 mT + M ) .

Original: If, in particular, m and M are of order O (1 / √ T ) , then we can derive the optimal convergence for constant bandwidth step-sizes, comparable to the literature for constant step-sizes (Yan et al., 2018;
Revised: If, in particular, m and M are of order O (1 / √ T ) , then we can derive the optimal convergence for constant bandwidth step-sizes, comparable to the literature for constant step-sizes (Yan et al., 2018;

Original: Liu et al., 2020; Mai and Johansson, 2020; Defazio, 2020).
Revised: Liu et al., 2020; Mai and Johansson, 2020; Defazio, 2020).

Original: It is not easy to explicitly minimize the right-hand-side of (7) with respect to N .
Revised: It is not easy to explicitly minimize the right-hand-side of (7) with respect to N .

Original: However, N =  (log α T ) / 2  attempts to balance the last two terms and appears to be a good choice in practice.
Revised: However, N = (cid:98) (log α T ) / 2 (cid:99) attempts to balance the last two terms and appears to be a good choice in practice.

Original: The theorem (see (8) establishes an O (ln T/ √ T ) rate under step-decay bandwidth step-size.
Revised: The theorem (see (8) establishes an O (ln T/ √ T ) rate under step-decay bandwidth step-size.

Original: If M = m , which means that the step-size follows the boundary functions, we get a near-optimal (upto ln T ) rate for stochastic momentum with a step-decay step-size on nonconvex problems.
Revised: If M = m , which means that the step-size follows the boundary functions, we get a near-optimal (up to ln T ) rate for stochastic momentum with a step-decay step-size on non-convex problems.

Original: We believe that this is the ﬁrst near-optimal rate for stochastic momentum with step-decay step-size.
Revised: We believe that this is the ﬁrst near-optimal rate for stochastic momentum with step-decay step-size.

Original: The next result shows how an exponentially increasing stage-length allow to sharpen this guarantee even further.
Revised: The next result shows how an exponentially increasing stage-length allow to sharpen this guarantee even further.

Original: Theorem 4.3.
Revised: Theorem 4.3.

Original: Suppose the same setting as Lemma 4.1.
Revised: Suppose the same setting as Lemma 4.1.

Original: Consider Algorithm 2, if the functions δ ( t ) = 1 /α t − 1 with α > 1 , S t[-= S 
-]
Revised: Consider Algorithm 2, if the functions δ ( t ) = 1 /α t − 1 with α > 1 , S t[-= S 
-]

Original: {+= (cid:100) S 
+}
Revised: {+= (cid:100) S 
+}

Original: α t − 1  with S 0 = √ T , we have
Revised: α t − 1 (cid:101) with S 0 = √ T , we have

Original: [Equation]
Revised: [Equation]

Original: The stage length S t in Theorem 4.3 increases exponentially from S 1 = √ T  over N =  log α
Revised: The stage length S t in Theorem 4.3 increases exponentially from S 1 = (cid:100)√ T (cid:101) over N = (cid:98) log α

Original: (( α − 1) √ T + 1)  stages, resulting in an O (1 / √ T ) optimal rate for SGDM under the bandwidth-based step-decay scheme.
Revised: (( α − 1) √ T + 1) (cid:99) stages, resulting in an O (1 / √ T ) optimal rate for SGDM under the bandwidth-based step-decay scheme.

Original: This removes the ln T term of Theorem 8.
Revised: This removes the ln T term of Theorem 8.

Original: To the best of our knowledge, this work is the ﬁrst that is able to achieve an optimal rate for stochastic momentum with step-decay step-size in a general non-convex setting.
Revised: To the best of our knowledge, this work is the ﬁrst that is able to achieve an optimal rate for stochastic momentum with step-decay step-size in a general non-convex setting.

Original: Remark 4.4.
Revised: Remark 4.4.

Original: ( Better convergence than Liu et al.
Revised: ( Better convergence than Liu et al.

Original: We notice that reference Liu et al.
Revised: We notice that reference Liu et al.

Original: (2020) analyzes multi-stage momentum and obtains the bound
Revised: (2020) analyzes multi-stage momentum and obtains the bound

Original: [Equation]
Revised: [Equation]

Original: Here, ˜ x is a uniformly sampled iterate (unlike our results, which favour later iterates) and N is the number of stages.
Revised: Here, ˜ x is a uniformly sampled iterate (unlike our results, which favour later iterates) and N is the number of stages.

Original: The result uses a time-varying momentum parameter, whose value is determined by the step-size η t , and also assumes an inverse relationship between the step-size and stage-length, i.e. that η t S t is constant.
Revised: The result uses a time-varying momentum parameter, whose value is determined by the step-size η t , and also assumes an inverse relationship between the step-size and stage-length, i.e. that η t S t is constant.

Original: Hence, N is of O (log α T ) and the convergence guarantee in (9) is of
Revised: Hence, N is of O (log α T ) and the convergence guarantee in (9) is of

Original: O (1 / log α T ) , which is far worse than the rate of Theorem 4.2 and the optimal rate of Theorem 4.3.
Revised: O (1 / log α T ) , which is far worse than the rate of Theorem 4.2 and the optimal rate of Theorem 4.3.

Original:  N UMERICAL E XPERIMENTS
Revised:  N UMERICAL E XPERIMENTS

Original: In this section, we design and evaluate several speciﬁc step-size policies that belong to the bandwidthbased family.
Revised: In this section, we design and evaluate several speciﬁc step-size policies that belong to the bandwidthbased family.

Original: We consider SGD with and without momentum, and compare their performances on neural network training tasks on the CIFAR10 and CIFAR100 datasets.
Revised: We consider SGD with and without momentum, and compare their performances on neural network training tasks on the CIFAR10 and CIFAR100 datasets.

Original: 5. B ASELINES AND P ARAMETER S ELECTION FOR THE B ANDWIDTH S TEP -S IZES
Revised: 5. B ASELINES AND P ARAMETER S ELECTION FOR THE B ANDWIDTH S TEP -S IZES

Original: The bandwidth framework allows for a uniﬁed and streamlined (worst-case) analysis of all step-size policies that lie in the corresponding band.
Revised: The bandwidth framework allows for a uniﬁed and streamlined (worst-case) analysis of all step-size policies that lie in the corresponding band.

Original: Within this family, the band gives a lot of freedom in crating innovative step-size policies with additional advantages.
Revised: Within this family, the band gives a lot of freedom in crating innovative step-size policies with additional advantages.

Original: In particular, we will design a number of step-size policies that add periodic perturbations to a baseline step-size, attempting to both avoid bad local minima and to improve the local convergence properties.
Revised: In particular, we will design a number of step-size policies that add periodic perturbations to a baseline step-size, attempting to both avoid bad local minima and to improve the local convergence properties.

Original: The step-decay bandwidth step-sizes divide the total number of iterations into a small number of stages, in which the boundary functions are constant.
Revised: The step-decay bandwidth step-sizes divide the total number of iterations into a small number of stages, in which the boundary functions are constant.

Original: The width of the band are determined by the constants m and M .
Revised: The width of the band are determined by the constants m and M .

Original: We will explore step-sizes that add a decreasing perturbation within each stage, starting at the upper band at the beginning of the stage, ending at the lower bound at the end of the stage, and decaying as 1 /i , 1 / √ i , linearly or according to a cosine function.
Revised: We will explore step-sizes that add a decreasing perturbation within each stage, starting at the upper band at the beginning of the stage, ending at the lower bound at the end of the stage, and decaying as 1 /i , 1 / √ i , linearly or according to a cosine function.

Original: As baseline, we consider the step-decay step-size that follows the lower boundary function mδ ( t ) .
Revised: As baseline, we consider the step-decay step-size that follows the lower boundary function mδ ( t ) .

Original: To use the same maximum value for the bandwidth step-sizes, we do not add any perturbation in the ﬁrst stage; cf.
Revised: To use the same maximum value for the bandwidth step-sizes, we do not add any perturbation in the ﬁrst stage; cf.

Original: Figure 5.
Revised: Figure 5.

Original: For the 1 / √ t -band, on the other hand, stages correspond to epochs and perturbing the step-size within a stage would be too frequent and lead to bias.
Revised: For the 1 / √ t -band, on the other hand, stages correspond to epochs and perturbing the step-size within a stage would be too frequent and lead to bias.

Original: Rather, we choose to add similar perturbations 
Revised: Rather, we choose to add similar perturbations as for the step-decay band, but adjust the perturbation between stages. In our experiments, the two

Original: [Table]
Revised: [Table]

Original: as for the step-decay band, but adjust the perturbation between stages.

Original: In our experiments, the two step-size policies perform roughly the same number of periods of perturbations over the training set.
Revised: step-size policies perform roughly the same number of periods of perturbations over the training set.

Original: As baseline, we consider the step-size η ti = m/ √ t .
Revised: As baseline, we consider the step-size η ti = m/ √ t .

Original: In all experiments, the hyper-parameters (e.g., m and M ) have been determined using grid search, see Section E.2 for details.
Revised: In all experiments, the hyper-parameters (e.g., m and M ) have been determined using grid search, see Section E.2 for details

Original:  5. B ANDWIDTH S CHEDULE H ELPS TO A VOID L OCAL M INI M A
Revised: . 5. B ANDWIDTH S CHEDULE H ELPS TO A VOID L OCAL M INI M A

Original: To demonstrate the potential beneﬁts of bandwidth-based non-monotonic step-size schedules, we consider the toy example (see Section E.3 for details and further results) from (Shi et al., 2020), which is non-convex and has four local minima 2 ; see Figure 2.
Revised: To demonstrate the potential beneﬁts of bandwidth-based non-monotonic step-size schedules, we consider the toy example (see Section E.3 for details and further results) from (Shi et al., 2020), which is non-convex and has four local minima 2 ; see Figure 2.

Original: We then compare the ﬁnal iterates of
Revised: We then compare the ﬁnal iterates of

Original: SGD with constant step-sizes (both large and small), step-decay, and a bandwidth-based step-decay step size which we call linear-mode (illustrated in Figure 5).
Revised: SGD with constant step-sizes (both large and small), step-decay, and a bandwidth-based step-decay step size which we call linear-mode (illustrated in Figure 5).

Original: As shown in Figure 2, a large constant step-size more easily escapes the bad local minima to approach the global minimum at (0 . 7 , − 0 .
Revised: As shown in Figure 2, a large constant step-size more easily escapes the bad local minima to approach the global minimum at (0 . 7 , − 0 .

Original: than a small constant step-size.
Revised: than a small constant step-size.

Original: However, with a large constant step-size, the ﬁnal iterates are scattered and end up far from the global minimum, which also has been observed in Figure 5 of (Shi et al., 2020).
Revised: However, with a large constant step-size, the ﬁnal iterates are scattered and end up far from the global minimum, which also has been observed in Figure 5 of (Shi et al., 2020).

Original: Therefore, we have to reduce the step-size at some points to reduce the error.
Revised: Therefore, we have to reduce the step-size at some points to reduce the error.

Original: This is exactly the intuition of step-decay step-size.
Revised: This is exactly the intuition of step-decay step-size.

Original: As shown in Figure 2, the scatter plots of SGD with step-decay (red) and step-decay with linear-mode (green) are more concentrated around the global minimum than the constant step-sizes.
Revised: As shown in Figure 2, the scatter plots of SGD with step-decay (red) and step-decay with linear-mode (green) are more concentrated around the global minimum than the constant step-sizes.

Original: To quantify the ability of different step-sizes to avoid the local minima, Table 1 reports the percentage of the ﬁnal iterates under the different step-size policies that are close to each minima.
Revised: To quantify the ability of different step-sizes to avoid the local minima, Table 1 reports the percentage of the ﬁnal iterates under the different step-size policies that are close to each minima.

Original: We can see that the ability of the step-decay policy (named baseline ) to escape the local minima is slightly worse than the large constant step-size, but Figure 2 shows that the variance of the near-optimal iterates is reduced signiﬁcantly.
Revised: We can see that the ability of the step-decay policy (named baseline ) to escape the local minima is slightly worse than the large constant step-size, but Figure 2 shows that the variance of the near-optimal iterates is reduced signiﬁcantly.

Original: In a similar way, we can see that linear-mode not only improves the ability to escape the local minima, but also produces ﬁnal iterates that are more concentrated around the global optimum.
Revised: In a similar way, we can see that linear-mode not only improves the ability to escape the local minima, but also produces ﬁnal iterates that are more concentrated around the global optimum.

Original: Hence, it appears (at least in this example) that non-monotonic step-size schedules allow
Revised: Hence, it appears (at least in this example) that non-monotonic step-size schedules allow

Original: SGD to escape local minima and produce ﬁnal iterates of high quality.
Revised: SGD to escape local minima and produce ﬁnal iterates of high quality.

Original: 5. N UMERICAL R ESULTS ON CIFAR10 AND CIFAR 
Revised: 5. N UMERICAL R ESULTS ON CIFAR10 AND CIFAR

Original: To illustrate the practical performance of the bandwidth-based step-sizes, we choose the well-known CIFAR10 and CIFAR100 (Kri) image classiﬁcation datasets.
Revised: To illustrate the practical performance of the bandwidth-based step-sizes, we choose the wellknown CIFAR10 and CIFAR100 (Krizhevsky, 2009) image classiﬁcation datasets.

Original: We consider the benchmark experiments of CIFAR10 on ResNet-18 (He et al., 2016) and CIFAR100 on a 28 × 10 wide residual network (WRN-28-10) (Zagoruyko and Komodakis, 2016), respectively.
Revised: We consider the benchmark experiments of CIFAR10 on ResNet-18 (He et al., 2016) and CIFAR100 on a 28 × 10 wide residual network (WRN-28-10) (Zagoruyko and Komodakis, 2016), respectively.

Original: All the experiments are repeatedtimes to eliminate the inﬂuence of randomness.
Revised: All the experiments are repeated 5 times to eliminate the inﬂuence of randomness.

Original: We begin by evaluating our step-sizes for SGD.
Revised: We begin by evaluating our step-sizes for SGD.

Original: The left column of Figure 3 present the results of the 1 / √ t -band step-sizes on the two datasets.
Revised: The left column of Figure 3 present the results of the 1 / √ t -band step-sizes on the two datasets.

Original: As shown in Figure 5, these stepsizes are all non-monotonic.
Revised: As shown in Figure 5, these stepsizes are all non-monotonic.

Original: The sudden increase in the step-size leads to a corresponding cliff-like reduction in accuracy followed by a recovery phase that consistently ends up at a better performance than in the previous stage.
Revised: The sudden increase in the step-size leads to a corresponding cliff-like reduction in accuracy followed by a recovery phase that consistently ends up at a better performance than in the previous stage.

Original: The three 1 / √ t -band step-sizes achieve signiﬁcant improvements compared to their baseline ( η ti = m/ √ t ), in terms of both test loss and test accuracy.
Revised: The three 1 / √ t -band step-sizes achieve signiﬁcant improvements compared to their baseline ( η ti = m/ √ t ), in terms of both test loss and test accuracy.

Original: Moreover, the linear-mode performs the best compared to other polynomial decaying modes.
Revised: Moreover, the linear-mode performs the best compared to other polynomial decaying modes.

Original: Then, the results of SGD with step-decay band (described in Section 5.1 or see Figure 5 in Appendix) on CIFAR10 and CIFAR 
Revised: Then, the results of SGD with step-decay band (described in Section 5.1 or see Figure 5 in Appendix) on CIFAR10 and CIFAR are given in Figure 3 (middle column), respectively. At the ﬁnal stage, the bandwidth step-sizes

Original: [Figure]
Revised: [Figure]

Original: are given in Figure 3 (middle column), respectively.

Original: At the ﬁnal stage, the bandwidth step-sizes improve both test loss (see Figure 6 in Appendix) and test accuracy compared to the baseline.
Revised: improve both test loss (see Figure 6 in Appendix) and test accuracy compared to the baseline.

Original: In particular, the cosine-mode performs the best on this problem.
Revised: In particular, the cosine-mode performs the best on this problem.

Original: In the second stage, baseline methods have a sharp boost.
Revised: In the second stage, baseline methods have a sharp boost.

Original: Our guess is that the noise accumulates quickly under a relatively large constant step-size.
Revised: Our guess is that the noise accumulates quickly under a relatively large constant step-size.

Original: But this phenomenon is only temporary.
Revised: But this phenomenon is only temporary.

Original: When we drop the step-size in the third stage, the performance improves.
Revised: When we drop the step-size in the third stage, the performance improves.

Original: Next, we evaluate the performance of step-decay bandwidth step-sizes on SGDM.
Revised: Next, we evaluate the performance of step-decay bandwidth step-sizes on SGDM.

Original: The results are reported in Figure 3 (right column).
Revised: The results are reported in Figure 3 (right column).

Original: The ﬁrst observation from Figure 3 (right column) is that the step-decay bandwidth step-sizes also work well for SGDM, and that again, the cosine-mode performs better than the others.
Revised: The ﬁrst observation from Figure 3 (right column) is that the step-decay bandwidth step-sizes also work well for SGDM, and that again, the cosine-mode performs better than the others.

Original: Another interesting observation is that the performance of vanilla SGD with cosine-mode (red) in Figure 3 is comparable to (even better than) SGDM with the baseline step-decay step-size (black) in Figure 3.
Revised: Another interesting observation is that the performance of vanilla SGD with cosine-mode (red) in Figure 3 is comparable to (even better than) SGDM with the baseline step-decay step-size (black) in Figure 3.

Original: A similar conclusion can also be made on CIFAR100.
Revised: A similar conclusion can also be made on CIFAR100.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have studied a general family of bandwidth step-sizes for non-convex optimization.
Revised: We have studied a general family of bandwidth step-sizes for non-convex optimization.

Original: The family speciﬁes a globally decaying band in which the actual step-size is allowed to vary, and includes both stage-wise and continuously decaying step-size policies as special cases.
Revised: The family speciﬁes a globally decaying band in which the actual step-size is allowed to vary, and includes both stage-wise and continuously decaying step-size policies as special cases.

Original: We have derived convergence rate guarantees for SGD and SGDM under all step-size policies in two important classes of bandwidth step-sizes ( 1 / √ t and step-decay), some of which are optimal.
Revised: We have derived convergence rate guarantees for SGD and SGDM under all step-size policies in two important classes of bandwidth step-sizes ( 1 / √ t and step-decay), some of which are optimal.

Original: Our results provide theoretical guarantees for several popular “cyclical” step-sizes (Loshchilov and Hutter, 2017; Smith, 2017), as long as they are tuned to lie within our bands.
Revised: Our results provide theoretical guarantees for several popular “cyclical” step-sizes (Loshchilov and Hutter, 2017; Smith, 2017), as long as they are tuned to lie within our bands.

Original: We have also designed a number of novel step-sizes that add periodic perturbations to the global trend in order to avoid bad local minima and to improve the local convergence properties.
Revised: We have also designed a number of novel step-sizes that add periodic perturbations to the global trend in order to avoid bad local minima and to improve the local convergence properties.

Original: These step-sizes were shown to have superior practical performance in neural network training tasks on the CIFAR data set.
Revised: These step-sizes were shown to have superior practical performance in neural network training tasks on the CIFAR data set.

Original: In the analysis of SGDM, we assume that the stochastic gradient is bounded (see Assumption 2(b)).
Revised: In the analysis of SGDM, we assume that the stochastic gradient is bounded (see Assumption 2(b)).

Original: It is interesting to see how to relax this assumption in some special cases, for example, when the step-size is constant throughout each stage.
Revised: It is interesting to see how to relax this assumption in some special cases, for example, when the step-size is constant throughout each stage.

Original: It would also be interesting to see if the bandwidth framework could be specialized to a more narrow class of step-sizes, for which we can provide even stronger convergence rates.
Revised: It would also be interesting to see if the bandwidth framework could be specialized to a more narrow class of step-sizes, for which we can provide even stronger convergence rates.

Original: A C ONVERGENCE FOR B ANDWIDTH P OLYNOMIAL -D ECAY S TEP -S IZES
Revised: A C ONVERGENCE FOR B ANDWIDTH P OLYNOMIAL -D ECAY S TEP -S IZES

Original: Our ﬁrst more speciﬁc result considers the 1 / √ t bandwidth step-size with ﬁxed stage length.
Revised: Our ﬁrst more speciﬁc result considers the 1 / √ t bandwidth step-size with ﬁxed stage length.

Original: Theorem A.1.
Revised: Theorem A.1.

Original: Under Assumptions 1 and 2(a), and assume that there exists a constant ∆ 0 >such that E [ f ( x t 1 ) − f ∗ ] ≤ ∆ 0 for each t ≥ 1 .
Revised: Under Assumptions 1 and 2(a), and assume that there exists a constant ∆ 0 >such that E [ f ( x t 1 ) − f ∗ ] ≤ ∆ 0 for each t ≥ 1 .

Original: If the step-size η ti ≤ 1 / (( ρ + 1) L ) , the stage length
Revised: If the step-size η ti ≤ 1 / (( ρ + 1) L ) , the stage length

Original: S t =
Revised: S t =

Original: S ≥ 1 , and the boundary function δ ( t ) = 1 / √ t for each 1 ≤ t ≤ N , we have
Revised: S ≥ 1 , and the boundary function δ ( t ) = 1 / √ t for each 1 ≤ t ≤ N , we have

Original: [Equation]
Revised: [Equation]

Original: Theorem A.1 shows how multi-stage SGD with polynomial-decay bandwidth step-sizes converges to a stationary point.
Revised: Theorem A.1 shows how multi-stage SGD with polynomial-decay bandwidth step-sizes converges to a stationary point.

Original: In the extreme case that S = 1 , the step-size reduces to m/ √ t ≤ η t 1 ≤ M/ √ t and our result is comparable to the non-asymptotic optimal rate derived for m =
Revised: In the extreme case that S = 1 , the step-size reduces to m/ √ t ≤ η t 1 ≤ M/ √ t and our result is comparable to the non-asymptotic optimal rate derived for m =

Original: M = η 0 in (Wang et al., 2021, Theorem 3.5).
Revised: M = η 0 in (Wang et al., 2021, Theorem 3.5).

Original: Multi-stage vs traditional 1 / √ t step-size ( S = 1 ) In general, during the initial iterations when the ﬁrst term of (10) dominates the error bound, the multi-stage technique can accelerate the convergence by a larger step-size and longer inner-loop S .
Revised: Multi-stage vs traditional 1 / √ t step-size ( S = 1 ) In general, during the initial iterations when the ﬁrst term of (10) dominates the error bound, the multi-stage technique can accelerate the convergence by a larger step-size and longer inner-loop S .

Original: However, a large S will make the error bound worse when the noise term begins to dominate the bound.
Revised: However, a large S will make the error bound worse when the noise term begins to dominate the bound.

Original: The next theorem analyzes an algorithm with a decreasing stage length.
Revised: The next theorem analyzes an algorithm with a decreasing stage length.

Original: Theorem A.2.
Revised: Theorem A.2.

Original: Under Assumptions 1 and 2(a),and assume that there exists a constant ∆ 0 >such that E [ f ( x t 1 ) − f ∗ ] ≤ ∆ 0 for each t ≥ 1 .
Revised: Under Assumptions 1 and 2(a),and assume that there exists a constant ∆ 0 >such that E [ f ( x t 1 ) − f ∗ ] ≤ ∆ 0 for each t ≥ 1 .

Original: If the step-size η ti ≤ 1 / (( ρ + 1) L ) , the stage length
Revised: If the step-size η ti ≤ 1 / (( ρ + 1) L ) , the stage length

Original: S t =  S 0 / √ t  with S 0 = √ T , and δ ( t ) = 1 / √ t for each 1 ≤ t ≤ N , we have
Revised: S t = (cid:100) S 0 / √ t (cid:101) with S 0 = √ T , and δ ( t ) = 1 / √ t for each 1 ≤ t ≤ N , we have

Original: [Equation]
Revised: [Equation]

Original: Schedule of Theorem A.2 vs Chen et al.
Revised: Schedule of Theorem A.2 vs Chen et al.

Original: The theorem establishes an optimal rate for multistage SGD with 1 / √ t bandwidth step-size.
Revised: The theorem establishes an optimal rate for multistage SGD with 1 / √ t bandwidth step-size.

Original: Note that Chen et al.
Revised: Note that Chen et al.

Original: (2019) also analyzes a stagewise algorithm with varying stage length, but their step-size decays as 1 /t and stage length increases with t .
Revised: (2019) also analyzes a stagewise algorithm with varying stage length, but their step-size decays as 1 /t and stage length increases with t .

Original: An important novelty with our result is that it uses a long initial stage, S 1 = √ T while a large stage length in Chen et al.
Revised: An important novelty with our result is that it uses a long initial stage, S 1 = √ T while a large stage length in Chen et al.

Original: (2019) requires a small initial step-size (of O (1 / √ T )) .
Revised: (2019) requires a small initial step-size (of O (1 / √ T )) .

Original: Figure 4 illustrates the performance of different step-size policies: 1) 1 / √ t with S t = 1 ; 2) 1 / √ t with S t =  n/b  where n is total sample size and b is the batch size; 3) 1 / √ t with time-decreasing S t = 
Revised: Figure 4 illustrates the performance of different step-size policies: 1) 1 / √ t with S t = 1 ; 2) 1 / √ t with S t = (cid:100) n/b (cid:101) where n is total sample size and b is the batch size; 3) 1 / √ t with time-decreasing S t = (cid:100) S 0 / √ t (cid:101) and

Original: S 0/ √ t and S 0 = √ T ; 4) and 1 /t step-size with S t = S 0 t and S 

Revised: S 0 = √ T ; 4) and 1 /t step-size with S t = S 0 t and S

Original: = 12 from Chen et al.
Revised: = 12 from Chen et al.

Original: We can see that the step-size policies proposed in Theorem A.2 are more stable and perform the best.
Revised: We can see that the step-size policies proposed in Theorem A.2 are more stable and perform the best.

Original: For completeness, we also compare the performance of step-size schedules proposed by Theorems 3.1 and 3.3 in Figure 4 (right).
Revised: For completeness, we also compare the performance of step-size schedules proposed by Theorems 3.1 and 3.3 in Figure 4 (right).

Original: Although step-decay with time-increasing stage length has a superior theoretical convergence guarantee, constant stage length performs better in this particular example.
Revised: Although step-decay with time-increasing stage length has a superior theoretical convergence guarantee, constant stage length performs better in this particular example.

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Original: B P ROOFS OF L EMMA AND T HEOREMS IN S ECTION
Revised: B P ROOFS OF L EMMA AND T HEOREMS IN S ECTION

Original: Lemma B.1.
Revised: Lemma B.1.

Original: Suppose that Assumption 1 and Assumption 2(a) hold.
Revised: Suppose that Assumption 1 and Assumption 2(a) hold.

Original: If we run the Algorithm 1 with
Revised: If we run the Algorithm 1 with

Original: T > 1 and η ti ≤ 1 / (( ρ + 1) L ) , we have
Revised: T > 1 and η ti ≤ 1 / (( ρ + 1) L ) , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: ( of Lemma B.1 )
Revised: ( of Lemma B.1 )

Original: The L -smoothness of f (see Assumption 1),
Revised: The L -smoothness of f (see Assumption 1),

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e., (cid:107)∇
Revised: e., (cid:107)∇

Original: [Equation]
Revised: [Equation]

Original: ( x ) − ∇ f ( y ) (cid:107) ≤
Revised: ( x ) − ∇ f ( y ) (cid:107) ≤

Original: L (cid:107) x − y (cid:107) for all x, y ∈ dom ( f ) implies that
Revised: L (cid:107) x − y (cid:107) for all x, y ∈ dom ( f ) implies that

Original: [Equation]
Revised: [Equation]

Original: Applying the L -smoothness property of f and recalling Algorithm 1 at current iterate x ti , we have
Revised: Applying the L -smoothness property of f and recalling Algorithm 1 at current iterate x ti , we have

Original: [Equation]
Revised: [Equation]

Original: Taking conditional expectation of F ti on the above inequality and due to the unbiased estimator g ti such that E [ g ti | F ti ] = ∇ f ( x ti ) , we obtain that
Revised: Taking conditional expectation of F ti on the above inequality and due to the unbiased estimator g ti such that E [ g ti | F ti ] = ∇ f ( x ti ) , we obtain that

Original: [Equation]
Revised: [Equation]

Original: By Assumption 2(a) that E [ (cid:107) g ti − ∇ f ( x ti ) (cid:107) 2 | F ti ] ≤ ρ (cid:107)∇ f ( x ti ) (cid:107) 

Revised: By Assumption 2(a) that E [ (cid:107) g ti − ∇ f ( x ti ) (cid:107) 2 | F ti ] ≤ ρ (cid:107)∇ f ( x ti ) (cid:107)

Original: +
Revised: +

Original: σ
Revised: σ

Original: , we have
Revised: , we have

Original: [Equation]
Revised: [Equation]

Original: Then incorporating the above inequality into (13) gives
Revised: Then incorporating the above inequality into (13) gives

Original: [Equation]
Revised: [Equation]

Original: If step-size η ti ≤ 1 / (( ρ + 1) L ) , we have − η ti +
Revised: If step-size η ti ≤ 1 / (( ρ + 1) L ) , we have − η ti +

Original: [Equation]
Revised: [Equation]

Original: η ti )
Revised: η ti )

Original: [Equation]
Revised: [Equation]

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: +1)≤ − η ti / 2 .
Revised: +1)≤ − η ti / 2 .

Original: For any t ≥ 1 , the inequality (15) can be estimated as:
Revised: For any t ≥ 1 , the inequality (15) can be estimated as:

Original: [Equation]
Revised: [Equation]

Original: Applying the assumption of step-size that η ti = n ( t, i ) δ ( t ) with m ≤ n ( t, i ) ≤ M for all t ∈ { 1 , 2 , · · · , N } to (16) gives
Revised: Applying the assumption of step-size that η ti = n ( t, i ) δ ( t ) with m ≤ n ( t, i ) ≤ M for all t ∈ { 1 , 2 , · · · , N } to (16) gives

Original: [Equation]
Revised: [Equation]

Original: and dividing mδ ( t ) 2 / 2 into the both sides, we have
Revised: and dividing mδ ( t ) 2 / 2 into the both sides, we have

Original: [Equation]
Revised: [Equation]

Original: Recalling the output ˆ x T of Algorithm 1 and then taking expectation, we have
Revised: Recalling the output ˆ x T of Algorithm 1 and then taking expectation, we have

Original: [Equation]
Revised: [Equation]

Original: Applying (18) recursively from i = 1 to S t and using the fact that x t +
Revised: Applying (18) recursively from i = 1 to S t and using the fact that x t +

Original: [Equation]
Revised: [Equation]

Original: x tS t +1 , the sum of (19) for t ≥ 1 can be estimated as
Revised: tS t [Equation] 1 , the sum of (19) for t ≥ 1 can be estimated as

Original: [Equation]
Revised: [Equation]

Original: Then plugging (20) into (19), we get
Revised: Then plugging (20) into (19), we get

Original: [Equation]
Revised: [Equation]

Original: as desired.
Revised: as desired.

Original: Proof.
Revised: Proof.

Original: ( of Theorem A.1 )
Revised: ( of Theorem A.1 )

Original: In this case, the step size η ti satisﬁes that m/ √ t ≤ η ti ≤ M/ √ t , where δ ( t ) = 1 / √ t .
Revised: In this case, the step size η ti satisﬁes that m/ √ t ≤ η ti ≤ M/ √ t , where δ ( t ) = 1 / √ t .

Original: By the deﬁnition of δ ( t ) , we have
Revised: By the deﬁnition of δ ( t ) , we have

Original: [Equation]
Revised: [Equation]

Original: Applying the assumption that E [ f ( x t 1 ) − f ∗ ] ≤ ∆ 0 where f ∗ = min x f ( x ) and the deﬁnition of δ ( t ) , we have
Revised: Applying the assumption that E [ f ( x t 1 ) − f ∗ ] ≤ ∆ 0 where f ∗ = min x f ( x ) and the deﬁnition of δ ( t ) , we have

Original: [Equation]
Revised: [Equation]

Original: Under Assumptions 1 and 2(a) and η ti < 1 / (( ρ + 1) L ) , thus Lemma B.1 holds.
Revised: Under Assumptions 1 and 2(a) and η ti < 1 / (( ρ + 1) L ) , thus Lemma B.1 holds.

Original: Incorporating these inequalities into Lemma B.1 gives
Revised: Incorporating these inequalities into Lemma B [Equation] 1 gives

Original: [Equation]
Revised: [Equation]

Original: Then the proof is ﬁnished
Revised: Then the proof is ﬁnished

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: ( of Theorem A.2 )
Revised: ( of Theorem A.2 )

Original: In this theorem, we consider SGD with the 1 / √ t bandwidth step-size, i.e., m √ t ≤ η ti ≤ M √ t for 1 ≤ t ≤ N , where the stage length S t =
Revised: In this theorem [Equation] we consider SGD with the 1 / √ t bandwidth step-size, i

Original:  S 0 / √ t  with S 0 = √ T , and the boundary function δ ( t ) = 1 / √ t .
Revised: [Equation] e., m √ t ≤ η ti ≤ M √ t for 1 ≤ t ≤ N , where the stage length S t = (cid:100) S 0 / √ t (cid:101) with S 0 = √ T , and the boundary function δ ( t ) = 1 / √ t .

Original: By the relationship that (cid:80)
Revised: By the relationship that (cid:80)

Original: Nt =1 S t = T  , we get that 
Revised: Nt =1 S t = T and x ≤ (cid:100) x (cid:101) ≤ x + 1 for all x , we get that [Equation]

Original: [Equation]
Revised: [Equation]

Original: Under Assumptions 1 and 2(a) and η ti < 1 / (( ρ + 1) L ) , thus Lemma B.1 holds.
Revised: Under Assumptions 1 and 2(a) and η ti < 1 / (( ρ + 1) L ) , thus Lemma B.1 holds.

Original: Following the same process as Theorem A.1, the inequality (22) also holds, that is
Revised: Following the same process as Theorem A.1, the inequality (22) also holds, that is

Original: N(cid:88)
Revised: N

Original: [Equation]
Revised: [Equation]

Original: Then incorporating the above inequality to Lemma B [Equation] 1, we have
Revised: Then incorporating the above inequalities to Lemma B.1, we have

Original: [Equation]
Revised: [Equation]

Original: which concludes the proof
Revised: which concludes the proof

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: ( of Theorem 3.1 )
Revised: ( of Theorem 3.1 )

Original: In this case [Equation] the step-size η ti exponentially decays every S iterations.
Revised: In this case, the step-size η ti exponentially decays every S iterations.

Original: By the deﬁnition of η ti
Revised: By the deﬁnition of η ti

Original: [Equation]
Revised: [Equation]

Original: we have δ ( t ) = 1 /α t − 1 for all 1 ≤ t ≤ N .
Revised: we have δ ( t ) = 1 /α t − 1 for all 1 ≤ t ≤ N .

Original: The sum of 1 /δ ( t ) can be estimated as
Revised: The sum of 1 /δ ( t ) can be estimated as

Original: [Equation]
Revised: [Equation]

Original: Recalling the assumption that E [ f ( x t 1 ) − f ∗ ] ≤ ∆ 0 for all t ≥ 1 , we have
Revised: Recalling the assumption that E [ f ( x t 1 ) − f ∗ ] ≤ ∆ 0 for all t ≥ [Equation]

Original:  N(cid:88)
Revised: we have N

Original: [Equation]
Revised: [Equation]

Original: Under Assumptions 1 and 2(a) and η ti < 1 / (( ρ + 1) L )
Revised: Under Assumptions 1 and 2(a) and η ti < 1 / (( ρ + 1) L )

Original: [Equation]
Revised: [Equation]

Original: thus Lemma B.1 holds.
Revised: thus Lemma B.1 holds.

Original: Then applying the result of Lemma B.1 and incorporating the above inequalities into Lemma B.1 gives
Revised: Then applying the result of Lemma B.1 and incorporating the above inequalities into Lemma B.1 gives

Original: [Equation]
Revised: [Equation]

Original: Substituting the speciﬁc values of N =  (log α T ) / 
Revised: Substituting the speciﬁc values of N = (cid:98) (log α T ) / 2 (cid:99) , S =

Original: [Equation]{+(cid:100) 
+}
Revised: {+(cid:100) 
+}

Original: S = 2 T/ log α T  into the above inequality
Revised: T/ log α T (cid:101) into the above inequality

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: then transform the base log α to the natural logarithm ln , we can get the desired result.
Revised: then transform the base log α to the natural logarithm ln , we can get the desired result

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: ( of Theorem 3.3 )
Revised: ( of Theorem 3.3 )

Original: If S 0 = √ T and S t =  S 

Revised: If S 0 = √ T and S t = (cid:100) S

Original: α t − 1  , the stage lengthN is log α
Revised: α t − 1 (cid:101) , by (cid:80) Nt =1 S t = T , the stage length

Original:  (( α − 1) √ T +1) .
Revised: N is (cid:98) log α (( α − 1) √ T + 1) (cid:99) .

Original: In this case, we have δ ( t ) = 1 /α t − 1 for each t ∈ [ N ] .
Revised: In this case, we have δ ( t ) = 1 /α t − 1 for each t ∈ [ N ] .

Original: Under Assumptions 1 and 2(a) and η ti < 1 / (( ρ + 1) L ) , thus Lemma B.1 holds.
Revised: Under Assumptions 1 and 2(a) and η ti < 1 / (( ρ + 1) L ) , thus Lemma B.1 holds.

Original: Before applying the result of Lemma B.1, we ﬁrst give the following estimations:
Revised: Before applying the result of Lemma B.1, we ﬁrst give the following estimations:

Original: [Equation]
Revised: [Equation]

Original: 1: Input: initial point x 11 ∈ R d , v 11 = 0 , # iterations T , # stages N , stage length { S t } Nt = such that (cid:80)
Revised: 1: Input: initial point x 11 ∈ R d , v 11 = 0 , # iterations T , # stages N , stage length { S t } Nt = such that (cid:80)

Original: Nt =1 S t = T , momentum parameter β ∈ (0 , 1) , the sequences { δ ( t ) }
Revised: Nt =1 S t = T , momentum parameter β ∈ (0 , 1) , the sequences { δ ( t ) }

Original: Nt =1 and (cid:110) { n ( t, i ) } S t i =1 (cid:111)
Revised: Nt =1 and (cid:110) { n ( t, i ) } S t i =1 (cid:111)

Original: N t =1 ∈ [ m, M ] with 0 < m ≤ M 2: for t = 1 : N do 3: for i = 1 : S t do 4:
Revised: N t =1 ∈ [ m, M ] with 0 < m ≤ M 2: for t = 1 : N do 3: for i = 1 : S t do 4:

Original: Query a stochastic gradient oracle O at x ti to get a vector g ti such that E [ g ti | F ti ] = ∇ f ( x ti ) 5:
Revised: Query a stochastic gradient oracle O at x ti to get a vector g ti such that E [ g ti | F ti ] = ∇ f ( x ti ) 5:

Original: Update step-size η ti = n ( t, i ) δ ( t ) , which belongs to the interval [ mδ ( t ) , Mδ ( t )] 6: v
Revised: Update step-size η ti = n ( t, i ) δ ( t ) , which belongs to the interval [ mδ ( t ) , Mδ ( t )] 6: v

Original: ti +

Revised: ti +

Original: = βv ti + (1 − β )
Revised: = βv ti + (1 − β )

Original: g ti 7: x ti +

Revised: g ti 7: x ti +

Original: = x ti − η ti v ti + 8: end for 9: v t + = v tS t +1 and x t + = x tS t +1 10: end for 11: Return: ˆ x T is uniformly chosen from (cid:8) x t ∗ 1 , x t ∗ 2 , · · · , x t ∗ S t ∗ (cid:9) , where the integer t ∗ is randomly chosen from { 1 , 2 , · · · , N } with probability P t =
Revised: = x ti − η ti v ti + 8: end for 9: v t + = v tS t +1 and x t + = x tS t +1 10: end for 11: Return: ˆ x T is uniformly chosen from (cid:8) x t ∗ 1 , x t ∗ 2 , · · · , x t ∗ S t ∗ (cid:9) , where the integer t ∗ is randomly chosen from { 1 , 2 , · · · , N } with probability P t =

Original: δ − 1 ( t ) /
Revised: δ − 1 ( t ) /

Original: ( (cid:80) Nl =

Revised: ( (cid:80) Nl =

Original: δ − 1 ( l )) and also
Revised: δ − 1 ( l )) and also

Original: N (cid:88) t = 2( E [ f ( x t 1 )] − E [ f ( x t +11 )]) mδ ( t )≤m (cid:32) ( α 2 − 1)
Revised: N (cid:88) t = 2( E [ f ( x t 1 )] − E [ f ( x t +11 )]) mδ ( t )≤m (cid:32) ( α 2 − 1)

Original: N (cid:88) t
Revised: N (cid:88) t

Original: = α 2( t − 2) E
Revised: = α 2( t − 2) E

Original: [ f ( x t 1 ) − f ∗ ] + ∆(cid:33)
Revised: [ f ( x t 1 ) − f ∗ ] + ∆(cid:33)

Original: [Equation]
Revised: [Equation]

Original: Applying these results into Lemma B.1, we have
Revised: Applying these results into Lemma B.1, we have

Original: [Equation]
Revised: [Equation]

Original: Thus the proof is complete.
Revised: Thus the proof is complete.

Original: C P ROOFS OF L EMMA AND T HEOREMS IN S ECTION
Revised: C P ROOFS OF L EMMA AND T HEOREMS IN S ECTION

Original: We recall the momentum scheme of Algorithm 2 below
Revised: We recall the momentum scheme of Algorithm 2 below

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original:  where β ∈ (0 , 1) .
Revised: 27) where β ∈ (0 , 1) .

Original: Before given the proofs, we introduce an extra variable z t i = 
Revised: Before giving the proofs, we introduce an extra variable z ti = x

Original: x ti 1 − β − β 1 − β x t i − 1 , then
Revised: ti 1 − β − β 1 − β x ti − 1 , then

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: However, the bandwidth-based step-size η ti in our analysis is time dependent and also possibly non-monotonic, so the commonly used equalities x ti +

Revised: However, the bandwidth-based step-size η ti in our analysis is time dependent and also possibly non-monotonic, so the commonly used equalities x ti +

Original: = x ti − ηg ti + β ( x ti − x ti − [Equation] Yan et al [Equation]{+al., 2018) or z ti +
+}
Revised: = x ti − ηg ti + β ( x ti − x ti − 1 ) (Yan et {+al., 2018) or z ti +
+}

Original: [-2018) or z ti +
-]
Revised: [-2018) or z ti +
-]

Original: =
Revised: =

Original: z ti − η t g ti (see lemma 3 of Liu et al.
Revised: z ti − η t g ti (see lemma 3 of Liu et al.

Original: (2020)) do not hold in our analysis.
Revised: (2020)) do not hold in our analysis.

Original: This signiﬁcantly increases the level of difﬁculty of the analysis.
Revised: This signiﬁcantly increases the level of difﬁculty of the analysis.

Original: The results of Lemma 4.1 is based on a sequence of lemmas introduced below.
Revised: The results of Lemma 4.1 is based on a sequence of lemmas introduced below.

Original: Lemma C.1.
Revised: Lemma C.1.

Original: Suppose that the objective function f satisﬁes Assumption 1.
Revised: Suppose that the objective function f satisﬁes Assumption 1.

Original: At each stage t , the step-size η ti is monotonically decreasing.
Revised: At each stage t , the step-size η ti is monotonically decreasing.

Original: Then, for i ≥ 2 , we have
Revised: Then, for i ≥ 2 , we have

Original: [Equation]
Revised: [Equation]

Original: ) + β 1 − β (cid:18) 1 − η ti η ti −(cid:19) (cid:0) f

Original: ( x ti ) − f ( x ti − 1 ) (cid:1) ≤ − η t

Original: i (cid:13)(cid:13) ∇ f ( x t i ) (cid:13)(cid:13) 2 + βL 2(1 − β ) 2 (cid:13)(cid:13)

Original: x

Original: t i − x t i − 1 (cid:13)(cid:13) 2 +

Original: L 2(1 − β ) 2 E

Original: [ (cid:13)(cid:13) x

Original: t i +1 − x t i (cid:13)(cid:13) 2 | F t i ] .

Original: Proof.
Revised: Proof.

Original: Using the L -smoothness of f (Assumption 1) and taking conditional expectation gives
Revised: Using the L -smoothness of f (Assumption 1) and taking conditional expectation gives

Revised: E [ f ( z ti +1 ) | F ti ]

Revised: [Equation]

Revised: ti

Original: [Equation]
Revised: [Equation]

Revised: Re-using the L -smoothness property of f at z ti and x ti gives

Original: [Equation]
Revised: [Equation]

Original: Then, combining the two inequalities above, we have
Revised: Then, combining the two inequalities above

Original: [Equation]
Revised: [Equation]

Revised: we have

Revised: [Equation]

Revised: [Equation]

Original: The step-size η ti for each stage is monotonically decreasing, i.e. η ti ≤ η ti − 1 for i ≥ 2 , so (cid:16) 1 −η ti η ti − 1 (cid:17) ≥ 0 .
Revised: The step-size η ti for each stage is monotonically decreasing, i [Equation] e. η ti ≤ η ti − 1 for i ≥ 2 , so (cid:16) 1 −

Revised: [Equation]

Original:  By the L -smoothness of f , the inner product of (31) can be estimated as
Revised: ti [Equation] ≥ 0 . By the L -smoothness of f , the inner product of (31) can be estimated as

Original: [Equation]
Revised: [Equation]

Original: Applying (32) into (31), we ﬁnd
Revised: Applying (32) into (31), we ﬁnd

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: we re-write the above inequality as
Revised: we re-write the above inequality as

Revised: E [ f ( z ti +1 ) | F ti ]

Revised: [Equation]

Revised: f

Original: [Equation]
Revised: [Equation]

Revised: z ti )

Revised: +

Revised: [Equation]

Original: The proof is complete.
Revised: The proof is complete.

Original: Lemma C.2.
Revised: Lemma C.2.

Original: Suppose that the objective function satisﬁes Assumption 1 and the step-size η ti is monotonically decreasing with η ti ≤ 1 L at each stage, then
Revised: Suppose that the objective function satisﬁes Assumption 1 and the step-size η ti is monotonically decreasing with η ti ≤ 1 L at each stage, then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First, due to the L -smoothness of the objective function f, we have
Revised: First, due to the L -smoothness of the objective function f

Original: [Equation]
Revised: [Equation]

Original: +
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where inequality (a) follows the Cauchy-Schwarz inequality that (cid:107) (1 − β)
Revised: where inequality (a) follows the Cauchy-Schwarz inequality that (cid:107) ( 1 − β

Revised: [Equation]

Revised: g ti + βv ti

Original: g ti + βv ti (cid:107) 2 ≤ (1 − β ) (cid:107) g ti (cid:107) 2 + β (cid:107) v ti (cid:107) 2 .
Revised: [Equation] 2 ≤ (1 − β ) (cid:107) g t i (cid:107) 2 + β (cid:107) v t i (cid:107) 2 .

Original: Then taking conditional expectation on both sides and due to that g ti is an unbiased estimator of ∇ f ( x ti ) , i.e., E [ g ti | F ti ] = ∇ f ( x ti ) , we have
Revised: Then taking conditional expectation on both sides and due to that g t i is an unbiased estimator of ∇ f ( x ti ) , i.e., E [ g ti | F ti ] = ∇ f ( x ti ) , we have

Original: [Equation]
Revised: [Equation]

Original: We recall the deﬁnition of v ti +1 and incorporate (26) into (27), then
Revised: We recall the deﬁnition of v ti +1 and incorporate (26) into (27), then

Original: [Equation]
Revised: [Equation]

Original: where ( a ) uses the fact that E [ g ti | F ti ] =
Revised: where ( a ) uses the fact that E [ g ti | F ti ] =

Original: ∇ f ( x ti ) ; ( b ) follows the procedure that x ti = x
Revised: ∇ f ( x ti ) ; ( b ) follows the procedure that x ti = x

Original: ti − 1 − η ti − 1 v ti ; (c) applies the fact that the step-size per stage is monotonically decreasing, i.e., η ti ≤ η ti − 1 for i ≥ 2 .
Revised: ti − 1 − η ti − 1 v ti ; (c) applies the fact that the step-size per stage is monotonically decreasing, i.e., η ti ≤ η ti − 1 for i ≥ 2 .

Original: Then multiplying 2 η ti (1 − β ) into (34) and combining (36), we get that
Revised: Then multiplying 2 η ti (1 − β ) into (34) and combining (36), we get that

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from the fact that η ti ≤ 1 /L.
Revised: where the last inequality follows from the fact that η ti ≤ 1 /L

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: ( of Lemma 4.1 )
Revised: ( of Lemma 4.1 )

Original: First we apply the result of Lemma C.1 and divided by η ti to the both side, we have
Revised: First we apply the result of Lemma C.1 and divided by η ti to the both side, we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then we recall the result of Lemma C
Revised: Then we recall the result of Lemma C.

Original: [Equation]
Revised: [Equation]

Revised: multiplying a constant r = βL 2(1 − β 2 )(1 − β ) 2 > 0 and dividing η ti to the both side, and then incorporating it into (37)

Original: [Equation]
Revised: [Equation]

Original: ∇ f ( x ti ) (cid:13)(cid:13)+ 2( η ti ) 2 (1 − β ) 2 E (cid:104)(cid:13)(cid:13) g ti (cid:13)(cid:13) 2 | F ti (cid:105) + ( η ti ) 3 β (1 − β ) L (cid:13)(cid:13) v ti (cid:13)(cid:13) 2 , multiplying a constant r = βL 2(1 − β 2 )(1 − β ) 2 > 0 and dividing η ti to the both side, and then incorporating it into (37), we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: We deﬁne a function W ti +1 as follows:
Revised: We deﬁne a function W ti +1 as follows:

Original: [Equation]
Revised: [Equation]

Original: Because of η ti ≤ η ti − 1 at each stage, we have − 1 /η ti ≤ − 1 /η
Revised: Because of η ti ≤ η ti − 1 at each stage, we have − 1 /η ti ≤ − 1 /η

Original: ti − 1 ( i ≥ 2 ), then
Revised: ti − 1 ( i ≥ 2 ), then

Original: [Equation]
Revised: [Equation]

Original: Taking conditional expectation on W ti +1 and applying (38) to the above inequality
Revised: Taking conditional expectation on W ti +1 and applying (38) to the above inequality

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: We recall that v 11 = 0 , due to the assumption that E [ (cid:107) g ti (cid:107) 2 ] ≤ G 2 , and v ti +1 is a convex combination of g ti and v ti , then by induction if E [ (cid:107) v ti (cid:107) 2 ] ≤ G 2 , then 
Revised: We recall that v 11 = 0 , due to the assumption that E [ (cid:107) g ti (cid:107) 2 ] ≤ G 2 , and v ti +1 is a convex combination of g t i and v t i , then by induction if E [ (cid:107) v t i (cid:107) 2 ] ≤ G 2 , then [Equation]

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have E [ (cid:107) v ti (cid:107) 2 ] is bounded by G 2 .
Revised: Therefore, we have E [ (cid:107) v ti (cid:107) 2 ] is bounded by G 2 .

Original: Then we apply E [ (cid:107) g ti (cid:107) 2 ] ≤ G 2 , E [ (cid:107) v ti (cid:107) 2 ] ≤ G 2 and η ti ≤ 1 /L into (40)
Revised: Then we apply E [ (cid:107) g ti (cid:107) 2 ] ≤ G 2 , E [ (cid:107) v ti (cid:107) 2 ] ≤ G 2 and η ti ≤ 1 /L into (40)

Original: [Equation]
Revised: [Equation]

Original: The step-size is decreasing at each stage, then η ti ≤ η ti − 1 ( i ≥ 2 ), thus 1 η ti − 1 η ti − 1 ≥ 0 .
Revised: The step-size is decreasing at each stage, then η ti ≤ η ti − 1 ( i ≥ 2 ), thus 1 η ti − 1 η ti − 1 ≥ 0 .

Original: Due to the fact that v ti is bounded (see (41)), i.e., E [ (cid:107) v ti (cid:107) 2 ] ≤ G 2 , we have
Revised: Due to the fact that v ti is bounded (see (41)), i.e., E [ (cid:107) v ti (cid:107) 2 ] ≤ G 2 , we have

Original: [Equation]
Revised: [Equation]

Original: Recalling the deﬁnition of z ti , and applying the assumption that E [ f ( x ti ) − f ∗ ] ≤ ∆ 0 for each t, i ≥and f is L -smooth on its domain, and η ti ≤ 1 /L gives
Revised: Recalling the deﬁnition of z ti , and applying the assumption that E [ f ( x ti ) − f ∗ ] ≤ ∆ 0 for each t, i ≥and f is L -smooth on its domain, and η ti ≤ 1 /L gives

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the inequality (a) dues to the fact that f ( x ti − 1 ) ≤ f ( x
Revised: where the inequality (a) dues to the fact that f ( x ti − 1 ) ≤ f ( x

Original: [Equation]
Revised: [Equation]

Original: + (cid:10) ∇
Revised: + (cid:10) ∇

Original: [Equation]
Revised: [Equation]

Original: ( x ti ) , x ti − 1 − x ti (cid:11) +
Revised: ( x ti ) , x ti − 1 − x ti (cid:11) +

Original: L 2 (cid:13) (cid:13) x
Revised: L 2 (cid:13) (cid:13) x

Original: t i − 1 − x t i (cid:13) (cid:13) 2 .
Revised: t i − 1 − x t i (cid:13) (cid:13) 2 .

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: Let ∆ z = ∆ 0 1 − β + βG 2 2(1 − β ) 2 L .
Revised: Let ∆ z = ∆ 0 1 − β + βG 2 2(1 − β ) 2 L .

Original: Finally, applying (43) and (45), the bounded assumption on E [ f ( x ti ) − f ∗ ] , and η ti ≤ 1 /L , we have
Revised: Finally, applying (43) and (45), the bounded assumption on E [ f ( x ti ) − f ∗ ] , and η ti ≤ 1 /L , we have

Original: [Equation]
Revised: [Equation]

Original: 46) where A 1 = 
Revised: 46) where A 1 = β

Original: [Equation]

Original: ∆

Original: [Equation]
Revised: [Equation]

Original: The bandwidth step-size highly rises the difﬁculty of the analysis for momentum, especially when the step-size has an increase between the stages, i.e. η t − 1 S t − 1 :
Revised: The bandwidth step-size highly rises the difﬁculty of the analysis for momentum, especially when the step-size has an increase between the stages, i.e. η t − 1 S t − 1 :

Original: [Equation]
Revised: [Equation]

Original: η t 0 < η t 1 .
Revised: η t 0 < η t 1 .

Original: Before given the results, we consider two situations:
Revised: Before giving the results, we consider two situations:

Original: • η t 0 > η t 1 .
Revised: • η t 0 > η t 1 .

Original: We can apply Lemma 4.1 from i = 1 to S t .
Revised: We can apply Lemma 4.1 from i = 1 to S t .

Original: Recalling the deﬁnition of W t i +1 , we have W t + = W tS t +1 , then S t
Revised: Recalling the deﬁnition of W t i +1 , we have W t + = W tS t +1 , then S t

Original: [Equation]
Revised: [Equation]

Original: • Otherwise if η t 0 ≤ η t 1 , the results of Lemma 4.1 only hold from i = 2 to S t .
Revised: • Otherwise if η t 0 ≤ η t 1 , the results of Lemma 4.1 only hold from i = 2 to S t .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: For the bandwidth step-size, the initial step-size of stage t , η t 1 , is possibly larger than the ending step-size of the previous stage, η t 0 .
Revised: For the bandwidth step-size, the initial step-size of stage t , η t 1 , is possibly larger than the ending step-size of the previous stage, η t 0 .

Original: Thus, we can not use the simpler condition (47), but have to rely on (48) in our derivations below.
Revised: Thus, we can not use the simpler condition (47), but have to rely on (48) in our derivations below.

Original: Lemma C.3.
Revised: Lemma C.3.

Original: Suppose the same setting as Lemma 4.1, we have
Revised: Suppose the same setting as Lemma 4.1, we have

Original: [Equation]
Revised: [Equation]

Original: where (cid:80)
Revised: where (cid:80)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , C
Revised: , C

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: 0 1 −
Revised: 0 1 −

Original: [Equation]
Revised: [Equation]

Original: , C 2 = C 

Revised: , C 2 = C

Original: + A 2 G 2 , and
Revised: + A 2 G 2 , and

Original: A 1 , B 1 , r , and ∆ z are deﬁned in Lemma 4.1.
Revised: A 1 , B 1 , r , and ∆ z are deﬁned in Lemma 4.1.

Original: Proof.
Revised: Proof.

Original: Applying the result of Lemma 4.1 from i = 2 to S t , the step-size η ti ∈ [ mδ ( t ) , Mδ ( t )] , and
Revised: Applying the result of Lemma 4.1 from i = 2 to S t , the step-size η ti ∈ [ mδ ( t ) , Mδ ( t )] , and

Original: W t + =
Revised: W t + =

Original: [Equation]
Revised: [Equation]

Original: t +1 , we have
Revised: t +1 , we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Recalling the output of Algorithm 2, we have
Revised: Recalling the output of Algorithm 2, we have

Original: [Equation]
Revised: [Equation]

Original: Then we divide δ ( t ) into the both side of (49), apply (49) from t = 1 to N and let (cid:80) δ
Revised: Then we divide δ ( t ) into the both side of (49), apply (49) from t = 1 to N and let (cid:80) δ

Original: = (cid:80)
Revised: = (cid:80)

Original: Nt =1 S t δ ( t ) −
Revised: Nt =1 S t δ ( t ) −

Original: E [
Revised: E [

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: First, we estimate (cid:80)
Revised: First, we estimate (cid:80)

Original: Nt =1 E [ (cid:107) ∇ f ( x t 1 ) (cid:107) 2 ] δ ( t ) .
Revised: Nt =1 E [ (cid:107) ∇ f ( x t 1 ) (cid:107) 2 ] δ ( t ) .

Original: From Lemma C.2, let i = 1 , then incorporating the inequalities (34) and (35), we have
Revised: From Lemma C.2, let i = 1 , then incorporating the inequalities (34) and (35), we have

Original: [Equation]
Revised: [Equation]

Original: Then dividing 2( η t 1 ) 2 (1 − β ) 2 to the both side and applying the fact that (cid:107) x t 1 − x t 0 (cid:107) =
Revised: Then dividing 2( η t 1 ) 2 (1 − β ) 2 to the both side and applying the fact that (cid:107) x t 1 − x t 0 (cid:107) =

Original: ( η t 0 ) 2 (cid:107) v t 1 (cid:107) 2 ,
Revised: ( η t 0 ) 2 (cid:107) v t 1 (cid:107) 2 ,

Original: E [ (cid:107) g ti (cid:107) 2 ] ≤ G 2 and E [ (cid:107) v ti (cid:107) 2 ] ≤ G 2 for any i, t , f ( x t 2 ) − f ∗ ≤ ∆ 0 and η t 1 ≥ mδ ( t ) , we have
Revised: E [ (cid:107) g ti (cid:107) 2 ] ≤ G 2 and E [ (cid:107) v ti (cid:107) 2 ] ≤ G 2 for any i, t , f ( x t 2 ) − f ∗ ≤ ∆ 0 and η t 1 ≥ mδ ( t ) , we have

Original: E [ (cid:13)(cid:13) ∇ f ( x t 1 ) (cid:13)(cid:13)
Revised: E [ (cid:13)(cid:13) ∇ f ( x t 1 ) (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (52) where A 2 = (cid:16) 1 + β 2(1 − β )(cid:17) .
Revised: (52) where A 2 = (cid:16) 1 + β 2(1 − β )(cid:17) .

Original: Then
Revised: Then

Original: N (cid:88) t
Revised: N (cid:88) t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Next we turn to estimate (cid:80)
Revised: Next we turn to estimate (cid:80)

Original: Nt =1 E [ W t
Revised: Nt =1 E [ W t

Original: [Equation]
Revised: [Equation]

Original: Recalling the deﬁnition of W ti +1 , we have W ti ≥ 0 .
Revised: Recalling the deﬁnition of W ti +1 , we have W ti ≥ 0 .

Original: Applying the inequalities (43), (45) and the assumption that f ( x ti ) − f ∗ ≤ ∆ 0 for any i, t , we have
Revised: Applying the inequalities (43), (45) and the assumption that f ( x ti ) − f ∗ ≤ ∆ 0 for any i, t , we have

Original: [Equation]
Revised: [Equation]

Original: dividing δ ( t ) and applying (54) from t = 1 to N , we have
Revised: dividing δ ( t ) and applying (54) from t = 1 to N , we have

Original: N (cid:88)
Revised: N (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Then we consider
Revised: Then we consider

Original: N (cid:88)
Revised: N (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Recalling the deﬁnition of E [ W t +
Revised: Recalling the deﬁnition of E [ W t +

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and applying the assumption that E [ f ( x ti ) − f ∗ ] ≤ ∆ 0 and E [ f ( z ti ) − f ∗ ] ≤ ∆ z , and η t + = η tS t ,
Revised: and applying the assumption that E [ f ( x ti ) − f ∗ ] ≤ ∆ 0 and E [ f ( z ti ) − f ∗ ] ≤ ∆ z , and η t + = η tS t ,

Original: E [ (cid:13)(cid:13) x
Revised: E [ (cid:13)(cid:13) x

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: − x t +10 (cid:13)(cid:13) 2 ] =
Revised: − x t +10 (cid:13)(cid:13) 2 ] =

Original: ( η tS t ) 2 E [ (cid:13)(cid:13) v t +11 (cid:13)(cid:13) 2 ] ≤ ( η tS t ) 2 G 2 , we have
Revised: ( η tS t ) 2 E [ (cid:13)(cid:13) v t +11 (cid:13)(cid:13) 2 ] ≤ ( η tS t ) 2 G 2 , we have

Original: [Equation]
Revised: [Equation]

Original: (57) where (a) follows from η tS t ≥ mδ ( t ) and η tS t ≤ 1 /L .
Revised: (57) where (a) follows from η tS t ≥ mδ ( t ) and η tS t ≤ 1 /L .

Original: Applying (57) into (56), we have
Revised: Applying (57) into (56), we have

Original: N (cid:88)
Revised: N (cid:88)

Original: [Equation]
Revised: [Equation]

Original: 58) where the second inequality follows that δ ( t ) is decreasing, so δ ( t ) ≤ δ ( t − 1) , then − 1 /δ ( t ) ≤ − 1 /δ
Revised: 58) where the second inequality follows that δ ( t ) is decreasing, so δ ( t ) ≤ δ ( t − 1) , then − 1 /δ ( t ) ≤ − 1 /δ

Original: ( t − 1) .
Revised: ( t − 1) .

Original: Finally, due to that η t S t ∈ [ mδ ( t ) , Mδ ( t )] , we have
Revised: Finally, due to that η t S t ∈ [ mδ ( t ) , Mδ ( t )] , we have

Original: N (cid:88)
Revised: N (cid:88)

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Incorporate the inequalities (53), (55), (58) and (59) into (51), we have
Revised: Incorporate the inequalities (53), (55), (58) and (59) into (51), we have

Original: E [ (cid:107)∇
Revised: E [ (cid:107)∇

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The above result can be re-written as (recall δ (1) = 1 )
Revised: The above result can be re-written as (recall δ (1) = 1 )

Original: [Equation]
Revised: E [

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where C 0 =
Revised: where C 0 =

Original: r
Revised: r

Original: ( G 2 L + 2∆ 0 ) , C 

Revised: ( G 2 L + 2∆ 0 ) , C

Original: =
Revised: =

Original: A 1 + ∆ z + ∆ 0 1 − β [Equation] and C 2 = C 0 + A 2 G [Equation]
Revised: A 1 + ∆ z + ∆ 0 1 − β , and C 2 = C 0 + A 2 G 2 .

Original: Proof.
Revised: Proof.

Original: ( of Theorem 4.2 )
Revised: ( of Theorem 4.2 )

Original: In this case, given the total number iteration T ≥ 1 , the number of stages
Revised: In this case, given the total number iteration T ≥ 1 , the number of stages

Original: N ≥ 1 , S t = S =  T/N  , δ ( t ) = 1 /α t − 1 for each 1 ≤ t ≤ N , then the boundary function at the ﬁnal stage δ ( N )
Revised: N ≥ 1 , S t = S = (cid:100) T/N (cid:101) , δ ( t ) = 1 /α t − 1 for each 1 ≤ t ≤ N , then the boundary function at the ﬁnal stage δ ( N )

Original: = 1 /α
Revised: = 1 /α

Original: N − 1 and δ ( N + 1) = 1 /α N .
Revised: N − 1 and δ ( N + 1) = 1 /α N .

Original: Applying the speciﬁc value of δ ( t ) and N gives
Revised: Applying the speciﬁc value of δ ( t ) and N gives

Original: N (cid:88)
Revised: N [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then by S t =  T/N  and (61), we easily get
Revised: Then by S t = (cid:100) T/N (cid:101) and (61), we easily get

Original: [Equation]
Revised: [Equation]

Original: We then plug the above results into Lemma C.3,
Revised: We then plug the above results into Lemma C.3,

Original: [Equation]

Original:  [ (cid:107)∇
Revised: E [ (cid:107)∇

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  where C 0 ={+
+}
Revised: 64) where C {+
+}

Original:  r
Revised: = r

Original: ( G 2 L + 2∆ 0 ) , C 

Revised: ( G 2 L + 2∆ 0 ) , C

Original: =
Revised: =

Original: A 1 + ∆ z + ∆ 0 1 − β , and C 2 = C 0 + A 2 G 2 , A 2 = 1 + β 2(1 − β ) 2 , and
Revised: A 1 + ∆ z + ∆ 0 1 − β , and C 2 = C 0 + A 2 G 2 , A 2 = 1 + β 2(1 − β ) 2 , and

Original: W 1 1 , A 1 , B 1 , ∆ z , and r are deﬁned in Lemma 4.1.
Revised: W 11 , A 1 , B 1 , ∆ z , and r are deﬁned in Lemma 4.1.

Original: Especially, we consider the number of outer-stage N = {+(cid:98) (log α T ) / 2 (cid:99) , the stage length S t = (cid:100) 
+}
Revised: Especially, we consider the number of outer-stage N = {+(cid:98) (log α T ) / 2 (cid:99) , the stage length S t = (cid:100) 
+}

Original: (log α T ) / 2 , the stage length S t = 2 T/ log α T  , and the boundary functions δ ( t )
Revised: T/ log α T (cid:101) , and the boundary functions δ ( t )

Original: = 1 /α t − 1 for all t ∈ { 1 , 2 , · · · , N } .
Revised: = 1 /α t − 1 for all t ∈ { 1 , 2 , · · · , N } .

Original: Let N =  (log α T ) / 2  , we have 
Revised: Let N = (cid:98) (log α T ) / 2 (cid:99) , we have E

Original: [Equation]
Revised: [Equation]

Original: Therefore, we complete the proof.
Revised: Therefore, we complete the proof.

Original: Proof.
Revised: Proof.

Original: ( of Theorem 4.3 )
Revised: ( of Theorem 4.3 )

Original: In this theorem, we consider the boundary functions δ ( t ) = 1 /α t − 1 , and the stage length S t =[-S 
-]
Revised: In this theorem, we consider the boundary functions δ ( t ) = 1 /α t − 1 , and the stage length S t =[-S 
-]

Original: {+(cid:100) S 
+}
Revised: {+(cid:100) S 
+}

Original: α t − 1  with S 0 = √ T , then we have the number of stages N =  log α (( α − 1) √ T +1) and δ1 ( N ) =α/ (( α  − 1) √ T +1) .
Revised: α t − 1 (cid:101) with S 0 = √ T , then we have the number of stages N = (cid:98) log α (( α − 1) √ T + 1) (cid:99) and δ ( N ) = α N − 1 .

Original: Next we estimate (cid:80) δ
Revised: Next we estimate (cid:80) δ

Original: = (cid:80) N t =1 S t δ − 1 ( t ) =
Revised: = (cid:80) Nt =1 S t δ − 1 ( t ) ≥ (

Original: ( α − 1) 2 T 3 / 2{+
+} + 2( α − 1) T .
Revised: α − 1) 2 T 3 / {+
+} + 2( α − 1) T .

Original: Then applying these results into Lemma C.3, we have
Revised: Then applying these results into Lemma C.3, we have

Original: [Equation]
Revised: [Equation]

Original: D S UPPLEMENTARY C ONVERGENCE R ESULTS D. C ONVERGENCE OF SGDM WITH C ONSTANT AND 1 / √ t B ANDWIDTH D ECAYING STEP - SIZE
Revised: D S UPPLEMENTARY C ONVERGENCE R ESULTS D. C ONVERGENCE OF SGDM WITH C ONSTANT AND 1 / √ t B ANDWIDTH D ECAYING STEP - SIZE

Original: We focus on a single stage that N = 1 .
Revised: We focus on a single stage that N = 1 .

Original: In this case, we ﬁrst consider the constant step-size η ti = η 0 / √ T .
Revised: In this case, we ﬁrst consider the constant step-size η ti = η 0 / √ T .

Original: Recalling the result of Lemma 4.1 and letting η ti = η 0 / √ T for each i ≥ 1 , we have
Revised: Recalling the result of Lemma 4.1 and letting η ti = η 0 / √ T for each i ≥ 1 , we have

Original: [Equation]
Revised: [Equation]

Original: where B 1 = r (1 − β )(2 − β )
Revised: where B 1 = r (1 − β )(2 − β )

Original: +
Revised: +

Original: L 2(1 − β ) 2 and r = βL 2(1 −
Revised: L 2(1 − β ) 2 and r = βL 2(1 −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (1 −
Revised: (1 −

Original: [Equation]
Revised: [Equation]

Original: Next we turn to analyze the 1 / √ t bandwidth step-size η 1 i ∈ [ m/ √ i, M/ √ i ] (which is also monotonic decreasing).
Revised: Next we turn to analyze the 1 / √ t bandwidth step-size η 1 i ∈ [ m/ √ i, M/ √ i ] (which is also monotonic decreasing).

Original: Recalling the result of Lemma 4.
Revised: Recalling the result of Lemma 4.

Original: [Equation]
Revised: [Equation]

Original: and applying the result from i = 1 to T , we have
Revised: and applying the result from i = 1 to T , we have

Original: [Equation]
Revised: [Equation]

Original: Then applying the step-size η 1 i ∈ [ m/ √ i, M/ √ i ] gives
Revised: Then applying the step-size η 1 i ∈ [ m/ √ i, M/ √ i ] gives

Original: [Equation]
Revised: [Equation]

Original: Thus, we can achieve an O (1 / √ T ) optimal rate for SGDM with 1 / √ t bandwidth step-size on nonconvex problems.
Revised: Thus, we can achieve an O (1 / √ T ) optimal rate for SGDM with 1 / √ t bandwidth step-size on nonconvex problems.

Original: When M = m , then the step-size reduces to η 1 i = m/ √ i , we also provide the convergence guarantee for the commonly used 1 / √ i decaying step-size.
Revised: When M = m , then the step-size reduces to η 1 i = m/ √ i , we also provide the convergence guarantee for the commonly used 1 / √ i decaying step-size.

Original: D. C ONVERGENCE G UARANTEES FOR C YCLICAL S TEP -S IZES
Revised: D. C ONVERGENCE G UARANTEES FOR C YCLICAL S TEP -S IZES

Original: In Loshchilov and Hutter (2017), the authors proposed a cosine annealing step-size
Revised: In Loshchilov and Hutter (2017), the authors proposed a cosine annealing step-size

Original: [Equation]
Revised: [Equation]

Original: (69) where η t min and η t max are ranges of the step-size, and T cur accounts for how many epochs since the beginning of the current stage and T t accounts for the current stage length (epoch).
Revised: (69) where η t min and η t max are ranges of the step-size, and T cur accounts for how many epochs since the beginning of the current stage and T t accounts for the current stage length (epoch).

Original: At each stage t , the step-size is monotonically decaying within the range η t min and η t max .
Revised: At each stage t , the step-size is monotonically decaying within the range η t min and η t max .

Original: In this paper, we propose a general bandwidth framework for step-size which can cover this situation as long as mδ ( t ) ≤ η t min and η t max ≤ Mδ ( t ) .
Revised: In this paper, we propose a general bandwidth framework for step-size which can cover this situation as long as mδ ( t ) ≤ η t min and η t max ≤ Mδ ( t ) .

Original: If the ranges { η t min , η t max } and the stage length are chosen as for example 1 / √ t in Theorems A.1 and A.2 or step-decay in Theorems 3.1 and 3.3, the theoretical convergence of
Revised: If the ranges { η t min , η t max } and the stage length are chosen as for example 1 / √ t in Theorems A.1 and A.2 or step-decay in Theorems 3.1 and 3.3, the theoretical convergence of

Original: SGD under the cosine annealing step-size is guaranteed by our analysis in Section 3.
Revised: SGD under the cosine annealing step-size is guaranteed by our analysis in Section 3.

Original: Moreover, because the cosine annealing is monotonic at each stage, so the convergence of SGD with momentum under the cosine annealing policy is also guaranteed by the analysis of Section 4 as long as the ranges η t min , η t max are within our bands.
Revised: Moreover, because the cosine annealing is monotonic at each stage, so the convergence of SGD with momentum under the cosine annealing policy is also guaranteed by the analysis of Section 4 as long as the ranges η t min , η t max are within our bands.

Original: To the best of our knowledge, Li et al.
Revised: To the best of our knowledge, Li et al.

Original: (2021) provides a convergence guarantee for cosine step-size.
Revised: (2021) provides a convergence guarantee for cosine step-size.

Original: However, to achieve a near-optimal rate for the general smooth (non-convex) problems, the initial step-size is required to be bounded by O (1 / √ T ) which is obviously impractical when the total number of iteration T is large (also discussed in related work).
Revised: However, to achieve a near-optimal rate for the general smooth (non-convex) problems, the initial step-size is required to be bounded by O (1 / √ T ) which is obviously impractical when the total number of iteration T is large (also discussed in related work).

Original: In our framework, the cosine step-size is allowed to start from a larger step-size and gradually decay.
Revised: In our framework, the cosine step-size is allowed to start from a larger step-size and gradually decay.

Original: Besides, our results (e.g., Theorems 3.3 and 4.3) provide state-of-art convergence guarantees for cosine step-size which remove the log T term of Li et al.
Revised: Besides, our results (e.g., Theorems 3.3 and 4.3) provide state-of-the-art convergence guarantees for cosine step-size which remove the log T term of Li et al.

Original: Another interesting example is triangular cyclical step-size proposed by Smith (2017), which sets minimum η t min and maximum η t max boundaries and the learning rate cyclically varies (linearly increasing then linearly decreasing) in these bounds.
Revised: Another interesting example is triangular cyclical step-size proposed by Smith (2017), which sets minimum η t min and maximum η t max boundaries and the learning rate cyclically varies (linearly increasing then linearly decreasing) in these bounds.

Original: In each stage, the step-size is non-monotonic.
Revised: In each stage, the step-size is non-monotonic.

Original: In their paper, the author also consider a variant which cuts η t min and η t max in half after each stage.
Revised: In their paper, the author also consider a variant which cuts η t min and η t max in half after each stage.

Original: This is exactly the step-decay boundary we discussed.
Revised: This is exactly the step-decay boundary we discussed.

Original: Our analysis in Section 3 can provide the convergence guarantees for such kinds of step-sizes.
Revised: Our analysis in Section 3 can provide the convergence guarantees for such kinds of step-sizes.

Original: However, such cyclical step-size is not monotonic in each stage, so our analysis for SGDM in Section 4 is not suitable for this situation.
Revised: However, such cyclical step-size is not monotonic in each stage, so our analysis for SGDM in Section 4 is not suitable for this situation.

Original: D. D ISCUSSION ABOUT THE T RUST - REGION - ISH ALGORITHM BY C URTIS ET AL . (2019)

Original: Curtis et al.

Original: (2019) proposed a trust-region-ish algorithm which uses a careful normalization to adapt the learning rate based on the norm of stochastic gradient: given sequences γ t 1 ≥ γ t 2 > 0 , { α t } >

Original: [Equation]

Original: We observe that the step-size satisﬁes γ t 2 α t ≤ η t ≤ γ t 1 α t , i.e. is a bandwidth step-size with boundary functions δ ( t ) =

Original: α t and γ t 2 , γ t 1 ∈ [ m, M ] .

Original: Therefore, our analysis can easily provide convergence guarantees for such trust-region-ish algorithm if the step-size is selected as we do.

Original: Curtis et al.

Original: Theorem 5) provided the convergence guarantee E

Original: [-[ (cid:107)∇ f ( x ) (cid:107) 2 ] ≤ θ 2 θ 1 α 
-]
Revised: [-[ (cid:107)∇ f ( x ) (cid:107) 2 ] ≤ θ 2 θ 1 α 
-]

Original: ≈ O (1)

Original: + O ( α 0 ) under

Original: Non-PL condition with constant α t = α 0 .

Original: However, in Lemma B.1 if δ ( t ) = δ is a constant, then we can get E [ (cid:107)∇ f ( x ) (cid:107) 2 ] ≤ O ( 1 Tmδ + M 2 δm ) .

Original: Letting δ = 1 / √ T , we can achieve a convergence guarantee of O (1 / √ T ) , which is clearly stronger than that of Curtis et al.

Original: E A DDITIONAL D ETAILS OF THE E XPERIMENTS ON B ANDWIDTH S TEP -S IZES
Revised: E A DDITIONAL D ETAILS OF THE E XPERIMENTS ON B ANDWIDTH S TEP -S IZES

Original: In this section, we provide additional details about the numerical experiments in Section 5.
Revised: In this section, we provide additional details about the numerical experiments in Section 5.

Original: E. H OW TO D ESIGN THE B ANDWIDTH S TEP -S IZES AND S ELECT P ARAMETERS
Revised: E. H OW TO D ESIGN THE B ANDWIDTH S TEP -S IZES AND S ELECT P ARAMETERS

Original: To better understand the bandwidth step-sizes tested in the numerical experiments, we visualize the step-size η ti (y-axis is log( η ti ) ) vs the number of epochs in Figure 5.
Revised: To better understand the bandwidth step-sizes tested in the numerical experiments, we visualize the step-size η ti (y-axis is log( η ti ) ) vs the number of epochs in Figure 5.

Original: We ﬁrst consider the popular “step-decay” policy as the baseline.
Revised: We ﬁrst consider the popular “step-decay” policy as the baseline.

Original: During the ﬁrst stage, the bandwidth step-size follows the lower bound.
Revised: During the ﬁrst stage, the bandwidth step-size follows the lower bound.

Original: From the second stage and on, we let the initial step-size in each stage to be equal to the upper bound and the last step-size in the stage to reach the lower bound.
Revised: From the second stage and on, we let the initial step-size in each stage to be equal to the upper bound and the last step-size in the stage to reach the lower bound.

Original: Our numerical experience has shown that the best performance is obtained when the initial step-size of each stage is larger than the ﬁnal step-size of the previous stage, which means that the step-size experiences a sudden increase before it decreases again.
Revised: Our numerical experience has shown that the best performance is obtained when the initial step-size of each stage is larger than the ﬁnal step-size of the previous stage, which means that the step-size experiences a sudden increase before it decreases again.

Original: For the step-decay band, we consider four decay modes: 1 /i , 1 / √ i , linearly, and according to a cosine function (Loshchilov and Hutter, 2017) and update the step size each epoch.
Revised: For the step-decay band, we consider four decay modes: 1 /i , 1 / √ i , linearly, and according to a cosine function (Loshchilov and Hutter, 2017) and update the step size each epoch.

Original: If the training size is n and sample size per iteration is b , then one epoch is n/b iterations.
Revised: If the training size is n and sample size per iteration is b , then one epoch is n/b iterations.

Original: We also adopt the polynomial 1 / √ t step-size as the boundary function, named 1 / √ t -band, and update the step-size every epoch.
Revised: We also adopt the polynomial 1 / √ t step-size as the boundary function, named 1 / √ t -band, and update the step-size every epoch.

Original: We add similar perturbation as for the step-decay band, but we do not apply the perturbation per stage.
Revised: We add similar perturbation as for the step-decay band, but we do not apply the perturbation per stage.

Original: Otherwise, the perturbation is too frequent and just increases the variance of the iterates.
Revised: Otherwise, the perturbation is too frequent and just increases the variance of the iterates.

Original: We tune the frequency of the perturbations (denoted as N 0 ) to undergo a similar number of cycles as the step-decay perturbations.
Revised: We tune the frequency of the perturbations (denoted as N 0 ) to undergo a similar number of cycles as the step-decay perturbations.

Original: In the ﬁrst cycle, the bandwidth step-sizes agree with their lower bound, just as for step-decay.
Revised: In the ﬁrst cycle, the bandwidth step-sizes agree with their lower bound, just as for step-decay.

Original: From the second cycle, we begin to add the decreasing perturbations, e.g., 1 / √ i , 1 /i and linearly.
Revised: From the second cycle, we begin to add the decreasing perturbations, e.g., 1 / √ i , 1 /i and linearly.

Original: As discussed above, these perturbations are only adjusted between stages.
Revised: As discussed above, these perturbations are only adjusted between stages.

Original: Several different 1 / √ t bandwidth step-sizes are shown in Figure 5. 
Revised: Several different 1 / √ t bandwidth step-sizes are shown in Figure 5. [Figure]

Original: We perform a grid search for the initial step-size η 0 ∈ { 0 . 01 , 0 .
Revised: We perform a grid search for the initial step-size η 0 ∈ { 0 . 01 , 0 .

Original: 5 , 1 , 5 } of the baseline stepsizes.
Revised: 5 , 1 , 5 } of the baseline stepsizes.

Original: For step-decay step-size (baseline), we select the decay factor α from { 1 . 5 , 2 , 3 , 4 , · · · , 12 } and set the number of stages N = (cid:98) log α T/ 2 (cid:99) according to Theorems 3.1 and 4.2.
Revised: For step-decay step-size (baseline), we select the decay factor α from { 1 . 5 , 2 , 3 , 4 , · · · , 12 } and set the number of stages N = (cid:98) log α T/ 2 (cid:99) according to Theorems 3.1 and 4.2.

Original: We choose the lower bound parameter m =
Revised: We choose the lower bound parameter m =

Original: η 0 to agree with the baseline.
Revised: η 0 to agree with the baseline.

Original: The bandwidth s = M/m = αθ where θ ∈ { 0 . 5 , 0 . 8 , 1 , 1 . 2 , 1 . 3 , 1 . 5 , 1 . 8 } .
Revised: The bandwidth s = M/m = αθ where θ ∈ { 0 . 5 , 0 . 8 , 1 , 1 . 2 , 1 . 3 , 1 . 5 , 1 . 8 } .

Original: If θ > 1 , it means that the starting step-size at the current stage is larger than the ending step-size from the previous stage.
Revised: If θ > 1 , it means that the starting step-size at the current stage is larger than the ending step-size from the previous stage.

Original: For the 1 / √ t -band step-sizes, we choose η t i =
Revised: For the 1 / √ t -band step-sizes, we choose η ti =

Original: η 0 / (1 + a √ t ) as the baseline , and select the best a > 0 to make the ﬁnal step-size reach[Figure] the interval { 0 . 001 , 0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 , 1 } .
Revised: η 0 / (1 + a √ t ) as the baseline , and select the best a > 0 to make the ﬁnal step-size reach the interval { 0 . 001 , 0 . 005 , 0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 , 1 } .

Original: Moreover, we select the lower bound parameter m =
Revised: Moreover, we select the lower bound parameter m =

Original: η 0 to make sure that the lower bound agrees with the baseline.
Revised: η 0 to make sure that the lower bound agrees with the baseline.

Original: For the upper bound parameter
Revised: For the upper bound parameter

Original: M , we do a grid search for s = M/m ∈ { 2 , 3 , 4 , 5 , 6 } .
Revised: M , we do a grid search for s = M/m ∈ { 2 , 3 , 4 , 5 , 6 } .

Original: The number of perturbation cycles N 0 for the 1 / √ t -band step-sizes is chosen from { 1 , 2 , 3 , 4 } .
Revised: The number of perturbation cycles N 0 for the 1 / √ t -band step-sizes is chosen from { 1 , 2 , 3 , 4 } .

Original: All the hyper-parameters are selected to work best according to their performance on the test dataset.
Revised: All the hyper-parameters are selected to work best according to their performance on the test dataset.

Revised: [Figure]

Original: E. E XPERIMENTS D ETAILS ON CIFAR10 AND CIFAR100 D ATASETS
Revised: E. E XPERIMENTS D ETAILS ON CIFAR10 AND CIFAR100 D ATASETS

Original: In this subsection, we will give the implementation details for the experiments on CIFAR10 and
Revised: In this subsection, we will give the implementation details for the experiments on CIFAR10 and

Original: CIFAR100.
Revised: CIFAR100.

Original: The benchmark datasets CIFAR10 and CIFAR100 (Kri) both consist of 60000 colour images (50000 training images and the rest 10000 images for testing).
Revised: The benchmark datasets CIFAR10 and CIFAR100 (Krizhevsky, 2009) both consist of 60000 colour images (50000 training images and the rest 10000 images for testing).

Original: The maximum epochs called for the two datasets is 180 and the batch size is 128 .
Revised: The maximum epochs called for the two datasets is 180 and the batch size is 128 .

Original: All the experiments on CIFAR datasets are implemented in Python 3.7.4 and run on 2 x Nvidia Tesla V100 SXM2 GPUs with 32GB RAM.
Revised: All the experiments on CIFAR datasets are implemented in Python 3.7.4 and run on 2 x Nvidia Tesla V100 SXM2 GPUs with 32GB RAM.

Original: All experiments are repeated 5 times to eliminate the effect of the randomness.
Revised: All experiments are repeated 5 times to eliminate the effect of the randomness.

Original: The performance of different algorithms is evaluated in terms of their loss function value and classiﬁcation accuracy on the  test dataset.
Revised: The performance of different algorithms is evaluated in terms of their loss function value and classiﬁcation accuracy on the training and test datasets.

Original: All the results for  test accuracy and test loss are reported in Figures 3 and 6,  respectively.
Revised: All the results for training loss, test loss, and test accuracy are reported in Figures 7, 6, and 3, respectively.

Original: For CIFAR 10, we train an 18-layer Resident Network model (He et al., 2016) called ResNet-18.
Revised: For CIFAR 10, we train an 18-layer Resident Network model (He et al., 2016) called ResNet-18.

Original: We ﬁrst test the vanlia SGD with a weight decay of 0.0005.
Revised: We ﬁrst test the vanlia SGD with a weight decay of 0.0005.

Original: The initial step-size η 0 = 1 and a = 1 .for 1 / √ t step-size (baseline).
Revised: The initial step-size η 0 = 1 and a = 1 .for 1 / √ t step-size (baseline).

Original: For 1 / √ t -band step-sizes, we set: s = 2 and N 0 = 3 for 1 / √ i mode; s = 3 and N 0 = 3 for 1 /i mode; and s = 4 and N 0 = 3 for linear mode.
Revised: For 1 / √ t -band step-sizes, we set: s = 2 and N 0 = 3 for 1 / √ i mode; s = 3 and N 0 = 3 for 1 /i mode; and s = 4 and N 0 = 3 for linear mode.

Original: For the step-decay step-size (baseline) and also the bandwidth step-sizes, the initial step-size η 0 = 0 .
Revised: For the step-decay step-size (baseline) and also the bandwidth step-sizes, the initial step-size η 0 = 0 .

Original: 5 and decay factor α = 6 .
Revised: 5 and decay factor α = 6 .

Original: For step-decay band step-sizes, the parameter θ is 1 .
Revised: For step-decay band step-sizes, the parameter θ is 1 .

Original: 3 for the 1 / √ i , 1 /i and linear modes, and is 1 . 2 for cosine mode.
Revised: 3 for the 1 / √ i , 1 /i and linear modes, and is 1 . 2 for cosine mode.

Original: We also implement the SGD with momentum (SGDM) algorithm, with the momentum parameter of 0 .
Revised: We also implement the SGD with momentum (SGDM) algorithm, with the momentum parameter of 0 .

Original: 9 and a weight decay of 0 . 0005 .
Revised: 9 and a weight decay of 0 . 0005 .

Original: For the step-decay step-size, the initial step-size η 0 is 0.05 and the decay factor α is 6 .
Revised: For the step-decay step-size, the initial step-size η 0 is 0.05 and the decay factor α is 6 .

Original: We choose the same initial step-size and decay factor for the step-decay bandwidth step-sizes.
Revised: We choose the same initial step-size and decay factor for the step-decay bandwidth step-sizes.

Original: The best θ is 1 .
Revised: The best θ is 1 .

Original: 3 for the four decay modes.
Revised: 3 for the four decay modes.

Original: In a similar way, we also detail our parameter selection for the experiments on CIFAR100.
Revised: In a similar way, we also detail our parameter selection for the experiments on CIFAR100.

Original: On this data set, we train a 28 × 10 wide residual network (WRN-28-10) (Zagoruyko and Komodakis, 2016).
Revised: On this data set, we train a 28 × 10 wide residual network (WRN-28-10) (Zagoruyko and Komodakis, 2016).

Original: We ﬁrst implement vanilla SGD with a weight decay of 0 . 0005 .
Revised: We ﬁrst implement vanilla SGD with a weight decay of 0 . 0005 .

Original: For the baseline of the 1 / √ t band, we set η 0 = 0 .
Revised: For the baseline of the 1 / √ t band, we set η 0 = 0 .

Original: 5 and a = 3 .
Revised: 5 and a = 3 .

Original: For the 1 / √ t -band step-sizes we use: s = 2 and N 0 = 3 for 1 / √ i -mode; s = 4 and N 0 = 3 for 1 /i
Revised: For the 1 / √ t -band step-sizes we use: s = 2 and N 0 = 3 for 1 / √ i -mode; s = 4 and N 0 = 3 for 1 /i

Original: -mode; s = 4 and N 0 = 2 for linear-mode.
Revised: -mode; s = 4 and N 0 = 2 for linear-mode.

Original: For the step-decay band, we choose η 0 = 0 .
Revised: For the step-decay band, we choose η 0 = 0 .

Original: 5 and α = 6 .
Revised: 5 and α = 6 .

Original: The parameter θ is set to 1 .
Revised: The parameter θ is set to 1 .

Original: 3 for 1 /i mode and 1 . 2 for the other modes.
Revised: 3 for 1 /i mode and 1 . 2 for the other modes.

Original: Then we also apply the step-decay band step-sizes on the SGD with momentum (SGDM) algorithm, where the momentum parameter is 0 .
Revised: Then we also apply the step-decay band step-sizes on the SGD with momentum (SGDM) algorithm, where the momentum parameter is 0 .

Original: 9 and the weight decay parameter is 0 . 0005 .
Revised: 9 and the weight decay parameter is 0 . 0005 .

Original: We set the initial step-size η 0 = 0 .
Revised: We set the initial step-size η 0 = 0 .

Original: 1 and α = 6 for the baseline and other step-decay band step-sizes; θ = 1 .
Revised: 1 and α = 6 for the baseline and other step-decay band step-sizes; θ = 1 .

Original: 2 for 1 / √ i and linear modes; and θ = 1 .
Revised: 2 for 1 / √ i and linear modes; and θ = 1 .

Original: 3 for 1 /i and cosine modes.
Revised: 3 for 1 /i and cosine modes.

Revised: [Figure]

Original: E. E XPERIMENTS R ESULTS OF THE T OY E XAMPLE ON B ANDWIDTH S TEP -S IZE
Revised: E. E XPERIMENTS R ESULTS OF THE T OY E XAMPLE ON B ANDWIDTH S TEP -S IZE

Original: In this subsection, we describe the toy examplefrom the introduction in detail (see Figure 2 andTable 1). We also report additional results using other bandwidth step-sizes.[Figure]
Revised: In this subsection, we describe the toy example and also report additional results using other bandwidth step-sizes.

Original: The loss-function is the two-dimensional ( x, y ∈ R are the variables) non-convex function:
Revised: The loss-function is the two-dimensional ( x, y ∈ R are the variables) non-convex function:

Original: [Equation]
Revised: [Equation]

Original: which has four local minima (denoted by 1 to 4 3 ), one of which is global ( 4 ).
Revised: which has four local minima (denoted by 1 to 4 3 ), one of which is global ( 4 ).

Original: We executealgorithm runs with an initial point ( − 0 .
Revised: We executealgorithm runs with an initial point ( − 0 .

Original: The total number of iterations is set to T = 3000 .
Revised: The total number of iterations is set to T = 3000 .

Original: The gradient noise is drawn from the standard normal distribution.
Revised: The gradient noise is drawn from the standard normal distribution.

Original: The setting of the experiments follows (Shi et al., 2020).
Revised: The setting of the experiments follows (Shi et al., 2020).

Original: The step-sizes we tested in this part are similar to Section E.1.
Revised: The step-sizes we tested in this part are similar to Section E.1.

Original: The difference is that here we update the step-size per iterate instead of per epoch, as we did on the CIFAR datasets.
Revised: The difference is that here we update the step-size per iterate instead of per epoch, as we did on the CIFAR datasets.

Original: We report the percentage (%) of the ﬁnal iterate close to each local minima in Table 2.
Revised: We report the percentage (%) of the ﬁnal iterate close to each local minima in Table 2.

Original: Note that the results for constant (large and small) step-size, step-decay (baseline), and step-decay with linear have already been presented in the introduction of the main document.
Revised: Note that the results for constant (large and small) step-size, step-decay (baseline), and step-decay with linear have already been presented in the introduction of the main document.

Original: The large constant step-size is 0 .
Revised: The large constant step-size is 0 .

Original: 1 and the small constant step-size is 0 . 05 .
Revised: 1 and the small constant step-size is 0 . 05 .

Original: As we can see, 1 / √ t -band with 1 / √ i , 1 /i and linear modes more likely to escape the bad local minima and ﬁnd the global solution than their baseline.
Revised: As we can see, 1 / √ t -band with 1 / √ i , 1 /i and linear modes more likely to escape the bad local minima and ﬁnd the global solution than their baseline.

Original: Except the result of step-decay with linear (shown in Figure 2 and Table 1), we also ﬁnd that other bandwidth step-sizes achieve good performance and work better than 
Revised: Except the result of step-decay with linear (shown in Figure 2 and Table 1), we also ﬁnd that other bandwidth step-sizes achieve good performance and work better than the baseline.

Revised: In the toy experiment, we also test the performance of step-decay (baseline) and step-decay with linear on different initial points.

Original: [Table]
Revised: We evenly select 100 initial points x 0 from the region [ − 1 , 1] × [ − 1 , 1] . At each initial point, we repeat the same process as previous initial point ( − 0 . 9 , 0 . 9) and do 10000 runs. We record the percentage of ﬁnal iterate close to global minima 4 at each initial point in Figure 8.

Original:  [Table] 
Revised: The x -axis denotes the distance of the initial point x 0 and the global minima x ∗ . We can see that compared to its baseline, step-decay with linear is less sensitive to the selection of the initial point and a relatively large θ > 1 works better if the initial point is far from the global minima. [Table] [Figure]