Original: E IGENCURVE : O PTIMAL L EARNING R ATE S CHEDULE FOR SGD ON Q UADRATIC O BJECTIVES WITH S KEWED H ESSIAN S PECTRUMS
Revised: E IGENCURVE : O PTIMAL L EARNING R ATE S CHEDULE FOR SGD ON Q UADRATIC O BJECTIVES WITH S KEWED H ESSIAN S PECTRUMS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Learning rate schedulers have been widely adopted in training deep neural networks.
Revised: Learning rate schedulers have been widely adopted in training deep neural networks.

Original: Despite their practical importance, there is a discrepancy between its practice and its theoretical analysis.
Revised: Despite their practical importance, there is a discrepancy between its practice and its theoretical analysis.

Original: For instance, it is not known what schedules of SGD achieve best convergence, even for simple problems such as optimizing quadratic objectives.
Revised: For instance, it is not known what schedules of SGD achieve best convergence, even for simple problems such as optimizing quadratic objectives.

Original: In this paper, we propose Eigencurve, the ﬁrst family of learning rate schedules that can achieve minimax optimal convergence rates (up to a constant) for SGD on quadratic objectives when the eigenvalue distribution of the underlying Hessian matrix is skewed.
Revised: In this paper, we propose Eigencurve, the ﬁrst family of learning rate schedules that can achieve minimax optimal convergence rates (up to a constant) for SGD on quadratic objectives when the eigenvalue distribution of the underlying Hessian matrix is skewed.

Original: The condition is quite common in practice.
Revised: The condition is quite common in practice.

Original: Experimental results show that Eigencurve can signiﬁcantly outperform step decay in image classiﬁcation tasks on CIFAR-10, especially when the number of epochs is small.
Revised: Experimental results show that Eigencurve can signiﬁcantly outperform step decay in image classiﬁcation tasks on CIFAR-10, especially when the number of epochs is small.

Original: Moreover, the theory inspires two simple learning rate schedulers for practical applications that can approximate eigencurve.
Revised: Moreover, the theory inspires two simple learning rate schedulers for practical applications that can approximate eigencurve.

Original: For some problems, the optimal shape of the proposed schedulers resembles that of cosine decay, which sheds light to the success of cosine decay for such situations.
Revised: For some problems, the optimal shape of the proposed schedulers resembles that of cosine decay, which sheds light to the success of cosine decay for such situations.

Original: For other situations, the proposed schedulers are superior to cosine decay.
Revised: For other situations, the proposed schedulers are superior to cosine decay.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many machine learning models can be represented as the following optimization problem:
Revised: Many machine learning models can be represented as the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: such as logistic regression, deep neural networks.
Revised: such as logistic regression, deep neural networks.

Original: To solve the above problem, stochastic gradient descent (SGD) (Robbins & Monro, 1951) has been widely adopted due to its computation efﬁciency in large-scale learning problems (Bottou & Bousquet, 2008), especially for training deep neural networks.
Revised: To solve the above problem, stochastic gradient descent (SGD) (Robbins & Monro, 1951) has been widely adopted due to its computation efﬁciency in large-scale learning problems (Bottou & Bousquet, 2008), especially for training deep neural networks.

Original: Given the popularity of SGD in this ﬁeld, different learning rate schedules have been proposed to further improve its convergence rates.
Revised: Given the popularity of SGD in this ﬁeld, different learning rate schedules have been proposed to further improve its convergence rates.

Original: Among them, the most famous and widely used ones are inverse time decay, step decay (Gofﬁn, 1977), and cosine scheduler (Loshchilov & Hutter, 2017).
Revised: Among them, the most famous and widely used ones are inverse time decay, step decay (Gofﬁn, 1977), and cosine scheduler (Loshchilov & Hutter, 2017).

Original: The learning rates generated by the inverse time decay scheduler depends on the current iteration number inversely.
Revised: The learning rates generated by the inverse time decay scheduler depends on the current iteration number inversely.

Original: Such a scheduling strategy comes from the theory of SGD on strongly convex functions, and is extended to non-convex objectives like neural networks while still achieving good performance.
Revised: Such a scheduling strategy comes from the theory of SGD on strongly convex functions, and is extended to non-convex objectives like neural networks while still achieving good performance.

Original: Step decay scheduler keeps the learning rate piecewise constant and decreases it by a factor after a given amount of epochs.
Revised: Step decay scheduler keeps the learning rate piecewise constant and decreases it by a factor after a given amount of epochs.

Original: It is theoretically proved in Ge et al.
Revised: It is theoretically proved in Ge et al.

Original: (2019) that when the objective is quadratic, step decay scheduler outperforms inverse time decay.
Revised: (2019) that when the objective is quadratic, step decay scheduler outperforms inverse time decay.

Original: Empirical results are also provided in the same work to demonstrate the better convergence property of step decay in training neural networks when compared with inverse time decay.
Revised: Empirical results are also provided in the same work to demonstrate the better convergence property of step decay in training neural networks when compared with inverse time decay.

Original: However, even step decay is proved to be near optimal on quadratic objectives, it is not truly optimal.
Revised: However, even step decay is proved to be near optimal on quadratic objectives, it is not truly optimal.

Original: There still exists a log T gap away from the minimax optimal convergence rate, which turns out to be non-trivial in a wide range of settings and may greatly impact step decay’s empirical performance.
Revised: There still exists a log T gap away from the minimax optimal convergence rate, which turns out to be non-trivial in a wide range of settings and may greatly impact step decay’s empirical performance.

Original: Cosine decay scheduler (Loshchilov & Hutter, 2017) generates cosine-like learning rates in the range [0 , T ] , with
Revised: Cosine decay scheduler (Loshchilov & Hutter, 2017) generates cosine-like learning rates in the range [0 , T ] , with

Original: T being the maximum iteration.
Revised: T being the maximum iteration.

Original: It is a heuristic scheduling strategy which relies on the observation that good performance in practice can be achieved via slowly decreasing the learning rate in the beginning and “reﬁning” the solution in the end with a very small learning rate.
Revised: It is a heuristic scheduling strategy which relies on the observation that good performance in practice can be achieved via slowly decreasing the learning rate in the beginning and “reﬁning” the solution in the end with a very small learning rate.

Original: Its convergence property on smooth non-convex functions has been shown in Li et al.
Revised: Its convergence property on smooth non-convex functions has been shown in Li et al.

Original: , but the provided bound is still not tight enough to explain its success in practice.
Revised: , but the provided bound is still not tight enough to explain its success in practice.

Original: Except cosine decay scheduler, all aforementioned learning rate schedulers have (or will have) a tight convergence bound on quadratic objectives.
Revised: Except cosine decay scheduler, all aforementioned learning rate schedulers have (or will have) a tight convergence bound on quadratic objectives.

Original: In fact, studying their convergence property on quadratic objective functions is quite important for understanding their behaviors in general nonconvex problems.
Revised: In fact, studying their convergence property on quadratic objective functions is quite important for understanding their behaviors in general nonconvex problems.

Original: Recent studies in Neural Tangent Kernel (NTK) (Arora et al., 2019; Jacot et al., 2020) suggest that when neural networks are sufﬁciently wide, the gradient descent dynamic of neural networks can be approximated by NTK.
Revised: Recent studies in Neural Tangent Kernel (NTK) (Arora et al., 2019; Jacot et al., 2020) suggest that when neural networks are sufﬁciently wide, the gradient descent dynamic of neural networks can be approximated by NTK.

Original: In particular, when the loss function is least-square loss, neural network’s inference is equivalent to kernel ridge regression with respect to the NTK in expectation.
Revised: In particular, when the loss function is least-square loss, neural network’s inference is equivalent to kernel ridge regression with respect to the NTK in expectation.

Original: In other words, for regression tasks, the non-convex objective in neural networks resembles quadratic objectives when the network is wide enough.
Revised: In other words, for regression tasks, the non-convex objective in neural networks resembles quadratic objectives when the network is wide enough.

Original: The existence of log T gap in step decay’s convergence analysis, which will be proven to be tight in a wide range of settings, implies that there is still room for improvement in theory.
Revised: The existence of log T gap in step decay’s convergence analysis, which will be proven to be tight in a wide range of settings, implies that there is still room for improvement in theory.

Original: Meanwhile, the existence of cosine decay scheduler, which has no strong theoretical convergence guarantees but possesses good empirical performance in certain tasks, suggests that its convergence rate may depend on some speciﬁc properties of the objective determined by the network and dataset in practice.
Revised: Meanwhile, the existence of cosine decay scheduler, which has no strong theoretical convergence guarantees but possesses good empirical performance in certain tasks, suggests that its convergence rate may depend on some speciﬁc properties of the objective determined by the network and dataset in practice.

Original: Hence it is natural to ask what those key properties may be, and whether it is possible to ﬁnd theoretically-optimal schedulers whose empirical performance are comparable to cosine decay if those properties are available.
Revised: Hence it is natural to ask what those key properties may be, and whether it is possible to ﬁnd theoretically-optimal schedulers whose empirical performance are comparable to cosine decay if those properties are available.

Original: In this paper, we offer an answer to these questions.
Revised: In this paper, we offer an answer to these questions.

Original: We ﬁrst derive a novel eigenvalue-distribution-based learning rate scheduler called eigencurve for the quadratic function.
Revised: We ﬁrst derive a novel eigenvalue-distribution-based learning rate scheduler called eigencurve for the quadratic function.

Original: Combining with the different eigenvalue distributions of different types of networks, new neural-network-based learning rate schedulers can be generated based on our proposed paradigm, which achieve better convergence properties than step decay in Ge et al.
Revised: Combining with the different eigenvalue distributions of different types of networks, new neural-network-based learning rate schedulers can be generated based on our proposed paradigm, which achieve better convergence properties than step decay in Ge et al.

Original: Speciﬁcally, eigencurve closes the log T gap in step decay and reaches minimax optimal convergence rates if the Hessian spectrum is skewed.
Revised: Speciﬁcally, eigencurve closes the log T gap in step decay and reaches minimax optimal convergence rates if the Hessian spectrum is skewed.

Original: We summarize the main contributions of this paper as follows.
Revised: We summarize the main contributions of this paper as follows.

Original: To the best of our knowledge, this is the ﬁrst work that incorporates the eigenvalue distribution of objective function’s Hessian matrix into designing learning rate schedulers.
Revised: To the best of our knowledge, this is the ﬁrst work that incorporates the eigenvalue distribution of objective function’s Hessian matrix into designing learning rate schedulers.

Original: Accordingly, based on the eigenvalue distribution of the Hessian, we propose a novel eigenvalue distribution based learning rate scheduler named eigencurve . 2.
Revised: Accordingly, based on the eigenvalue distribution of the Hessian, we propose a novel eigenvalue distribution based learning rate scheduler named eigencurve . 2.

Original: Theoretically, eigencurve can achieve optimal convergence rate (up to a constant) for SGD on quadratic objectives when the eigenvalue distribution of the Hessian is skewed.
Revised: Theoretically, eigencurve can achieve optimal convergence rate (up to a constant) for SGD on quadratic objectives when the eigenvalue distribution of the Hessian is skewed.

Original: Furthermore, even when the Hessian is not skewed, eigencurve can still achieve no worse convergence rate than the step decay schedule in Ge et al.
Revised: Furthermore, even when the Hessian is not skewed, eigencurve can still achieve no worse convergence rate than the step decay schedule in Ge et al.

Original: (2019), whose convergence rate are proven to be sub-optimal in a wide range of settings.
Revised: (2019), whose convergence rate are proven to be sub-optimal in a wide range of settings.

Original: Empirically, on image classiﬁcation tasks, eigencurve achieves optimal convergence rate for several models on CIFAR-10 and ImageNet if the loss can be approximated by quadratic objectives.
Revised: Empirically, on image classiﬁcation tasks, eigencurve achieves optimal convergence rate for several models on CIFAR-10 and ImageNet if the loss can be approximated by quadratic objectives.

Original: Moreover, it obtains much better performance than step decay on CIFAR-10, especially when the number of epochs is small. 4.
Revised: Moreover, it obtains much better performance than step decay on CIFAR-10, especially when the number of epochs is small. 4.

Original: Intuitively, our learning rate scheduler sheds light on the theoretical property of cosine decay and provides a perspective of understanding the reason why it can achieve good performance on image classiﬁcation tasks.
Revised: Intuitively, our learning rate scheduler sheds light on the theoretical property of cosine decay and provides a perspective of understanding the reason why it can achieve good performance on image classiﬁcation tasks.

Original: The same idea has been used to inspire and discover several simple families of schedules that works in practice.
Revised: The same idea has been used to inspire and discover several simple families of schedules that works in practice.

Original: Problem Setup For the theoretical analysis and the aim to derive our eigenvalue-dependent learning rate schedulers, we mainly focus on the quadratic function, that is,
Revised: Problem Setup For the theoretical analysis and the aim to derive our eigenvalue-dependent learning rate schedulers, we mainly focus on the quadratic function, that is,

Original: [Equation]
Revised: [Equation]

Original: ξ )] , where f
Revised: ξ )] , where f

Original: [Equation]
Revised: [Equation]

Original: w, ξ )
Revised: w, ξ )

Original: [Equation]
Revised: [Equation]

Original: where ξ denotes the data sample.
Revised: where ξ denotes the data sample.

Original: Hence, the Hessian of f ( w ) is
Revised: Hence, the Hessian of f ( w ) is

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Letting us denote b = E ξ [ b ( ξ )] , we can obtain the optima of problem (1.2)
Revised: Letting us denote b = E ξ [ b ( ξ )] , we can obtain the optima of problem (1.2)

Original: [Equation]
Revised: [Equation]

Original: Given an initial iterate w 0 and the learning rate sequence { η t } , the stochastic gradient update is
Revised: Given an initial iterate w 0 and the learning rate sequence { η t } , the stochastic gradient update is

Original: [Equation]
Revised: [Equation]

Original: We denote that
Revised: We denote that

Original: [Equation]
Revised: [Equation]

Original: and ,
Revised: and ,

Original: [Equation]
Revised: [Equation]

Original: In this paper, we assume that
Revised: In this paper, we assume that

Original: [Equation]
Revised: [Equation]

Original: The reason for this assumption is presented in Appendix G.5.
Revised: The reason for this assumption is presented in Appendix G.5.

Original: [Table]
Revised: [Table]

Original: Related Work In convergence analysis, one key property that separates SGD from vanilla gradient descent is that in
Revised: Related Work In convergence analysis, one key property that separates SGD from vanilla gradient descent is that in

Original: SGD, noise in gradients dominates.
Revised: SGD, noise in gradients dominates.

Original: In gradient descent (GD), constant learning rate can achieve linear convergence
Revised: In gradient descent (GD), constant learning rate can achieve linear convergence

Original: O ( c T ) with 0 < c < 1 for strongly convex objectives, i.e. obtaining f ( w ( t ) )
Revised: O ( c T ) with 0 < c < 1 for strongly convex objectives, i.e. obtaining f ( w ( t ) )

Original: − f ( w ∗ ) ≤ (cid:15) in O (log( 1 (cid:15) )) iterations.
Revised: − f ( w ∗ ) ≤ (cid:15) in O (log( 1 (cid:15) )) iterations.

Original: However, in SGD, f ( w ( t ) ) cannot even be guaranteed to converge to f ( w ∗ ) due to the existence of gradient noise (Bottou et al., 2018).
Revised: However, in SGD, f ( w ( t ) ) cannot even be guaranteed to converge to f ( w ∗ ) due to the existence of gradient noise (Bottou et al., 2018).

Original: Intuitively, this noise leads to a variance proportional to the learning rate size, so constant learning rate will always introduce a Ω( η t )
Revised: Intuitively, this noise leads to a variance proportional to the learning rate size, so constant learning rate will always introduce a Ω( η t )

Original: =
Revised: =

Original: Ω( η 0 ) gap when compared with the convergence rate of GD.
Revised: Ω( η 0 ) gap when compared with the convergence rate of GD.

Original: Fortunately, inverse time decay scheduler solves the problem by decaying the learning rate inversely proportional to the iteration number t , which achieves O ( 1 T ) convergence rate for strongly convex objectives, speciﬁcally, O ( dσ 2 T · κ ) .
Revised: Fortunately, inverse time decay scheduler solves the problem by decaying the learning rate inversely proportional to the iteration number t , which achieves O ( 1 T ) convergence rate for strongly convex objectives, speciﬁcally, O ( dσ 2 T · κ ) .

Original: However, this is sub-optimal since the minimax optimal rate for SGD is O ( dσ 2 T )
Revised: However, this is sub-optimal since the minimax optimal rate for SGD is O ( dσ 2 T )

Original: (Ge et al., 2019; Jain et al., 2018).
Revised: (Ge et al., 2019; Jain et al., 2018).

Original: Moreover, in practice, κ can be very big for large neural networks, which makes inverse time decay scheduler undesirable for those models.
Revised: Moreover, in practice, κ can be very big for large neural networks, which makes inverse time decay scheduler undesirable for those models.

Original: This is when step decay (Gofﬁn, 1977) comes to play.
Revised: This is when step decay (Gofﬁn, 1977) comes to play.

Original: Empirically, it is widely adopted in tasks such as image classiﬁcation and serves as a baseline for a lot of models.
Revised: Empirically, it is widely adopted in tasks such as image classiﬁcation and serves as a baseline for a lot of models.

Original: Theoretically, it has been proven that step decay can achieve nearly optimal convergence rate O ( dσ 2 T · log T ) for strongly convex least square regression (Ge et al., 2019).
Revised: Theoretically, it has been proven that step decay can achieve nearly optimal convergence rate O ( dσ 2 T · log T ) for strongly convex least square regression (Ge et al., 2019).

Original: Nevertheless, step decay is not always the best choice for image classiﬁcation tasks.
Revised: Nevertheless, step decay is not always the best choice for image classiﬁcation tasks.

Original: In practice, cosine decay (Loshchilov & Hutter, 2017) can achieve comparable or even better performance, but the reason behind this superior performance is still unknown in theory (Gotmare et al., 2018).
Revised: In practice, cosine decay (Loshchilov & Hutter, 2017) can achieve comparable or even better performance, but the reason behind this superior performance is still unknown in theory (Gotmare et al., 2018).

Original: All the aforementioned results are summarized in
Revised: All the aforementioned results are summarized in

Original: Table 1, along with our results in this paper.
Revised: Table 1, along with our results in this paper.

Original: It is worth mentioning that the minimax optimal rate
Revised: It is worth mentioning that the minimax optimal rate

Original: O ( dσ 2 T ) can be achieved by iterate averaging methods (Jain et al., 2018; Bach & Moulines, 2013;
Revised: O ( dσ 2 T ) can be achieved by iterate averaging methods (Jain et al., 2018; Bach & Moulines, 2013;

Original: D´efossez & Bach, 2015; Frostig et al., 2015; Jain et al., 2016; Neu & Rosasco, 2018), but it is not a common practice to use them in training deep neural networks, so only the ﬁnal iterate (Shamir, 2012) behavior of SGD is analyzed in this paper, i.e. the point right after the last iteration.
Revised: D´efossez & Bach, 2015; Frostig et al., 2015; Jain et al., 2016; Neu & Rosasco, 2018), but it is not a common practice to use them in training deep neural networks, so only the ﬁnal iterate (Shamir, 2012) behavior of SGD is analyzed in this paper, i.e. the point right after the last iteration.

Original: Paper organization: Section 2 describes the motivation of our eigencurve scheduler.
Revised: Paper organization: Section 2 describes the motivation of our eigencurve scheduler.

Original: Sectionpresents the exact form and convergence rate of the proposed eigencurve scheduler, along with the lower bound of step decay.
Revised: Sectionpresents the exact form and convergence rate of the proposed eigencurve scheduler, along with the lower bound of step decay.

Original: Section 4 shows the experimental results.
Revised: Section 4 shows the experimental results.

Original: Section 5 discusses the discovery and limitation of eigencurve and Section 6 gives our conclusion.
Revised: Section 5 discusses the discovery and limitation of eigencurve and Section 6 gives our conclusion.

Original:  M OTIVATION
Revised:  M OTIVATION

Original: In this section, we will give the main motivation and intuition of our eigencurve learning rate scheduler.
Revised: In this section, we will give the main motivation and intuition of our eigencurve learning rate scheduler.

Original: We ﬁrst give the scheduling strategy to achieve the optimal convergence rate in the case that the Hessian is diagonal.
Revised: We ﬁrst give the scheduling strategy to achieve the optimal convergence rate in the case that the Hessian is diagonal.

Original: Then, we show that the inverse time learning rate is sub-optimal in most cases.
Revised: Then, we show that the inverse time learning rate is sub-optimal in most cases.

Original: Comparing these two scheduling methods brings up the reason why we should design eigenvalue distribution dependent learning rate scheduler.
Revised: Comparing these two scheduling methods brings up the reason why we should design eigenvalue distribution dependent learning rate scheduler.

Original: Letting H be a diagonal matrix diag( λ 1 , λ 2 , . .
Revised: Letting H be a diagonal matrix diag( λ 1 , λ 2 , . .

Original: . , λ d ) and reformulating Eqn.
Revised: . , λ d ) and reformulating Eqn.

Original: (1.5), we have
Revised: (1.5), we have

Original: [Equation]
Revised: [Equation]

Original: It follows,
Revised: It follows,

Original: [Equation]
Revised: [Equation]

Original: Since H is diagonal, we can set step size scheduling for each dimension separately.
Revised: Since H is diagonal, we can set step size scheduling for each dimension separately.

Original: Letting us choose step size η t coordinately with the step size η t,j = 1 λ j ( t +1) being optimal for the j -th coordinate, then we have the following proposition.
Revised: Letting us choose step size η t coordinately with the step size η t,j = 1 λ j ( t +1) being optimal for the j -th coordinate, then we have the following proposition.

Original: Proposition 1.
Revised: Proposition 1.

Original: Assume that H is diagonal matrix with eigenvalues
Revised: Assume that H is diagonal matrix with eigenvalues

Original: λ 1 ≥ λ 2 ≥ . .
Revised: λ 1 ≥ λ 2 ≥ . .

Original: . , λ d ≥ 0 and
Revised: . , λ d ≥ 0 and

Original: Eqn. (1.7) holds.
Revised: Eqn. (1.7) holds.

Original: If we set step size η t,j = 1 λ j ( t +1) , it holds that
Revised: If we set step size η t,j = 1 λ j ( t +1) , it holds that

Original: [Equation]
Revised: [Equation]

Original: The  equality here is proved in Lemma 15. From Eqn.
Revised: The leading equality here is proved in Appendix G.1, with the followed inequality proved in Appendix E. From Eqn.

Original: (2.2), we can observe that choosing proper step sizes coordinately can achieve the optimal convergence rate (Ge et al., 2019; Jain et al., 2018).
Revised: (2.2), we can observe that choosing proper step sizes coordinately can achieve the optimal convergence rate (Ge et al., 2019; Jain et al., 2018).

Original: Instead, if the widely used inverse time scheduler η t = 1 / ( L + µt ) is chosen, we can show that only a sub-optimal convergence rate can be obtained, especially when λ j ’s vary from each other.
Revised: Instead, if the widely used inverse time scheduler η t = 1 / ( L + µt ) is chosen, we can show that only a sub-optimal convergence rate can be obtained, especially when λ j ’s vary from each other.

Original: Proposition 2.
Revised: Proposition 2.

Original: If we set the inverse time step size η t = 1 ( L + µt ) , then we have
Revised: If we set the inverse time step size η t = 1 ( L + µt ) , then we have

Original: [Equation]
Revised: [Equation]

Original: Remark 1.
Revised: Remark 1.

Original: Eqn.
Revised: Eqn.

Original: (2.2) shows that if one can choose step size coordinately with step size η t,j =λ j ( t +1) , then SGD can achieve a convergence rate
Revised: (2.2) shows that if one can choose step size coordinate-wise with step size η t,j =λ j ( t +1) , then SGD can achieve a convergence rate

Original: [Equation]
Revised: [Equation]

Original: which matches the lower bound (Ge et al., 2019; Jain et al., 2018)
Revised: which matches the lower bound (Ge et al., 2019; Jain et al., 2018)

Original: In contrast, replacing L = λand µ = λ d in Proposition 2, we can obtain that the convergence rate of SGD being
Revised: In contrast, replacing L = λand µ = λ d in Proposition 2, we can obtain that the convergence rate of SGD being

Original: [Equation]
Revised: [Equation]

Original: Since it holds that λ j ≥ λ d , the convergence rate in Eqn.
Revised: Since it holds that λ j ≥ λ d , the convergence rate in Eqn.

Original: (2.4) is better than the one in Eqn.
Revised: (2.4) is better than the one in Eqn.

Original: (2.5) , especially when the eigenvalues of the Hessian ( H matrix) decay rapidly.
Revised: (2.5) , especially when the eigenvalues of the Hessian ( H matrix) decay rapidly.

Original: In fact, the upper bound in Eqn.
Revised: In fact, the upper bound in Eqn.

Original: (2.5) is tight for the inverse time decay scheduling (Ge et al., 2019).
Revised: (2.5) is tight for the inverse time decay scheduling (Ge et al., 2019).

Original: Main Intuition Though SGD with coordinately proper step sizes can obtain the optimal convergence rate, since H is commonly non-diagonal in practice, one is not able to choose proper step sizes coordinately.
Revised: Main Intuition Though SGD with coordinately proper step sizes can obtain the optimal convergence rate, since H is commonly non-diagonal in practice, one is not able to choose proper step sizes coordinately.

Original: However, the diagonal case still provide an important intuition for designing eigenvalue dependent learning rate scheduling.
Revised: However, the diagonal case still provide an important intuition for designing eigenvalue dependent learning rate scheduling.

Original: In fact, letting H =
Revised: In fact, letting H =

Original: U Λ U (cid:62) be the spectral decomposition of the Hessian and setting w (cid:48)
Revised: U Λ U (cid:62) be the spectral decomposition of the Hessian and setting w (cid:48)

Original: =
Revised: =

Original: U (cid:62) w , then the Hessian becomes a diagonal matrix from perspective of updating w (cid:48) , with the variance of the stochastic gradient being unchanged since U is a unitary matrix.
Revised: U (cid:62) w , then the Hessian becomes a diagonal matrix from perspective of updating w (cid:48) , with the variance of the stochastic gradient being unchanged since U is a unitary matrix.

Original: Proposition 1 and 2 imply that a good learning rate scheduler should decrease the error of each coordinate.
Revised: Proposition 1 and 2 imply that a good learning rate scheduler should decrease the error of each coordinate.

Original: The inverse time decay scheduler is only optimal for the coordinate related to the smallest eigenvalue.
Revised: The inverse time decay scheduler is only optimal for the coordinate related to the smallest eigenvalue.

Original: That’s the reason why the inverse time scheduler is sub-optimal.
Revised: That’s the reason why the inverse time scheduler is sub-optimal.

Original: Thus, we should reduce the learning rate gradually such that we can run a optimal learning rate associated to λ j to sufﬁciently drop the error of j -th coordinate.
Revised: Thus, we should reduce the learning rate gradually such that we can run a optimal learning rate associated to λ j to sufﬁciently drop the error of j -th coordinate.

Original: Furthermore, given the total iteration T and the eigenvalue distribution of the Hessian, we should allocate the running time for each optimal learning rate associated to λ j .
Revised: Furthermore, given the total iteration T and the eigenvalue distribution of the Hessian, we should allocate the running time for each optimal learning rate associated to λ j .

Original:  E IGENVALUE D EPENDENT S TEP S CHEDULING
Revised:  E IGENVALUE D EPENDENT S TEP S CHEDULING

Original: Just as discussed in Section 2, to obtain better convergence rate for SGD, we should consider Hessian’s eigenvalue distribution and schedule the learning rate based on the distribution.
Revised: Just as discussed in Section 2, to obtain better convergence rate for SGD, we should consider Hessian’s eigenvalue distribution and schedule the learning rate based on the distribution.

Original: In this section, we propose a novel learning rate scheduler for this task, which can be regarded as piecewise inverse time decay (see Figure 1).
Revised: In this section, we propose a novel learning rate scheduler for this task, which can be regarded as piecewise inverse time decay (see Figure 1).

Original: The method is very simple, we group eigenvalues according to their value and denote s i to be the number of eigenvalues lie in the range R i =
Revised: The method is very simple, we group eigenvalues according to their value and denote s i to be the number of eigenvalues lie in the range R i =

Original: [ µ · 2 i , µ · 2 i +1 ) , that is,
Revised: [ µ · 2 i , µ · 2 i +1 ) , that is,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Then, there are at most I max = log 2 κ such ranges.
Revised: Then, there are at most I max = log 2 κ such ranges.

Original: By the inverse time decay theory, the optimal learning rate associated to eigenvalues in the range R i should be
Revised: By the inverse time decay theory, the optimal learning rate associated to eigenvalues in the range R i should be

Original: [Equation]
Revised: [Equation]

Original: Our scheduling strategy is to run the optimal learning rate for eigenvalues in each R i for a period to sufﬁciently decrease the error associated to eigenvalues in R i .
Revised: Our scheduling strategy is to run the optimal learning rate for eigenvalues in each R i for a period to sufﬁciently decrease the error associated to eigenvalues in R i .

Original: To make the step size sequence { η t } Tt =1 monotonely decreasing, we deﬁne the step sizes as
Revised: To make the step size sequence { η t } Tt =1 monotonely decreasing, we deﬁne the step sizes as

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: To make the total error, that is the sum of error associated with R i , to be small, we should allocate ∆ i according to s i − 1 ’s. Intuitively, a large portion of eigenvalues lying in the range R i should allocate a large portion of iterations.
Revised: To make the total error, that is the sum of error associated with R i , to be small, we should allocate ∆ i according to s i − 1 ’s. Intuitively, a large portion of eigenvalues lying in the range R i should allocate a large portion of iterations.

Original: Speciﬁcally, we propose to allocate ∆ i as follows:
Revised: Speciﬁcally, we propose to allocate ∆ i as follows:

Original: [Equation]
Revised: [Equation]

Original: In the rest of this section, we will show that the step size scheduling according to Eqn.
Revised: In the rest of this section, we will show that the step size scheduling according to Eqn.

Original: (3.3) and (3.5) can achieve better convergence rate than the one in Ge et al.
Revised: (3.3) and (3.5) can achieve better convergence rate than the one in Ge et al.

Original: (2019) when s i is non-uniformly distributed.
Revised: (2019) when s i is non-uniformly distributed.

Original: In fact, a better ∆ i allocation can be calculated using numerical optimization.
Revised: In fact, a better ∆ i allocation can be calculated using numerical optimization.

Original: 3. T HEORETICAL A NALYSIS
Revised: 3. T HEORETICAL A NALYSIS

Original: Lemma 1.
Revised: Lemma 1.

Original: Let objective function f ( x ) be quadratic and Assumption (1.7) hold.
Revised: Let objective function f ( x ) be quadratic and Assumption (1.7) hold.

Original: Running SGD for
Revised: Running SGD for

Original: T -steps starting from w 0 and a learning rate sequence { η t } T t =1 deﬁned in Eqn.
Revised: T -steps starting from w 0 and a learning rate sequence { η t } T t =1 deﬁned in Eqn.

Original: (3.3) , the ﬁnal iterate w T +1 satisﬁes
Revised: (3.3) , the ﬁnal iterate w T + 1 satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Since the bias term is a high order term, the variance term in Eqn.
Revised: Since the bias term is a high order term, the variance term in Eqn.

Original: (3.6) dominates the error for w T +1 .
Revised: (3.6) dominates the error for w T +1 .

Original: For simplicity, instead of using numerical methods to ﬁnd the optimal { ∆ i } , we propose to use ∆ i deﬁned in Eqn.
Revised: For simplicity, instead of using numerical methods to ﬁnd the optimal { ∆ i } , we propose to use ∆ i deﬁned in Eqn.

Original: The value of ∆ i is linear to square root of the number of eigenvalues lying in the range [ µ · 2 i − 1 , µ · 2 i ) .
Revised: The value of ∆ i is linear to square root of the number of eigenvalues lying in the range [ µ · 2 i − 1 , µ · 2 i ) .

Original: Using such ∆ i , eigencurve has the following convergence property.
Revised: Using such ∆ i , eigencurve has the following convergence property.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let objective function f ( x ) be quadratic and Assumption (1.7) hold.
Revised: Let objective function f ( x ) be quadratic and Assumption (1.7) hold.

Original: Running SGD for
Revised: Running SGD for

Original: T -steps starting from w 0 , a learning rate sequence { η t } Tt =1 deﬁned in Eqn.
Revised: T -steps starting from w 0 , a learning rate sequence { η t } Tt =1 deﬁned in Eqn.

Original: (3.3) and ∆ i deﬁned in
Revised: (3.3) and ∆ i deﬁned in

Original: Eqn.
Revised: Eqn.

Original: (3.5) , the ﬁnal iterate w T +1 satisﬁes
Revised: (3.5) , the ﬁnal iterate w T +1 satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Please refer to Appendix D, F and G for the full proof of Lemma 1 and Theorem 1.
Revised: Please refer to Appendix D, F and G for the full proof of Lemma 1 and Theorem 1.

Original: The variance term 15 (
Revised: The variance term 15 (

Original: [Equation]
Revised: [Equation]

Original:  · σ 2 in above theorem shows that when s i ’s vary largely from each other, then the variance can be close to O (cid:0) dT · σ 2 (cid:1) which matches the lower bound (Ge et al., 2019).
Revised: I [Equation] T · σ 2 in above theorem shows that when s i ’s vary largely from each other, then the variance can be close to O (cid:0) dT · σ 2 (cid:1) which matches the lower bound (Ge et al., 2019).

Original: For example, letting I max = 100 , s 0 = 0 .
Revised: For example, letting I max = 100 , s 0 = 0 .

Original: 99 d and s i = 0 .
Revised: 99 d and s i = 0 .

Original: 0199 d , we can obtain that
Revised: 0199 d , we can obtain that

Original: [Equation]
Revised: [Equation]

Original: We can observe that if the variance of s i ’s is large, the variance term in Theorem 1 can be close to dσ 2 /T .
Revised: We can observe that if the variance of s i ’s is large, the variance term in Theorem 1 can be close to dσ 2 /T .

Original: More generally, as rigorously stated in Corollary 2, eigencurve achieves minimax optimal convergence rate if the Hessian spectrum satisﬁes an extra assumption of what-we-called “power power law”: the density of eigenvalue λ is exponentially decaying with increasing value of λ in log scale, i.e. ln( λ ) .
Revised: More generally, as rigorously stated in Corollary 2, eigencurve achieves minimax optimal convergence rate if the Hessian spectrum satisﬁes an extra assumption of what-we-called “power power law”: the density of eigenvalue λ is exponentially decaying with increasing value of λ in log scale, i.e. ln( λ ) .

Original: This assumption comes from the observation of estimated Hessian spectrums in practice (see Figure 2), which will be further illustrated in Section 4.1.
Revised: This assumption comes from the observation of estimated Hessian spectrums in practice (see Figure 2), which will be further illustrated in Section 4.1.

Original: Corollary 2.
Revised: Corollary 2.

Original: Given the same setting as in Theorem 1, when Hessian H ’s eigenvalue distribution p ( λ ) satisﬁes “power power law”, i.e.
Revised: Given the same setting as in Theorem 1, when Hessian H ’s eigenvalue distribution p ( λ ) satisﬁes “power power law”, i.e.

Original: [Equation]
Revised: [Equation]

Original: for some α > 1 , where Z = (cid:82) L µ ( µ/λ ) α dλ , there exists a constant C ( α ) which only depends on α , such that the ﬁnal iterate w T +1 satisﬁes
Revised: for some α > 1 , where Z = (cid:82) L µ ( µ/λ ) α dλ , there exists a constant C ( α ) which only depends on α , such that the ﬁnal iterate w T +1 satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Please refer to Appendix G.3 for the proof.
Revised: Please refer to Appendix G.3 for the proof.

Original: As for the worst-case guarantee, it is easy to check that only when s i ’s are equal to each other, that is, s i =
Revised: As for the worst-case guarantee, it is easy to check that only when s i ’s are equal to each other, that is, s i =

Original: d/I max = d/ log 2 ( κ ) , the variance term reaches its maximum.
Revised: d/I max = d/ log 2 ( κ ) , the variance term reaches its maximum.

Original: Corollary 3.
Revised: Corollary 3.

Original: Given the same setting as in Theorem 1, when s i = d/ log 2 ( κ ) for all 0 ≤ i ≤
Revised: Given the same setting as in Theorem 1, when s i = d/ log 2 ( κ ) for all 0 ≤ i ≤

Original: I max − 1 , the variance term in Theorem 1 reaches its maximum and w T +1 satisﬁes
Revised: I max − 1 , the variance term in Theorem 1 reaches its maximum and w T +1 satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Remark 2.
Revised: Remark 2.

Original: When s i ’s vary from each other, our eigenvalue dependent learning rate scheduler can achieve faster convergence rate than eigenvalue independent scheduler such as step decay which suffers from an extra log ( T ) term (Ge et al., 2019).
Revised: When s i ’s vary from each other, our eigenvalue dependent learning rate scheduler can achieve faster convergence rate than eigenvalue independent scheduler such as step decay which suffers from an extra log ( T ) term (Ge et al., 2019).

Original: Only when s i ’s are equal to each other, Corollary 3 shows that the bound of variance matches to lower bound up to log κ which is same to the one in Proposition 3 of Ge et al.
Revised: Only when s i ’s are equal to each other, Corollary 3 shows that the bound of variance matches to lower bound up to log κ which is same to the one in Proposition 3 of Ge et al.

Original: Furthermore, we show that this log T gap between step decay and eigencurve certainly exists for problem instances of skewed Hessian spectrums.
Revised: Furthermore, we show that this log T gap between step decay and eigencurve certainly exists for problem instances of skewed Hessian spectrums.

Original: For simplicity, we only discuss the case where
Revised: For simplicity, we only discuss the case where

Original: H is diagonal.
Revised: H is diagonal.

Original: Theorem 4.
Revised: Theorem 4.

Original: Let objective function f ( x ) be quadratic.
Revised: Let objective function f ( x ) be quadratic.

Original: We run SGD for T -steps starting from w 0 and a step decay learning rate sequence { η t } Tt =1 deﬁned in Algorithm 1 of Ge et al.
Revised: We run SGD for T -steps starting from w 0 and a step decay learning rate sequence { η t } Tt =1 deﬁned in Algorithm 1 of Ge et al.

Original: (2019) with η 1 = 1 /L .
Revised: (2019) with η 1 = 1 /L .

Original: As long as (1) H is diagonal, (2) The equality in Assumption (1.7) holds, i.e. E ξ (cid:2) n t n (cid:62) t (cid:3) =
Revised: As long as (1) H is diagonal, (2) The equality in Assumption (1.7) holds, i.e. E ξ (cid:2) n t n (cid:62) t (cid:3) =

Original: σ 2 H and (3) T/
Revised: σ 2 H and (3) T/

Original: log T ≥ max( κ, 2 16 , 16) , the ﬁnal iterate w T +1 satisﬁes,
Revised: log T ≥ max( κ, 2 16 , 16) , the ﬁnal iterate w T +1 satisﬁes,

Original: [Equation]
Revised: [Equation]

Original: The proof is provided in Appendix G.4.
Revised: The proof is provided in Appendix G.4.

Original: Removing this extra log T term may not seem to be a big deal in theory, but experimental results suggest the opposite.
Revised: Removing this extra log T term may not seem to be a big deal in theory, but experimental results suggest the opposite.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To demonstrate eigencurve ’s practical value, empirical experiments are conducted on the task of image classiﬁcation 1 .
Revised: To demonstrate eigencurve ’s practical value, empirical experiments are conducted on the task of image classiﬁcation 1 .

Original: Two well-known dataset are used: CIFAR-10 (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009).
Revised: Two well-known dataset are used: CIFAR-10 (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009).

Original: For full experimental results on more datasets, please refer to
Revised: For full experimental results on more datasets, please refer to

Original: Appendix A.
Revised: Appendix A.

Original: 4. H ESSIAN S PECTRUM ’ S S KEWNESS IN P RACTICE
Revised: 4. H ESSIAN S PECTRUM ’ S S KEWNESS IN P RACTICE

Original: According to estimated 2 eigenvalue distributions of Hessian on CIFAR-10 and ImageNet, as shown in Figure 3, it can be observed that all of them are highly skewed and share a similar tendency: A large portion of small eigenvalues and a tiny portion of large eigenvalues .
Revised: According to estimated 2 eigenvalue distributions of Hessian on CIFAR-10 and ImageNet, as shown in Figure 3, it can be observed that all of them are highly skewed and share a similar tendency: A large portion of small eigenvalues and a tiny portion of large eigenvalues .

Original: This phenomenon has also been observed and explained by other researchers in the past(Sagun et al., 2017; Arjevani &
Revised: This phenomenon has also been observed and explained by other researchers in the past(Sagun et al., 2017; Arjevani &

Original: Field, 2020).
Revised: Field, 2020).

Original: On top of that, when we plot both eigenvalues and density in log scale, the “power power law” arises.
Revised: On top of that, when we plot both eigenvalues and density in log scale, the “power power law” arises.

Original: Therefore, if the loss surface can be approximated by quadratic objectives [Figure] then eigencurve has already achieved optimal convergence rate for those practical settings.
Revised: Therefore, if the loss surface can be approximated by quadratic objectives, then eigencurve has already achieved optimal convergence rate for those practical settings.

Original: The exact values of the extra constant terms are presented in Appendix A. [Figure]
Revised: The exact values of the extra constant terms are presented in Appendix A.2.

Original: 4. I MAGE C LASSIFICATION ON CIFAR-10 WITH E IGENCURVE S CHEDULING
Revised: 4. I MAGE C LASSIFICATION ON CIFAR-10 WITH E IGENCURVE S CHEDULING

Original: This optimality in theory induces eigencurve ’s superior performance in practice, which is demonstrated in Table 2 and Figure 4.
Revised: This optimality in theory induces eigencurve ’s superior performance in practice, which is demonstrated in Table 2 and Figure 4.

Revised: The full set of ﬁgures is available in Appendix A.7.

Original: All models are trained with stochastic gradient descent (SGD), no momentum, batch size 128 and weight decay wd = 0 . 0005 .
Revised: All models are trained with stochastic gradient descent (SGD), no momentum, batch size 128 and weight decay wd = 0 . 0005 .

Original: For full details of the experiment setup, please refer to Appendix B.
Revised: For full details of the experiment setup, please refer to Appendix B.

Original: [Table]  D ISCUSSION
Revised: 4. I NSPIRED P RACTICAL S CHEDULES WITH S IMPLE F ORMS

Original: Inspired Practical Schedules with Simple Forms By simplifying the form of eigencurve and capturing some of its key properties, two simple and practical schedules are proposed: Elastic Step Decay and Cosine-power Decay, whose empirical performance are better than or at least comparable to cosine decay.
Revised: By simplifying the form of eigencurve and capturing some of its key properties, two simple and practical schedules are proposed: Elastic Step Decay and Cosine-power Decay, whose empirical performance are better than or at least comparable to cosine decay.

Original: Due to page limit, we leave all the experimental results in Appendix A.4, A.5.
Revised: Due to page limit, we leave all the experimental results in Appendix A.4, A.5, A.6. [Equation] α (4.2) [Figure]

Original: Elastic
Revised: i t y ( [Figure] i t [Figure]

Revised: i

Original: [Equation]
Revised: [Figure] [Table]

Original: Cosine-power
Revised: 0.36 Cosine decay 1.42 ± 0. 84.23 ± 0. 1.94 ± 0. 90.56 ± 0.

Revised: 2.03 ± 0.

Revised: 87.99 ± 0.13 Eigencurve 1.36 ± 0.

Revised: 85.62 ± 0.

Revised: 1.33 ± 0.

Original: [Equation]
Revised: 90.65 ± 0. 1.87 ± 0. 88.73 ± 0. =

Revised: Inverse Time Decay 0.73 ± 0.

Revised: 90.82 ± 0.

Revised: 0.62 ± 0.

Revised: 92.05 ± 0.

Revised: 1.32 ± 0.

Revised: *76.24 ± 13.77 Step Decay 0.26 ± 0.

Revised: 91.39 ± 1.

Revised: 0.28 ± 0.

Revised: 92.83 ± 0.

Revised: 0.59 ± 0.

Revised: 91.37 ± 0.20 General Step Decay 0.17 ± 0.

Revised: 93.97 ± 0.

Revised: 0.13 ± 0.

Revised: 94.18 ± 0.

Revised: 0.20 ± 0.

Revised: *92.36 ± 0.46 Cosine Decay 0.17 ± 0.

Revised: 94.04 ± 0.

Revised: 0.12 ± 0.

Revised: 94.62 ± 0.

Revised: 0.20 ± 0.

Revised: 93.17 ± 0.05 Eigencurve 0.14 ± 0.

Revised: 94.05 ± 0.

Revised: 0.12 ± 0.

Revised: 94.75 ± 0.

Original:  Cosine Decay and Eigencurve For ResNet- 18 on CIFAR-10 dataset, eigencurve scheduler presents an extremely similar learning rate curve to cosine decay, especially when the number of training epochs is set to 100 , as shown in Figure 5.
Revised: 0.18 ± 0. 92.88 ± 0.  D ISCUSSION Cosine Decay and Eigencurve For ResNet- 18 on CIFAR-10 dataset, eigencurve scheduler presents an extremely similar learning rate curve to cosine decay, especially when the number of training epochs is set to 100 , as shown in Figure 5.

Original: This directly links cosine decay to our theory: the empirically superior performance of cosine decay is very likely to stem from the utilization of the “skewness” among Hessian matrix’s eigenvalues.
Revised: This directly links cosine decay to our theory: the empirically superior performance of cosine decay is very likely to stem from the utilization of the “skewness” among Hessian matrix’s eigenvalues.

Original: For other situations, especially when the number of iterations is small, as shown in Table 2, eigencurve presents a better performance than cosine decay .
Revised: For other situations, especially when the number of iterations is small, as shown in Table 2, eigencurve presents a better performance than cosine decay .

Original: Sensitiveness to Hessian’s Eigenvalue Distributions One limitation of eigencurve is that it requires a precomputed eigenvalue distribution of objective functions’s Hessian matrix, which can be time-consuming for large models.
Revised: Sensitiveness to Hessian’s Eigenvalue Distributions One limitation of eigencurve is that it requires a precomputed eigenvalue distribution of objective functions’s Hessian matrix, which can be time-consuming for large models.

Original: This issue can be overcome by reusing the estimated eigenvalue distribution from similar settings.
Revised: This issue can be overcome by reusing the estimated eigenvalue distribution from similar settings.

Original: Further experiments on CIFAR-10 suggest the effectiveness of this approach.
Revised: Further experiments on CIFAR-10 suggest the effectiveness of this approach.

Original: Please refer to Appendix A.3 for more details.
Revised: Please refer to Appendix A.3 for more details.

Original: This evidence suggests that eigencurve ’s performance is not very sensitive to estimated eigenvalue distributions.
Revised: This evidence suggests that eigencurve ’s performance is not very sensitive to estimated eigenvalue distributions.

Original: C ONCLUSION
Revised: Relationship with Numerically Near-optimal Schedulers In Zhang et al. (2019), a dynamic programming algorithm was proposed to ﬁnd almost optimal schedulers if the exact loss of the quadratic objective is accessible. While it is certainly the case, Eigencurve still possesses several additional advantages when comparing with this method. First, Eigencurve can be used to ﬁnd simple-formed schedulers.

Revised: Compared to schedulers numerically computed by dynamic programming, Eigencurve provide an analytic framework, so it is able to bypass the Hessian spectrum estimation if some useful assumptions of the Hessian spectrum can be obtained, such as ”power power law”.

Revised: Second, Eigencurve has a clear theoretical convergence guarantee.

Revised: Dynamic programming can ﬁnd almost optimal schedulers, but the convergence property of the computed scheduler is still unclear.

Original:  In this paper, a novel learning rate schedule named eigencurve is proposed, which utilizes the “skewness” of objective’s Hessian matrix’s eigenvalue distribution and reaches minimax optimal convergence rates for SGD on quadratic objectives with skewed Hessian spectrums.
Revised: Our work ﬁlls this gap.  C ONCLUSION In this paper, a novel learning rate schedule named eigencurve is proposed, which utilizes the “skewness” of objective’s Hessian matrix’s eigenvalue distribution and reaches minimax optimal convergence rates for SGD on quadratic objectives with skewed Hessian spectrums.

Original: This condition of skewed Hessian spectrums is observed and indeed satisﬁed in practical settings of image classiﬁcation.
Revised: This condition of skewed Hessian spectrums is observed and indeed satisﬁed in practical settings of image classiﬁcation.

Original: Theoretically, eigencurve achieves no worse convergence guarantee than step decay for quadratic functions and reaches minimax optimal convergence rate (up to a constant) with skewed Hessian spectrums, e.g. under “power power law”.
Revised: Theoretically, eigencurve achieves no worse convergence guarantee than step decay for quadratic functions and reaches minimax optimal convergence rate (up to a constant) with skewed Hessian spectrums, e.g. under “power power law”.

Original: Empirically, experimental results on
Revised: Empirically, experimental results on

Original: CIFAR-10 show that eigencurve signiﬁcantly outperforms step decay, especially when the number of epochs is small.
Revised: CIFAR-10 show that eigencurve signiﬁcantly outperforms step decay, especially when the number of epochs is small.

Original: The idea of eigencurve offers a possible explanation for cosine decay’s effectiveness in practice and inspires two practical families of schedules with simple forms.
Revised: The idea of eigencurve offers a possible explanation for cosine decay’s effectiveness in practice and inspires two practical families of schedules with simple forms.

Original: A M ORE E XPERIMENTAL R ESULTS A. R IDGE R EGRESSION
Revised: A M ORE E XPERIMENTAL R ESULTS A. R IDGE R EGRESSION

Original: We compare different types of schedulings on ridge regression
Revised: We compare different types of schedulings on ridge regression

Original: [Equation]
Revised: [Equation]

Original: This experiment is only an empirical proof of our theory.
Revised: This experiment is only an empirical proof of our theory.

Original: In fact, the optima of ridge regression has a closed form and can be directly computed with
Revised: In fact, the optima of ridge regression has a closed form and can be directly computed with

Original: [Equation]
Revised: [Equation]

Original: Thus the optimal training loss f ( w ∗ ) can be calculated accordingly.
Revised: Thus the optimal training loss f ( w ∗ ) can be calculated accordingly.

Original: In all experiments, we use the loss gap f ( w T ) − f ( w ∗ ) as our performance metric.
Revised: In all experiments, we use the loss gap f ( w T ) − f ( w ∗ ) as our performance metric.

Original: Experiments are conducted on a4a datasets (Chang & Lin, 2011; Dua & Graff, 2017)
Revised: Experiments are conducted on a4a datasets (Chang & Lin, 2011; Dua & Graff, 2017)

Original: ( https:// www.csie.ntu.edu.tw/˜cjlin/libsvmtools/datasets/binary.html#a4a/ ), which contains 4 , 781 samples and 123 features.
Revised: ( https:// www.csie.ntu.edu.tw/˜cjlin/libsvmtools/datasets/binary.html#a4a/ ), which contains 4 , 781 samples and 123 features.

Original: This dataset is chosen majorly because it has a moderate number of samples and features, which enables us to compute the exact Hessian matrix
Revised: This dataset is chosen majorly because it has a moderate number of samples and features, which enables us to compute the exact Hessian matrix

Original: H = 2( X (cid:62) X/n + αI ) and its corresponding eigenvalue distribution in acceptable time and space consumption.
Revised: H = 2( X (cid:62) X/n + αI ) and its corresponding eigenvalue distribution in acceptable time and space consumption.

Original: In all of our experiments, we set α = 10 − 3 .
Revised: In all of our experiments, we set α = 10 − 3 .

Original: The model is optimized via SGD without momentum, with  initial learning rate η 0 ∈ { 0 . 1 , 0 . 06 , 0 . 
Revised: The model is optimized via SGD without momentum, with batch size 1, initial learning rate η 0 ∈ { 0 . 1 , 0 . 06 , 0 . 03 , 0 .

Original: 0003 , 0 . 0002 , 0 . 0001 } and learning rate of last iteration η min ∈ { 0 . 1 , 0 . 01 , 0 .
Revised: 0003 , 0 . 0002 , 0 . 0001 } and learning rate of last iteration η min ∈ { 0 . 1 , 0 . 01 , 0 .

Original: 001 , 0 . 0001 , 0 . 00001 , 0 , “UNRESTRICTED” } .
Revised: 001 , 0 . 0001 , 0 . 00001 , 0 , “UNRESTRICTED” } .

Original: Here “UNRESTRICTED” denotes the case where η min is not set, which is useful for eigencurve , who can decide the learning rate curve without setting η min .
Revised: Here “UNRESTRICTED” denotes the case where η min is not set, which is useful for eigencurve , who can decide the learning rate curve without setting η min .

Original: Given η 0 and η min , we adjust all schedulers as follows.
Revised: Given η 0 and η min , we adjust all schedulers as follows.

Original: For inverse time decay η t = η 0 / (1 + γη 0 t ) and exponential decay η t = γ t η 0 , the hyperparameter γ is computed accordingly based on η 0 and η min .
Revised: For inverse time decay η t = η 0 / (1 + γη 0 t ) and exponential decay η t = γ t η 0 , the hyperparameter γ is computed accordingly based on η 0 and η min .

Original: For cosine decay, η 0 and η min is directly used, with no restart adopted.
Revised: For cosine decay, η 0 and η min is directly used, with no restart adopted.

Original: For eigencurve , the learning rate curve is linearly scaled to match the given η min .
Revised: For eigencurve , the learning rate curve is linearly scaled to match the given η min .

Original: In addition, for eigencurve , weset β = 2 . 0 and use the eigenvalue distribution of the Hessian matrix. This distribution is directly computed via eigenvalue decomposition, as shown in Figure 6.
Revised: In addition, for eigencurve , we use the eigenvalue distribution of the Hessian matrix, which is directly computed via eigenvalue decomposition, as shown in Figure 6.

Original: All experimental results demonstrate that eigencurve can obtain similar or better training losses when compared with other schedulers, as shown in Table 3 and Figure 7.
Revised: All experimental results demonstrate that eigencurve can obtain similar or better training losses when compared with other schedulers, as shown in Table 3.

Original: [Table]
Revised: [Table]

Original: A. E XACT V ALUE OF THE E XTRA T ERM ON CIFAR-10 E XPERIMENTS
Revised: A. E XACT V ALUE OF THE E XTRA T ERM ON CIFAR-10 E XPERIMENTS

Original: In Section 4.
Revised: In Section 4.

Original: [Table]
Revised: [Table]

Original: we have already given the qualitative evidence that shows eigencurve ’s optimality for practical settings on CIFAR-10.
Revised: we have already given the qualitative evidence that shows eigencurve ’s optimality for practical settings on CIFAR-10.

Original: Here we strengthen this argument by providing the quantitative evidence as well.
Revised: Here we strengthen this argument by providing the quantitative evidence as well.

Original: The exact value of the extra term is presented in Table
Revised: The exact value of the extra term is presented in Table

Original: [Table]
Revised: [Table]

Original: where we assume CIFAR-10 has batch size 128 , number of epochs 200 and weight decay 5 × 10 − 4 , while
Revised: where we assume CIFAR-10 has batch size 128 , number of epochs 200 and weight decay 5 × 10 − [Table]

Original:  ImageNet has batch size 256 , number of epochs 90 and weight decay 10 − 4 .
Revised: while ImageNet has batch size 256 , number of epochs 90 and weight decay 10 − 4 .

Original: [Table]
Revised: [Table]

Original: It is worth noticing that the extra term’s value of eigencurve is independent from the number of iterations T .
Revised: It is worth noticing that the extra term’s value of eigencurve is independent from the number of iterations T .

Original: So basically eigencurve has already achieved the minimax optimal rate (up to a constant) for models and datasets listed in Table [Table] if the loss landscape around the optima can be approximated by quadratic functions.
Revised: So basically eigencurve has already achieved the minimax optimal rate (up to a constant) for models and datasets listed in Table 4, if the loss landscape around the optima can be approximated by quadratic functions.

Original: For full details of the estimation process, please refer to
Revised: For full details of the estimation process, please refer to

Original: Appendix B
Revised: Appendix B.

Original: [Table]

Original: A. R EUSING EIGENCURVE FOR D IFFERENT M ODELS ON CIFAR- 
Revised: A. R EUSING EIGENCURVE FOR D IFFERENT M ODELS ON CIFAR-

Original: For image classiﬁcation tasks on CIFAR-10, we check the performance of reusing ResNet-18’s eigenvalue distribution for other models.
Revised: For image classiﬁcation tasks on CIFAR-10, we check the performance of reusing ResNet-18’s eigenvalue distribution for other models.

Original: As shown in Table 5, experimental results demonstrate that
Revised: As shown in Table 5, experimental results demonstrate that

Original: Hessian’s eigenvalue distribution of Resnet-18 on CIFAR-10 can be applied to GoogLeNet/VGG and still achieves good peformance.
Revised: Hessian’s eigenvalue distribution of Resnet-18 on CIFAR-10 can be applied to GoogLeNet/VGG and still achieves good peformance.

Original: Here the experiment settings are exactly the same as Section 4.
Revised: Here the experiment settings are exactly the same as Section 4.

Original: in main paper. [Table]
Revised: in main paper, except we only run one trial of experiment and use common step decay with three intervals + decay factor 10.

Original: A. I MAGE N ET C LASSIFICATION ON E LASTIC S TEP D ECAY
Revised: A. I MAGE N ET C LASSIFICATION WITH E LASTIC S TEP D ECAY

Original: One key observation in CIFAR-10 experiments is the existence of “power power law” shown in
Revised: One key observation in CIFAR-10 experiments is the existence of “power power law” shown in

Original: Figure 3.
Revised: Figure 3.

Original: Also, notice that in the form of eigencurve , speciﬁcally Eqn.
Revised: Also, notice that in the form of eigencurve , speciﬁcally Eqn.

Original: (3.5), iteration interval  length ∆ i is proportional to the square root of eigenvalue density s i in range [ µ · 2 i , µ · 2 i +1 ) .
Revised: (3.5), iteration interval [Table] length ∆ i is proportional to the square root of eigenvalue density s i in range [ µ · 2 i , µ · 2 i +1 ) .

Original: Combining those two facts together, it suggests the length of “learning rate interval” should have lengths exponentially decreasing.
Revised: Combining those two facts together [Table] it suggests the length of “learning rate interval” should have lengths exponentially decreasing.

Original: Based on this idea, Elastic Step Decay (ESD) is proposed, which has the following form,
Revised: Based on this idea [Table] Elastic Step Decay (ESD) is proposed, which has the following form,

Original: [Equation]
Revised: [Equation]

Original: Compared to general step decay with adjustable interval lengths, elastic step decay does not require manual adjustment for the length of each interval.
Revised: Compared to general step decay with adjustable interval lengths, elastic step decay does not require manual adjustment for the length of each interval.

Original: Instead, they are all controlled by one hyperparameter r ∈ (0 , 1) , which decides the “shrinking speed” of interval lengths.
Revised: Instead, they are all controlled by one hyperparameter r ∈ (0 , 1) , which decides the “shrinking speed” of interval lengths.

Original: Experiments on
Revised: Experiments on

Original: CIFAR-10, CIFAR-100 and ImageNet demonstrate its superiority in practice, as shown in Table 6,
Revised: CIFAR-10, CIFAR-100 and ImageNet demonstrate its superiority in practice, as shown in Table 6,

Original: Table 7 and Figurer 8, 9, 10, 11.
Revised: Table 7.

Original: For experiments on CIFAR-10/CIFAR-100, we adopt the same settings as eigencurve .
Revised: For experiments on CIFAR-10/CIFAR-100, we adopt the same settings as eigencurve , except we only run one trial of experiment and use common step decay with three intervals + decay factor 10.

Original: For experiments on ImageNet, we use ResNet-50 trained via SGD without momentum, batch sizeand weight decay wd = 10 − 4 .
Revised: For experiments on ImageNet, we use ResNet-50 trained via SGD without momentum, batch size 256 and weight decay wd = 10 − 4 .

Original: Since no momentum is used, the initial learning rate is set to η 0 = 1 . 

Revised: Since no momentum is used, the initial learning rate is set to η 0 = 1 .

Original: instead of η 0 = 0 .
Revised: instead of η 0 = 0 .

Original: The hyperparameters η min is set to be 0 for cosine decay scheduler.
Revised: The hyperparameters η min is set to be 0 for cosine decay scheduler.

Original: As for the dataset, we use the common ILSVRC 2012 dataset, which contains 1000 classes, around 1.2M images for training and 50,000 images for validation.
Revised: As for the dataset, we use the common ILSVRC 2012 dataset, which containsclasses, around 1.2M images for training and 50,000 images for validation.

Original: For all experiments, we search r ∈ { 1 / 2 , 1 / √ 2 } for ESD, with other hyperparameter search and selection process being the same as eigencurve . 
Revised: For all experiments, we search r ∈ { 1 / 2 , 1 / √ 2 } for ESD, with other hyperparameter search and selection process being the same as eigencurve . [Table] [Table]

Revised: L ANGUAGE M ODELING WITH E LASTIC

Original: [Table]
Revised: [Table]

Revised: More experiments on language modeling are conducted to further demonstrate Elastic Step Decay’s superiority over other schedulers.

Revised: For all experiments

Original: [Table]
Revised: [Table]

Original: A. I MAGE C LASSIFICATION ON I MAGE N ET WITH C OSINE - POWER S CHEDULING
Revised: we follow almost the same settings in Zaremba et al. (2015), where a large regularized LSTM recurrent neural network (Hochreiter & Schmidhuber, 1997) is trained on Penn Treebank (Marcus et al., 1993) for language modeling task. The Penn Treebank dataset has a training set of 929k words, a validation set of 73k words and a test set of 82k words.

Revised: SGD without momentum is adopted for training, with batch size 20 and 35 unrolling steps in LSTM.

Revised: Other details are exactly the same.

Revised: The only difference is that in Zaremba et al.

Revised: [Table]

Revised: it usesepochs to train the large regularized LSTM

Revised: [Table]

Revised: which is changed to 30 epochs in our settings

Revised: [Table]

Revised: since we found that the model starts overﬁtting after 30 epochs.

Revised: We conducted hyperparameter search for all schedules, as shown in Table

Revised: [Table]

Revised: [Table]

Revised: Experimental results show that Elastic Step Decay signiﬁcantly outperforms other schedulers, as shown in Table

Revised: [Table]

Revised: [Table]

Revised: A. I MAGE C LASSIFICATION ON I MAGE N ET WITH C OSINE - POWER S CHEDULING

Original: Another key observation in CIFAR-10 experiments is that eigencurve ’s learning rate curve shape changes in a ﬁxed tendency: more “concave” learning rate curves for less training epochs , which inspire the cosine-power schedule in following form.
Revised: Another key observation in CIFAR-10 experiments is that eigencurve ’s learning rate curve shape changes in a ﬁxed tendency: more “concave” learning rate curves for less training epochs , which inspire the cosine-power schedule in following form.

Original: [Equation]
Revised: [Equation]

Original: Results in Table 8 show the schedulings’ performance with α = 0 .
Revised: Results in Table 10 show the schedulings’ performance with α = 0 .

Original: 5 / 1 / 2 , which are denoted as √
Revised: 5 / 1 / 2 , which are denoted as √

Original: Cosine/Cosine/Cosine 2 respectively.
Revised: Cosine/Cosine/Cosine 2 respectively.

Original: Notice that the best scheduler gradually moves from small α to larger α when the number of epochs increases.
Revised: Notice that the best scheduler gradually moves from small α to larger α when the number of epochs increases.

Original: For #epoch=270, since the number of epochs is large enough to make model converge, it is reasonable that the accuracy gap between all schedulers is small.
Revised: For #epoch=270, since the number of epochs is large enough to make model converge, it is reasonable that the accuracy gap between all schedulers is small.

Original: For experiments on ImageNet, we use ResNet-18 trained via SGD without momentum, batch size 256 and weight decay wd = 10 − 4 .
Revised: For experiments on ImageNet, we use ResNet-18 trained via SGD without momentum, batch size 256 and weight decay wd = 10 − 4 .

Original: Since no momentum is used, the initial learning rate is set to η 0 = 1 . 

Revised: Since no momentum is used, the initial learning rate is set to η 0 = 1 .

Original: instead of η 0 = 0 .
Revised: instead of η 0 = 0 .

Original: The hyperparameters η min is set to be 0 for all cosine-power scheduler.
Revised: The hyperparameters η min is set to be 0 for all cosine-power scheduler.

Original: As for the dataset, we use the common ILSVRC 2012 dataset, which contains 1000 classes, around 1.2M images for training and 50,000 images for validation.
Revised: As for the dataset, we use the common ILSVRC 2012 dataset, which contains 1000 classes, around 1.2M images for training and 50,000 images for validation.

Original: [Table]
Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Revised: Please refer to Figure 8, 9, 10, 11, 12 and 13.

Original: B D ETAILED E XPERIMENTAL S ETTINGS FOR I MAGE C LASSIFICATION ON CIFAR-10/CIFARB. B ASIC S ETTINGS
Revised: B D ETAILED E XPERIMENTAL S ETTINGS FOR I MAGE C LASSIFICATION ON CIFAR-10/CIFARB. B ASIC S ETTINGS

Original: As mentioned in the main paper, all models are trained with stochastic gradient descent (SGD), no momentum, batch size 128 and weight decay wd = 0 . 0005 .
Revised: As mentioned in the main paper all models are trained with stochastic gradient descent (SGD), no momentum, batch size 128 and weight decay wd = 0 . 0005 .

Original: Furthermore, we perform a grid search to choose the best hyperparameters of all schedulers, with a validation set created from 5 ,samples in the training set, i.e. one-tenth of the training set.
Revised: Furthermore, we perform a grid search to choose the best hyperparameters of all schedulers, with a validation set created from 5 , samples in the training set, i.e. one-tenth of the training set.

Original: The remaining 45 , 000 samples are then used for training the model.
Revised: The remaining 45 , 000 samples are then used for training the model.

Original: After obtaining hyperparameters with the best validation accuracy, we train the model again with the full training set and test the trained model on the test set.
Revised: After obtaining hyperparameters with the best validation accuracy, we train the model again with the full training set and test the trained model on the test set.

Original: Here the grid search explores hyperparameters η 0 ∈ { 1 . 0 , 0 . 6 , 0 . 3 , 0 . 2 , 0 . 1 } and η min ∈ { 0 . 01 , 0 .
Revised: Here the grid search explores hyperparameters η 0 ∈ { 1 . 0 , 0 . 6 , 0 . 3 , 0 . 2 , 0 . 1 } and η min ∈ { 0 . 01 , 0 .

Original: 001 , 0 . 0001 , 0 , “UNRESTRICTED” } , where η 0 denotes the initial learning rate and η min stands for the learning rate of last iteration.
Revised: 001 , 0 . 0001 , 0 , “UNRESTRICTED” } , where η 0 denotes the initial learning rate and η min stands for the learning rate of last iteration.

Original: “UNRESTRICTED” denotes the case where η min is not set, which is useful for eigencurve , who can decide the learning rate curve without setting η min .
Revised: “UNRESTRICTED” denotes the case where η min is not set, which is useful for eigencurve , who can decide the learning rate curve without setting η min .

Original: Given η 0 and η min , we adjust all schedulers as follows.
Revised: Given η 0 and η min , we adjust all schedulers as follows.

Original: For inverse time decay, the hyperparameter γ is computed accordingly based on η 0 and η min .
Revised: For inverse time decay, the hyperparameter γ is computed accordingly based on η 0 and η min .

Original: For cosine decay, η 0 and η min is directly used, with no restart adopted.
Revised: For cosine decay, η 0 and η min is directly used, with no restart adopted.

Original: For step decay, common settings of CIFAR-10/CIFAR-100 is adopted, which decays learning rate by a factor of 10 at T/ 3  and  2 T/ 3 .
Revised: For step decay*, we search the interval number in { 3 , 4 , 5 , (cid:98) log T (cid:99) , (cid:98) log T (cid:99) + 1 } and decay factor in { 2 , 5 , 10 } .

Original: We have also tried the form in Ge et al.
Revised: For step decay proposed in Ge et al.

Original: , but its performance is worse than the common setting of step decay, hence the result is not presented.
Revised: (2019), the interval number is ﬁxed to be (cid:98) log T (cid:99) , along with a decay factor 2 .

Original: For eigencurve , two major modiﬁcations are made to make it more suitable for practical settings:
Revised: For eigencurve , two major modiﬁcations are made to make it more suitable for practical settings:

Original: [Equation]
Revised: [Equation]

Original: Here we change 1 /L to η 0 so that it is possible to adjust the initial learning rate of eigencurve .
Revised: Here we change 1 /L to η 0 so that it is possible to adjust the initial learning rate of eigencurve .

Original: We also change the ﬁxed constant 2 to a general constant β > 1 , which is aimed at making the learning rate curve smoother.
Revised: We also change the ﬁxed constant 2 to a general constant β > 1 , which is aimed at making the learning rate curve smoother.

Original: The learning rate curve of eigencurve is then linearly scaled to match the given η min .
Revised: The learning rate curve of eigencurve is then linearly scaled to match the given η min .

Original: Notice that the learning rate η 0 can be larger than 1 /L , while the loss still does not explode.
Revised: Notice that the learning rate η 0 can be larger than 1 /L , while the loss still does not explode.

Original: There are several explanations for this phenomenon.
Revised: There are several explanations for this phenomenon.

Original: First, in basic non-smooth analysis of GD and SGD with inverse time decay scheduler, the learning rate can be larger than 1 /L
Revised: First, in basic non-smooth analysis of GD and SGD with inverse time decay scheduler, the learning rate can be larger than 1 /L

Original: if the gradient norm is bounded (Shamir & Zhang, 2012).
Revised: if the gradient norm is bounded (Shamir & Zhang, 2012).

Original: Second, deep learning has a non-convex loss landscape, especially when the parameter is far away from the optima.
Revised: Second, deep learning has a non-convex loss landscape, especially when the parameter is far away from the optima.

Original: Hence it is common to use larger learning rate at ﬁrst.
Revised: Hence it is common to use larger learning rate at ﬁrst.

Original: As long as the loss does not explode, it is okay.
Revised: As long as the loss does not explode, it is okay.

Original: So we still include large learning rate η 0 in our grid search process.
Revised: So we still include large learning rate η 0 in our grid search process.

Original: B. S ETTINGS FOR E IGENCURVE
Revised: B. S ETTINGS FOR E IGENCURVE

Original: In addition, for our eigencurve scheduler, we use PyHessian (Yao et al., 2020) to generate Hessian matrix’s eigenvalue distribution for all models.
Revised: In addition, for our eigencurve scheduler, we use PyHessian (Yao et al., 2020) to generate Hessian matrix’s eigenvalue distribution for all models.

Original: The whole process consists of three phases, which are illustrated as follows.
Revised: The whole process consists of three phases, which are illustrated as follows

Original: 1) Training the model Almost all CNN models on CIFAR-10 have non-convex objectives, thus the Hessian’s eigenvalue distributions are different for different parameters.
Revised: 1) Training the model Almost all CNN models on CIFAR-10 have non-convex objectives, thus the Hessian’s eigenvalue distributions are different for different parameters.

Original: Normally, we want the this distribution to reﬂect the overall tendency of most parts of the training process.
Revised: Normally, we want the this distribution to reﬂect the overall tendency of most parts of the training process.

Original: According to the phenomenon demonstrated in Appendix E, ﬁgure A.11-A.17 of Yao et al.
Revised: According to the phenomenon demonstrated in Appendix E, ﬁgure A.11-A.17 of Yao et al.

Original: (2020), the eigenvalue distribution of ResNet’s Hessian presents similar tendency after training 30 epochs, which suggests that the Hessian’s eigenvalue distribution can be used after sufﬁcient training.
Revised: (2020), the eigenvalue distribution of ResNet’s Hessian presents similar tendency after training 30 epochs, which suggests that the Hessian’s eigenvalue distribution can be used after sufﬁcient training.

Original: In all CIFAR-10 experiments , we use the Hessian’s eigenvalue distribution of models after training 180 epochs.
Revised: In all CIFAR-10 experiments, we use the Hessian’s eigenvalue distribution of models after training 180 epochs.

Original: Since the goal here is to sufﬁciently train the model, not to obtain good performance, common baseline settings are adopted for training.
Revised: Since the goal here is to sufﬁciently train the model, not to obtain good performance, common baseline settings are adopted for training.

Original: For all models used for eigenvalue distribution estimation, we adopt SGD with momentum = 0 .
Revised: For all models used for eigenvalue distribution estimation, we adopt SGD with momentum = 0 .

Original: 9 , batch size 128 , weight decay wd =0 .
Revised: 9 , batch size 128 , weight decay wd =

Original: 0005 and initial learning rate 0 .
Revised: 0005 and initial learning rate 0 .

Original: On top of that, we use step decay, which decays the learning rate by a factor of 10 at epoch 80 and 120 .
Revised: On top of that, we use step decay, which decays the learning rate by a factor of 10 at epoch 80 and 120 .

Original: All of them are default settings of the PyHessian code ( https://github.com/amirgholami/PyHessian/blob/master/training.
Revised: All of them are default settings of the PyHessian code ( https://github.com/amirgholami/PyHessian/blob/master/training.

Original: py , commit: f4c3f77).
Revised: py , commit: f4c3f77).

Original: ImageNet adopts a similar setting, with training epochs being 90, SGD with momentum
Revised: ImageNet adopts a similar setting, with training epochs being 90, SGD with momentum

Original: = 0 .
Revised: = 0 .

Original: 9 , batch size 256 , weight decay wd = 0 .
Revised: 9 , batch size 256 , weight decay wd = 0 .

Original: 0001 , inital learning rate 0 .
Revised: 0001 , inital learning rate 0 .

Original: 1 and step decay schedule decays learning rate by a factor of 10 at epoch 30 and 60 .
Revised: 1 and step decay schedule decays learning rate by a factor of 10 at epoch 30 and 60 .

Original: 2) Estimating Hessian matrix’s eigenvalue distribution for the trained model After obtaining the checkpoint of a sufﬁciently trained model, we then run PyHessian to estimate the Hessian’s eigenvalue distribution for that checkpoint.
Revised: 2) Estimating Hessian matrix’s eigenvalue distribution for the trained model After obtaining the checkpoint of a sufﬁciently trained model, we then run PyHessian to estimate the Hessian’s eigenvalue distribution for that checkpoint.

Original: The goal here is to obtain the Hessian’s eigenvalue distribution with sufﬁcient precision.
Revised: The goal here is to obtain the Hessian’s eigenvalue distribution with sufﬁcient precision.

Original: To be more speciﬁc, the length of intervals around each estimated eigenvalue.
Revised: To be more speciﬁc, the length of intervals around each estimated eigenvalue.

Original: PyHessian estimates the eigenvalue spectral density (ESD) of a model’s Hessian, in other words, the output is a list of eigenvalue intervals, along with the density of each interval, where the whole density adds up to 1 .
Revised: PyHessian estimates the eigenvalue spectral density (ESD) of a model’s Hessian, in other words, the output is a list of eigenvalue intervals, along with the density of each interval, where the whole density adds up to 1 .

Original: Precision means the interval length here.
Revised: Precision means the interval length here.

Original: It is natural that the estimation precision is related to the complexity of the PyHessian algorithm, e.g. the better precision it yields, the more time and space it consumes.
Revised: It is natural that the estimation precision is related to the complexity of the PyHessian algorithm, e.g. the better precision it yields, the more time and space it consumes.

Original: More speciﬁcally, the algorithm has a time complexity of O
Revised: More speciﬁcally, the algorithm has a time complexity of O

Original: ( Nn 2 v d ) and space complexity O
Revised: ( Nn 2 v d ) and space complexity O

Original: ( Bd + n v d ) , where d is the number of model parameters, N is the number of samples used for estimating the ESD, B is the batch size and n v is the iteration number of Stochastic Lanczos Quadrature used in PyHessian, which controls the estimation precision (see Algorithm 1 of Yao et al.
Revised: ( Bd + n v d ) , where d is the number of model parameters, N is the number of samples used for estimating the ESD, B is the batch size and n v is the iteration number of Stochastic Lanczos Quadrature used in PyHessian, which controls the estimation precision (see Algorithm 1 of Yao et al.

Original: In our experiments, we use n v = 5000 for ResNet-18 and n v = 3000 for GoogLeNet/VGG16, which gives an eigenvalue distribution estimation with precision around 10 − 5 to 10 − 4 .
Revised: In our experiments, we use n v = 5000 for ResNet-18 and n v = 3000 for GoogLeNet/VGG16, which gives an eigenvalue distribution estimation with precision around 10 − 5 to 10 − 4 .

Original: N and B are both set to 200 due to GPU memory constraint, i.e. we use one mini-batch to estimate the eigenvalue distribution.
Revised: N and B are both set to 200 due to GPU memory constraint, i.e. we use one mini-batch to estimate the eigenvalue distribution.

Original: It turns out that this one-batch estimation is good enough and yields similar results to full dataset settings shown in Yao et al.
Revised: It turns out that this one-batch estimation is good enough and yields similar results to full dataset settings shown in Yao et al.

Original: However, space complexity is still a bottleneck here.
Revised: However, space complexity is still a bottleneck here.

Original: Due to the large number of n v and space complexity O ( Bd + n v d ) of PyHessian, the value of d cannot be very large.
Revised: Due to the large number of n v and space complexity O ( Bd + n v d ) of PyHessian, the value of d cannot be very large.

Original: In practice, with a
Revised: In practice, with a

Original: NVIDIA GeForce 

Revised: NVIDIA GeForce

Original: Ti GPU, which has around 11 GB memory, the maximum acceptable parameter number d is around 200 K − 400 K .
Revised: Ti GPU, which has around 11 GB memory, the maximum acceptable parameter number d is around 200 K − 400 K .

Original: This implies that the model has to be compressed.
Revised: This implies that the model has to be compressed.

Original: In our experiments, we reduce the number of channels by a factor of C for all models.
Revised: In our experiments, we reduce the number of channels by a factor of C for all models.

Original: For ResNet-18,
Revised: For ResNet-18,

Original: C = 16 .
Revised: C = 16 .

Original: For GoogLeNet, C = 4 .
Revised: For GoogLeNet, C = 4 .

Original: For VGG16, C = 8 .
Revised: For VGG16, C = 8 .

Original: Notice that those compressed models are only used for eigenvalue distribution estimation.
Revised: Notice that those compressed models are only used for eigenvalue distribution estimation.

Original: In experiments of comparing different scheduling, we still use the original model with no compression.
Revised: In experiments of comparing different scheduling, we still use the original model with no compression.

Original: One may refer to https://github.com/opensource12345678/why_cosine_ works/tree/main/eigenvalue_distribution for generated eigenvalue distributions.
Revised: One may refer to https://github.com/opensource12345678/why_cosine_ works/tree/main/eigenvalue_distribution for generated eigenvalue distributions.

Original: 3) Generating eigencurve scheduler with the estimated eigenvalue distribution After obtaining the eigenvalue distribution, we do a preprocessing before plug it into our eigencurve scheduler.
Revised: Generating eigencurve scheduler with the estimated eigenvalue distribution After obtaining the eigenvalue distribution, we do a preprocessing before plug it into our eigencurve scheduler.

Original: First, we notice that there are negative eigenvalues in the ﬁnal distribution.
Revised: First, we notice that there are negative eigenvalues in the ﬁnal distribution.

Original: Theoretically, if the parameter is right at the optimal point, no negative eigenvalues should exist for Hessian matrix.
Revised: Theoretically, if the parameter is right at the optimal point, no negative eigenvalues should exist for Hessian matrix.

Original: Thus we conjecture that those negative eigenvalues are caused by the fact that the model is closed to optima w ∗ , but not exactly at that point.
Revised: Thus we conjecture that those negative eigenvalues are caused by the fact that the model is closed to optima w ∗ , but not exactly at that point.

Original: Furthermore, the estimation precision loss can be another cause.
Revised: Furthermore, the estimation precision loss can be another cause.

Original: In fact, most of those negative eigenvalues are small, e.g. 98 .
Revised: In fact, most of those negative eigenvalues are small, e.g. 98 .

Original: 6% of those negative eigenvalues lie in [ − 0 .
Revised: 6% of those negative eigenvalues lie in [ − 0 .

Original: 1 , 0) , and can be generally ignored without much loss.
Revised: 1 , 0) , and can be generally ignored without much loss.

Original: In our case, we set them to their absolute values.
Revised: In our case, we set them to their absolute values.

Original: Second, for a given weight decay value wd , we need to take the implicit L2 regularization into account, since it affects the Hessian matrix as well.
Revised: Second, for a given weight decay value wd , we need to take the implicit L2 regularization into account, since it affects the Hessian matrix as well.

Original: Therefore, for all eigenvalues after the ﬁrst step, we add wd to them.
Revised: Therefore, for all eigenvalues after the ﬁrst step, we add wd to them.

Original: After preprocessing, we plug the eigenvalue distribution into our eigencurve scheduler and generates the exact form of eigencurve .
Revised: After preprocessing, we plug the eigenvalue distribution into our eigencurve scheduler and generates the exact form of eigencurve .

Original: [Equation]
Revised: [Equation]

Original: For experiments with 100 epochs, we set β = 1 . 000005 , so that the learning rate curve is much smoother.
Revised: For experiments with 100 epochs, we set β = 1 . 000005 , so that the learning rate curve is much smoother.

Original: For experiments with 10 epochs, we set β = 2 . 0 .
Revised: For experiments with 10 epochs, we set β = 2 . 0 .

Original: In our experiments, β serves as a ﬁxed constant, not hyperparameters.
Revised: In our experiments, β serves as a ﬁxed constant, not hyperparameters.

Original: So no hyperparameter search is conducted on β .
Revised: So no hyperparameter search is conducted on β .

Original: One can do that in practice though, if computation resource allows.
Revised: One can do that in practice though, if computation resource allows.

Original: B. C OMPUTE R ESOURCE AND I MPLEMENTATION D ETAILS
Revised: B. C OMPUTE R ESOURCE AND I MPLEMENTATION D ETAILS

Original: All the code for results in main paper can be found in https://github.com/ opensource12345678/why_cosine_works/tree/main , which is released under the
Revised: All the code for results in main paper can be found in https://github.com/ opensource12345678/why_cosine_works/tree/main , which is released under the

Original: MIT license.
Revised: MIT license.

Original: All experiments on CIFAR-10/CIFAR-100 are conducted on a single NVIDIA GeForce
Revised: All experiments on CIFAR-10/CIFAR-100 are conducted on a single NVIDIA GeForce

Original: Ti GPU, where ResNet-18/GoogLeNet/VGG16 takes around 20mins/90mins/40mins to trainepochs, respectively.
Revised: Ti GPU, where ResNet-18/GoogLeNet/VGG16 takes around 20mins/90mins/40mins to trainepochs, respectively.

Original: High-precision eigenvalue distribution estimation, e.g. n v ≥ 3000 , requires around 1-2 days to complete, but this is no longer necessary given the released results.
Revised: High-precision eigenvalue distribution estimation, e.g. n v ≥ 3000 , requires around 1-2 days to complete, but this is no longer necessary given the released results.

Original: The ResNet-18 model is implemented in Tensorﬂow 2.0.
Revised: The ResNet-18 model is implemented in Tensorﬂow 2.0.

Original: We use tensorﬂow-gpu 2.3.1 in our code.
Revised: We use tensorﬂow-gpu 2.3.1 in our code.

Original: The GoogLeNet and VGG16 model is implemented in Pytorch, speciﬁcally, 1.7.0+cu101.
Revised: The GoogLeNet and VGG16 model is implemented in Pytorch, speciﬁcally, 1.7.0+cu101.

Original: B. L ICENSE OFCIFAR-10 AND P Y H ESSIAN
Revised: B. L ICENSE OF P Y H ESSIAN

Original: According to https://github.com/wichtounet/cifar-10/blob/master/
Revised: According to https://github.com/amirgholami/PyHessian/blob/master/

Original: LICENSE and https://github.com/wichtounet/cifar-10/blob/master/

Original: LICENSE ,both CIFAR-10 and PyHessian (Yao et al., 2020) are released under the MIT License.
Revised: LICENSE , PyHessian (Yao et al., 2020) is released under the MIT License.

Original: C D ETAILED E XPERIMENTAL S ETTINGS FOR I MAGE C LASSIFICATION ON I MAGE N ET
Revised: C D ETAILED E XPERIMENTAL S ETTINGS FOR I MAGE C LASSIFICATION ON I MAGE N ET

Original: According to https://github.com/floydhub/imagenet/blob/master/LICENSE ,
Revised: One may refer to https://www.image-net.org/download for speciﬁc terms of access for

Original: ImageNet is released under the BSD 3-Clause ”New” or ”Revised” License.
Revised: ImageNet.

Original: The dataset can be downloaded from https://image-net.org/challenges/LSVRC/2012/ 2012-downloads.php , with training set being “Training images (Task 1 & 2)” and validation set being “Validation images (all tasks)”.
Revised: The dataset can be downloaded from https://image-net.org/challenges/ LSVRC/2012/2012-downloads.php , with training set being “Training images (Task 1 & 2)” and validation set being “Validation images (all tasks)”.

Original: Notice that registration and veriﬁcation of institute is required for successful download.
Revised: Notice that registration and veriﬁcation of institute is required for successful download.

Original: All experiments on ImageNet are conducted on two NVIDIA GeForce 2080 Ti GPUs with data parallelism.
Revised: All experiments on ImageNet are conducted on two NVIDIA GeForce 2080 Ti GPUs with data parallelism.

Original: The ResNet-18 model takes around 2 days to train 90 epochs, about 20mins-30mins per epoch.
Revised: The ResNet-18 model takes around 2 days to train 90 epochs, about 20mins-30mins per epoch.

Original: The ResNet-18 model on ImageNet is implemented in Pytorch, speciﬁcally, 1.7.0+cu101.
Revised: The ResNet-18 model on ImageNet is implemented in Pytorch, speciﬁcally, 1.7.0+cu101.

Original: D I MPORTANT P ROPOSITIONS AND L EMMAS
Revised: D I MPORTANT P ROPOSITIONS AND L EMMAS

Original: Proposition 3.
Revised: Proposition 3.

Original: Letting f ( x ) be a monotonically increasing function in the range [ t 0 , ˜ t ] , then it holds that
Revised: Letting f ( x ) be a monotonically increasing function in the range [ t 0 , ˜ t ] , then it holds that

Original: [Equation]
Revised: [Equation]

Original: If f ( x ) is monotonically decreasing in the range [ t 0 , ˜ t ] , then it holds that
Revised: If f ( x ) is monotonically decreasing in the range [ t 0 , ˜ t ] , then it holds that

Original: [Equation]
Revised: [Equation]

Original: Lemma 2.
Revised: Lemma 2.

Original: Function f ( x ) = exp( − αx ) x 2 with 0 <
Revised: Function f ( x ) = exp( − αx ) x 2 with 0 <

Original: α and x ∈ (0 , 1] is monotone decreasing in the range x ∈ ( 2 α , + ∞ ) and monotone increasing in the range x ∈ [0 , 2 α ] .
Revised: α and x ∈ (0 , 1] is monotone decreasing in the range x ∈ ( 2 α , + ∞ ) and monotone increasing in the range x ∈ [0 , 2 α ] .

Original: Proof.
Revised: Proof.

Original: We can obtain the derivative of f ( x ) as
Revised: We can obtain the derivative of f ( x ) as

Original: [Equation]
Revised: [Equation]

Original: Thus, it holds that ∇ f ( x ) ≥ 0 when x ∈ [0 , 2 α ] .
Revised: Thus, it holds that ∇ f ( x ) ≥ 0 when x ∈ [0 , 2 α ] .

Original: This implies that f ( x ) is monotone increasing when x ∈ [0 , 2 α ] .
Revised: This implies that f ( x ) is monotone increasing when x ∈ [0 , 2 α ] .

Original: Similarly, we can obtain that f ( x ) is monotone decreasing when x ∈ ( 2 α , + ∞ ) .
Revised: Similarly, we can obtain that f ( x ) is monotone decreasing when x ∈ ( 2 α , + ∞ ) .

Original: Lemma 3.
Revised: Lemma 3.

Original: It holds that 
Revised: It holds that (cid:90)

Original: [Equation]
Revised: [Equation]

Revised: (D.

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: E P ROOF OF S ECTION
Revised: E P ROOF OF S ECTION

Original: Proof of Proposition 1.
Revised: Proof of Proposition 1.

Original: By iteratively applying Eqn.
Revised: By iteratively applying Eqn.

Original: [Equation] we can obtain that
Revised: (2.1), we can obtain that

Original: [Equation]
Revised: [Equation]

Original: Summing up each coordinate, we can obtain the result.
Revised: Summing up each coordinate, we can obtain the result.

Original: Proof of Proposition 2.
Revised: Proof of Proposition 2.

Original: Let us denote σ 2 j = λ 2 j σ 2 .
Revised: Let us denote σ 2 j = λ 2 j σ 2 .

Original: By Eqn.
Revised: By Eqn.

Original: (2.1), we can obtain that
Revised: [Equation] we can obtain that

Original: E (cid:104)
Revised: [Equation] (cid:32) [Equation] i

Original: λ
Revised: [Equation] i [Equation] (cid:88)

Original: [Equation]
Revised: [Equation]

Original: ( w t +1 ,j − w ∗ ,j ) 2 (cid:105) ≤
Revised: (cid:32)

Original: Π ti =1 (1 − η i,j λ j ) 2 · λ j ( w 1 ,j − w ∗ ,j ) 2 + t (cid:88)k
Revised: [Equation] (cid:88)

Original: = Π i =
Revised: [Equation]

Original: k
Revised: λ

Original: (1 − η i,j λ j ) η 2 k,j σ 2 j ≤ exp (cid:32) −t (cid:88) i = η i,j λ j (cid:33) · λ j ( w 1 ,j − w ∗ ,j ) 2 + t (cid:88) k = exp (cid:32) −t (cid:88) i =
Revised: [Equation]

Original: k η i,j λ j (cid:33) η 2 k,j σ 2 j = exp (cid:32) − 2 λ j t (cid:88) i =
Revised: (cid:33) η 2 k,j σ 2 j = exp (cid:32) − 2 λ j t (cid:88) i =

Original: L + µi (cid:33) · λ j ( w 1 ,j − w ∗ ,j ) 2 + t (cid:88) k
Revised: L + µi (cid:33) · λ j ( w 1 ,j − w ∗ ,j ) 2 + t (cid:88) k

Original: = exp (cid:32) − 2 λ j t (cid:88) i =
Revised: = exp (cid:32) − 2 λ j t (cid:88) i =

Original: k
Revised: k

Original: L + µi (cid:33) η 2 k,j σ 2 j ≤ exp (cid:18) 2 λ j µ ln (cid:18) L + µ
Revised: L + µi (cid:33) η 2 k,j σ 2 j ≤ exp (cid:18) 2 λ j µ ln (cid:18) L + µ

Original: L + µt (cid:19)(cid:19) · λ j ( w 1 ,j − w ∗ ,j ) 2 + t (cid:88) k
Revised: L + µt (cid:19)(cid:19) · λ j ( w 1 ,j − w ∗ ,j ) 2 + t (cid:88) k

Original: = exp (cid:18)
Revised: = exp (cid:18)

Original: 2 λ j µ ln (cid:18) L + µk
Revised: 2 λ j µ ln (cid:18) L + µk

Original: L + µt (cid:19)(cid:19) · σ 2 j ( L + µk )=
Revised: L + µt (cid:19)(cid:19) · σ 2 j ( L + µk )=

Original: (cid:18) L + µ
Revised: (cid:18) L + µ

Original: L + µt (cid:19) 2 λjµ · λ j ( w 1 ,j − w 
Revised: L + µt (cid:19) 2 λjµ · λ j ( w 1 ,j − w ∗ ,j ) 2 + t (cid:88) k = ( L + µk ) (cid:16) 2 λ jµ − 2 (cid:17) ( L + µt ) 2 λjµ · σ 2 j ≤ (cid:18) L + µ

Original: [Equation]
Revised: L + µt (cid:19) 2 λjµ · λ j ( w 1 ,j − w ∗ ,j ) 2 + σ 2 j 2 λ j − µ ( L + µt ) 2 λjµ −( L + µt ) 2 λjµ + σ 2 j ( L + µt )= (cid:18) L + µ L + µt (cid:19) 2 λjµ · λ j ( w 1 ,j − w ∗ ,j ) 2 + 2 λ j − µ · L + µt · σ 2 j + σ 2 j ( L + µt ) 2 .

Original: The third inequality is because function F ( x ) = 1 / ( L + µx ) is monotone decreasing in the range [1 , ∞ ) , and it holds that
Revised: The third inequality is because function F ( x ) = 1 / ( L + µx ) is monotone decreasing in the range [1 , ∞ ) , and it holds that

Original: [Equation]
Revised: [Equation]

Original: The last inequality is because function F ( x ) =
Revised: The last inequality is because function F ( x ) =

Original: ( L + µx ) 2 λ j /µ − 2 is monotone increasing in the range [0 , ∞ ) , and it holds that
Revised: ( L + µx ) 2 λ j /µ − 2 is monotone increasing in the range [0 , ∞ ) , and it holds that

Original: [Equation]
Revised: [Equation]

Original: By µ ≤ λ j , σ 2 j = λ 2 j σ 2 , and summing up from i = 1 to d , we can obtain the result.
Revised: By µ ≤ λ j , σ 2 j = λ 2 j σ 2 , and summing up from i = 1 to d , we can obtain the result.

Original: F P RELIMINARIES
Revised: F P RELIMINARIES

Original: Lemma 4.
Revised: Lemma 4.

Original: Let objective function f ( x ) be quadratic.
Revised: Let objective function f ( x ) be quadratic.

Original: Running SGD for T -steps starting from wand a learning rate sequence { η t } Tt =1 , the ﬁnal iterate w T +1 satisﬁes
Revised: Running SGD for T -steps starting from wand a learning rate sequence { η t } Tt =1 , the ﬁnal iterate w T +1 satisﬁes

Original: [Equation]
Revised: [Equation]

Original: where P t =
Revised: where P t =

Original: I − η t H .
Revised: I − η t H .

Original: Proof.
Revised: Proof.

Original: Reformulating Eqn.
Revised: Reformulating Eqn.

Original: (1.5), we have
Revised: (1.5), we have

Original: [Equation]
Revised: [Equation]

Original: Thus [Equation] we can obtain that
Revised: Thus, we can obtain that

Original: [Equation]
Revised: [Equation]

Original: We can decompose above stochastic process associated with SGD’s update into two simpler processes as follows:
Revised: We can decompose above stochastic process associated with SGD’s update into two simpler processes as follows:

Original: [Equation]
Revised: [Equation]

Original: and w [Equation] w 
Revised: and w vt +1 − w ∗

Original: [Equation]

Original: =
Revised: =

Original: [Equation] w [Equation] n t , with w v 0 = w ∗ , (F.3) which entails that
Revised: P t ( w vt − w ∗ ) + η t n t , with w v 0 = w ∗ , (F.3) which entails that

Original: [Equation]
Revised: [Equation]

Original: where the last equality in Eqn.
Revised: where the last equality in Eqn.

Original: (F.4) and Eqn.
Revised: (F.4) and Eqn.

Original: (F.5) is because the commutative property P t P t [Equation]
Revised: (F.5) is because the commutative property P t P t (cid:48)

Original: =
Revised: =

Original: [Equation]
Revised: ( I − η t H )

Original: P
Revised: ( I − η t (cid:48) H ) = ( I − η t (cid:48) H )

Original: [Equation]
Revised: ( I − η t H ) =

Original:  holds for ∀ t, t (cid:48) [Equation]
Revised: P t (cid:48) P t holds for ∀ t, t (cid:48) .

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Revised: [Equation]

Revised:  

Original: [Equation]
Revised: [Equation]

Original: where the last equality is because when τ and τ (cid:48) are different, it holds that
Revised: where the last equality is because when τ and τ (cid:48) are different, it holds that

Original: [Equation]
Revised: [Equation]

Original: due to independence between n τ and n τ (cid:48) .
Revised: due to independence between n τ and n τ (cid:48) .

Original: Lemma 5.
Revised: Lemma 5.

Original: Given the assumption that E ξ (cid:2) n t n (cid:62) t (cid:3) (cid:22) σ 2 H , then the variance term satisﬁes thatT
Revised: Given the assumption that E ξ (cid:2) n t n (cid:62) t (cid:3) (cid:22) σ 2 H , then the variance term satisﬁes that

Original: [Equation]
Revised: [Equation]

Original: where P t =
Revised: where P t =

Original: I − η t H .
Revised: I − η t H .

Original: Proof.
Revised: Proof.

Original: Denote A τ (cid:44) P T . . .
Revised: Denote A τ (cid:44) P T . . .

Original: P τ +1 H 1 2 , then
Revised: P τ +1 H 12 , then

Original: [Equation]
Revised: [Equation]

Original: where the second equality is entailed by the fact that H 1 2 , P τ +1 , . .
Revised: where the second equality is entailed by the fact that H 1 2 , P τ +1 , . .

Original: , P T are symmetric matrices.
Revised: , P T are symmetric matrices.

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we have,
Revised: we have,

Original: [Equation]
Revised: [Equation]

Original: where the third and sixth equality come from the cyclic property of trace, while the ﬁrst inequality is because of the condition E ξ (cid:2) n t n (cid:62) t (cid:3) (cid:22) σ 2 H , where ∀ x,
Revised: where the third and sixth equality come from the cyclic property of trace, while the ﬁrst inequality is because of the condition E ξ (cid:2) n t n (cid:62) t (cid:3) (cid:22) σ 2 H , where ∀ x,

Original: [Equation]
Revised: [Equation]

Original: ⇒
Revised: ⇒

Original: [Equation]
Revised: [Equation]

Original: Lemma 6.
Revised: Lemma 6.

Original: Letting λ
Revised: Letting λ

Original: ˜
Revised: ˜

Original: j be the smallest positive eigenvalue of H , then the bias term satisﬁes that
Revised: j be the smallest positive eigenvalue of H , then the bias term satisﬁes that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Letting H =
Revised: Letting H =

Original: U Λ U (cid:62) be the spectral decomposition of H and u
Revised: U Λ U (cid:62) be the spectral decomposition of H and u

Original: j be j -th column of U , we can obtain that
Revised: j be j -th column of U , we can obtain that

Original: [Equation]
Revised: [Equation]

Original: Since λ ˜ j is the smallest positive eigenvalue of H
Revised: Since λ ˜ j is the smallest positive eigenvalue of H

Original: [Equation]
Revised: [Equation]

Original: it holds that
Revised: it holds that

Original: [Equation]
Revised: [Equation]

Original: G P ROOF OF T HEOREMS
Revised: G P ROOF OF T HEOREMS

Original: Lemma 7.
Revised: Lemma 7.

Original: Let learning rate η t is deﬁned in Eqn.
Revised: Let learning rate η t is deﬁned in Eqn.

Original: Assuming k ∈ [ t i (cid:48) − 1 , t i (cid:48) ] with 1 ≤ i (cid:48) ≤ ˜ i ≤
Revised: Assuming k ∈ [ t i (cid:48) − 1 , t i (cid:48) ] with 1 ≤ i (cid:48) ≤ ˜ i ≤

Original: T , the sequence { η t } Tt =0 satisﬁes that
Revised: T , the sequence { η t } Tt =0 satisﬁes that

Original: [Equation]
Revised: [Equation]

Original: where α i is deﬁned as
Revised: where α i is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First, we divide learning rates into two groups: those who are guaranteed to cover a full interval and those who may not.
Revised: First, we divide learning rates into two groups: those who are guaranteed to cover a full interval and those who may not.

Original: [Equation]
Revised: [Equation]

Original: Furthermore, because η t is monotonically decreasing with respect to t , by Proposition 3, we have
Revised: Furthermore, because η t is monotonically decreasing with respect to t , by Proposition 3, we have

Original: [Equation]
Revised: [Equation]

Original: Lemma 8.
Revised: Lemma 8.

Original: Letting sequence { α i } be deﬁned in Eqn.
Revised: Letting sequence { α i } be deﬁned in Eqn.

Original: (G.2) , given 1 ≤ ˜ i , it holds that
Revised: (G.2) , given 1 ≤ ˜ i , it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Notice that g ( k ) :
Revised: Notice that g ( k ) :

Original: = ( α i − 1 + 

Revised: = ( α i − 1 +

Original: i − 1 µ ( k − t i − 1 ))
Revised: i − 1 µ ( k − t i − 1 ))

Original: 2 ˜ i − i +2 − 2 is a monotonically increasing function, we have,
Revised: 2 ˜ i − i +2 − 2 is a monotonically increasing function, we have,

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: Lemma 9.
Revised: Lemma 9.

Original: Letting { α i } be a positive sequence
Revised: Letting { α i } be a positive sequence

Original: [Equation]
Revised: [Equation]

Original: given 1 ≤ ˜ i
Revised: given 1 ≤ ˜ i

Original: [Equation]
Revised: [Equation]

Original: it holds that
Revised: it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First
Revised: First

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: =  
Revised: =  

Original: ˜ i +1 (cid:89)
Revised: ˜ i +1 (cid:89)

Original: j = i + (cid:18)
Revised: j = i + (cid:18)

Original: α j −α j (cid:19) 2 ˜ i − j +2 
Revised: α j −α j (cid:19) 2 ˜ i − j +2 

Original:    α 2 ˜
Revised:    α 2 ˜

Original: i − i +2 −
Revised: i − i +2 −

Original: i −α 2 ˜ i − i +2 i 
Revised: i −α 2 ˜ i − i +2 i 

Original:  =
Revised:  =

Original: α − 1 ˜ i +

Revised: α − 1 ˜ i +

Original: + ˜ i (cid:88)
Revised: + ˜ i (cid:88)

Original: i =  
Revised: i =  

Original: ˜ i +1 (cid:89)
Revised: ˜ i +1 (cid:89)

Original: j = i + (cid:18)
Revised: j = i + (cid:18)

Original: α j −α j (cid:19) 2 ˜ i − j +2 
Revised: α j −α j (cid:19) 2 ˜ i − j +2 

Original:  α − 1 i − ˜ i +1 (cid:88) i =
Revised:  α − 1 i − ˜ i +1 (cid:88) i =

Original:  
Revised:  

Original: ˜ i +1 (cid:89)
Revised: ˜ i +1 (cid:89)

Original: j = i + (cid:18)
Revised: j = i + (cid:18)

Original: α j −α j (cid:19) 2 ˜ i − j +2 
Revised: α j −α j (cid:19) 2 ˜ i − j +2 

Original:    α 2 ˜
Revised:    α 2 ˜

Original: i − i +2 −
Revised: i − i +2 −

Original: i −α 2 ˜ i − i +2 i 
Revised: i −α 2 ˜ i − i +2 i 

Original:  .
Revised:  .

Original: Furthermore
Revised: Furthermore

Original: [Table]
Revised: [Table]

Original: we reformulate the term (cid:80)
Revised: we reformulate the term (cid:80)

Original: [Table]
Revised: [Table]

Original: ii
Revised: ii

Original: [Table]
Revised: [Table]

Original: as follows
Revised: as follows

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Combining above results
Revised: Combining above results

Original: [Equation]
Revised: [Equation]

Original: we can obtain that
Revised: we can obtain that

Original: [Equation]
Revised: [Equation]

Original: Lemma 10.
Revised: Lemma 10.

Original: Letting us denote v t +1 ,j (cid:44) (cid:80) tk =0 η 2 k (cid:81) ti = k +1 (
Revised: Letting us denote v t +1 ,j (cid:44) (cid:80) tk =0 η 2 k (cid:81) ti = k +1 (

Original: 1 − η i λ j ) 2 with η i deﬁned in Eqn.
Revised: 1 − η i λ j ) 2 with η i deﬁned in Eqn.

Original: (3.3) , for 1 ≤ t ≤ t (cid:48) , it holds that
Revised: (3.3) , for 1 ≤ t ≤ t (cid:48) , it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: If v t +1 ,j ≤ max( v t,j , η t /λ j ) holds for ∀ t ≥ 1 , then it naturally follows that
Revised: If v t +1 ,j ≤ max( v t,j , η t /λ j ) holds for ∀ t ≥ 1 , then it naturally follows that

Original: [Equation]
Revised: [Equation]

Original: where the last equality is entailed by the fact that t ≤ t (cid:48) and η t deﬁned in Eqn.
Revised: where the last equality is entailed by the fact that t ≤ t (cid:48) and η t deﬁned in Eqn.

Original: (3.3) is monotonically decreasing.
Revised: (3.3) is monotonically decreasing.

Original: We then prove v t +1 ,j ≤ max( v t,j , η t /λ j ) holds for ∀ t ≥ 1 .
Revised: We then prove v t +1 ,j ≤ max( v t,j , η t /λ j ) holds for ∀ t ≥ 1 .

Original: For ∀ t ≥ 1 , we have
Revised: For ∀ t ≥ 1 , we have

Original: [Equation]
Revised: [Equation]

Original: 1) If v t +1 ,j ≤ v t,j , then it naturally follows v t +1 ,j ≤ max( v t,j , η t /λ j ) .
Revised: 1) If v t +1 ,j ≤ v t,j , then it naturally follows v t +1 ,j ≤ max( v t,j , η t /λ j ) .

Original: 2) If v t +1 ,j > v t,j , denote a (cid:44) (1 − η t λ j ) 2 , b (cid:44) η 2 t , we have v t +1 ,j = av t,j + b , where a ∈ [0 , 1) and b ≥ 0 .
Revised: 2) If v t +1 ,j > v t,j , denote a (cid:44) (1 − η t λ j ) 2 , b (cid:44) η 2 t , we have v t +1 ,j = av t,j + b , where a ∈ [0 , 1) and b ≥ 0 .

Original: It follows,
Revised: It follows,

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: where the second inequality is entailed by the fact that 1 − η t λ j ∈ [0 , 1) .
Revised: where the second inequality is entailed by the fact that 1 − η t λ j ∈ [0 , 1) .

Original: Lemma 11.
Revised: Lemma 11.

Original: Letting v t,j be deﬁned as Lemma 10 and index ˜
Revised: Letting v t,j be deﬁned as Lemma 10 and index ˜

Original: i satisfy λ j ∈ [ µ · 2 ˜ i , µ · 2 ˜ i +1 ) , then v ˜ i +1 ,j has the following property
Revised: i satisfy λ j ∈ [ µ · 2 ˜ i , µ · 2 ˜ i +1 ) , then v ˜ i +1 ,j has the following property

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By the fact that (1 − x ) ≤ exp( − x ) , we have
Revised: By the fact that (1 − x ) ≤ exp( − x ) , we have

Original: [Equation]
Revised: [Equation]

Original: Setting t =
Revised: Setting t =

Original: t ˜
Revised: t ˜

Original: i +1 − 1 in above equation, we have
Revised: i +1 − 1 in above equation, we have

Original: [Equation]
Revised: [Equation]

Original: Now we bound the variance term.
Revised: Now we bound the variance term.

Original: First, we have
Revised: First, we have

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality is because η k ≤ 1 /L .
Revised: where the ﬁrst inequality is because η k ≤ 1 /L .

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: we can obtain
Revised: we can obtain

Original: [Equation]
Revised: [Equation]

Original: Furthermore, combining with Eqn.
Revised: Furthermore, combining with Eqn.

Original: (G.1) and the condition λ j ∈ [ µ · 2 ˜ i , µ · 2 ˜ i +1 ) , we can obtain
Revised: (G.1) and the condition λ j ∈ [ µ · 2 ˜ i , µ · 2 ˜ i +1 ) , we can obtain

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the last inequality is because of the condition λ j ∈ [ µ · 2 ˜ i , µ · 2 ˜ i +1 ) and the deﬁnition of α i .
Revised: where the last inequality is because of the condition λ j ∈ [ µ · 2 ˜ i , µ · 2 ˜ i +1 ) and the deﬁnition of α i .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: Lemma 12.
Revised: Lemma 12.

Original: Let objective function f ( x ) be quadratic and Assumption (1.7) hold.
Revised: Let objective function f ( x ) be quadratic and Assumption (1.7) hold.

Original: Running SGD for T -steps starting from w 0 and a learning rate sequence { η t } Tt =1 deﬁned in Eqn.
Revised: Running SGD for T -steps starting from w 0 and a learning rate sequence { η t } Tt =1 deﬁned in Eqn.

Original: (3.3) , the ﬁnal iterate w T +1 satisﬁes
Revised: (3.3) , the ﬁnal iterate w T +1 satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The target of this lemma is to obtain the explicit form to bound the variance term
Revised: The target of this lemma is to obtain the explicit form to bound the variance term

Original: [Equation]
Revised: [Equation]

Original: By the deﬁnition of v t +1 ,j in Lemma
Revised: By the deﬁnition of v t +1 ,j in Lemma

Original: [Equation]
Revised: [Equation]

Original: we can obtain that
Revised: we can obtain that

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is because λ j ∈ [2 ˜ i µ, 2 ˜ i +1 µ ) and there are s ˜
Revised: where the last inequality is because λ j ∈ [2 ˜ i µ, 2 ˜ i +1 µ ) and there are s ˜

Original: i such λ j ’s lie in this range.
Revised: i such λ j ’s lie in this range.

Original: By Eqn.
Revised: By Eqn.

Original: (3.3), we have
Revised: (3.3), we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: Combining with Lemma 4 and Lemma 6, we can obtain that
Revised: Combining with Lemma 4 and Lemma 6, we can obtain that

Original: [Equation]
Revised: [Equation]

Original: Lemma 13.
Revised: Lemma 13.

Original: For ∀ t ≥
Revised: For ∀ t ≥

Original: [Equation]
Revised: [Equation]

Original: the learning rate sequence { η t } Tt =1 deﬁned in Eqn.
Revised: the learning rate sequence { η t } Tt =1 deﬁned in Eqn.

Original: (3.3) satisﬁes
Revised: (3.3) satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For ∀ t ≥
Revised: For ∀ t ≥

Original: [Equation]
Revised: [Equation]

Original: there ∃ i ≥ 1 , where t ∈ [ t i − 1 , t i ) .
Revised: there ∃ i ≥ 1 , where t ∈ [ t i − 1 , t i ) .

Original: Given the form deﬁned in Eqn.
Revised: Given the form deﬁned in Eqn.

Original: (3.3), we have,
Revised: (3.3), we have,

Original: [Equation]
Revised: [Equation]

Original: Lemma 14.
Revised: Lemma 14.

Original: Let objective function f ( x ) be quadratic and Assumption (1.7) hold.
Revised: Let objective function f ( x ) be quadratic and Assumption (1.7) hold.

Original: Running SGD for T -steps starting from w 0 and a learning rate sequence { η t } Tt =1 deﬁned in Eqn.
Revised: Running SGD for T -steps starting from w 0 and a learning rate sequence { η t } Tt =1 deﬁned in Eqn.

Original: (3.3) , the ﬁnal iterate w T +1 satisﬁes
Revised: (3.3) , the ﬁnal iterate w T +1 satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The target of this lemma is to obtain the explicit form to bound the bias term.
Revised: The target of this lemma is to obtain the explicit form to bound the bias term.

Original: First, by Eqn.
Revised: First, by Eqn.

Original: (G.1) and the condition λ j ∈ [ µ · 2 ˜ i , µ · 2 ˜ i +1 ) , we haveexp (cid:32) [Equation] i
Revised: (G.1) and the condition λ j ∈ [ µ · 2 ˜ i , µ · 2 ˜ i +1 ) , we have

Original: [Equation]
Revised: [Equation]

Original: ˜ [Equation] [Equation] For λ j = µ , since µ ∈ [ µ · 2 ˜ i , µ · 2 ˜ i +1 ) for ˜ i
Revised: For λ j = µ , since µ ∈ [ µ · 2 ˜ i , µ · 2 ˜ i +1 ) for ˜ i

Original: = 0 , it follows, exp
Revised: = 0 , it follows, exp

Original: [Equation]
Revised: [Equation]

Original: Combining with Lemma 12, we obtain that,
Revised: Combining with Lemma 12, we obtain that,

Original: [Equation]
Revised: [Equation]

Original: G. P ROOF OF L EMMA
Revised: G. P ROOF OF L EMMA

Original: Lemma 15.
Revised: Lemma 1.

Original: Let objective function f ( x ) be quadratic and Assumption (1.7) hold.
Revised: Let objective function f ( x ) be quadratic and Assumption (1.7) hold.

Original: Running SGD for T -steps starting from w 0 and a learning rate sequence { η t } Tt =1 deﬁned in Eqn.
Revised: Running SGD for T -steps starting from w 0 and a learning rate sequence { η t } Tt =1 deﬁned in Eqn.

Original: (3.3) , the ﬁnal iterate w T +1 satisﬁes
Revised: (3.3) , the ﬁnal iterate w T +1 satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For ∀ t ≥
Revised: For ∀ t ≥

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the 5th equality is entailed by the fact that H (cid:62)
Revised: where the 5th equality is entailed by the fact that H (cid:62)

Original: =
Revised: =

Original: H is a symmetric matrix
Revised: H is a symmetric matrix

Original: [Equation]
Revised: [Equation]

Original: and the 9th equality uses both H (cid:62)
Revised: and the 9th equality uses both H (cid:62)

Original: =
Revised: =

Original: H and Eqn 1.4.
Revised: H and Eqn 1.4.

Original: Combine the above result with Lemma
Revised: Combine the above result with Lemma

Original: [Equation]
Revised: [Equation]

Original: we obtain that
Revised: we obtain that

Original: [Equation]
Revised: [Equation]

Original: G. P ROOF OF T HEOREM
Revised: G. P ROOF OF T HEOREM

Original: Theorem 
Revised: Theorem 1.

Original: [Equation]

Original: Let objective function f ( x ) be quadratic and Assumption (1.7) hold.
Revised: Let objective function f ( x ) be quadratic and Assumption (1.7) hold.

Original: Running SGD for
Revised: Running SGD for

Original: T -steps starting from w 0 , a learning rate sequence { η t } Tt =1 deﬁned in Eqn.
Revised: T -steps starting from w 0 , a learning rate sequence { η t } Tt =1 deﬁned in Eqn.

Original: (3.3) and ∆ i deﬁned in
Revised: (3.3) and ∆ i deﬁned in

Original: Eqn.
Revised: Eqn.

Original: [Equation] the ﬁnal iterate w T +1 satisﬁes
Revised: (3.5) , the ﬁnal iterate w T +1 satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We have 
Revised: We have µ

Original: [Equation]
Revised: [Equation]

Original: Combining with Lemma 1 and the deﬁnition of ∆ 1 , we can obtain that
Revised: Combining with Lemma 1 and the deﬁnition of ∆ 1 , we can obtain that

Original: [Equation]
Revised: [Equation]

Original: G. P ROOF OF C OROLLARY
Revised: G. P ROOF OF C OROLLARY

Original: Corollary 6.
Revised: Corollary 2.

Original: Given the same setting as in Theorem 1, when Hessian H ’s eigenvalue distribution p ( λ ) satisﬁes “power power law”, i.e.
Revised: Given the same setting as in Theorem 1, when Hessian H ’s eigenvalue distribution p ( λ ) satisﬁes “power power law”, i [Equation] e.

Original: [Equation]
Revised: [Equation]

Original: G.18) for some α > 1 , where Z = (cid:82) L µ ( µ/λ ) α dλ , there exists a constant C ( α ) which only depends on α , such that the ﬁnal iterate w T +1 satisﬁes 
Revised: for some α > 1 , where Z = (cid:82) L µ ( µ/λ ) α dλ , there exists a constant C ( α ) which only depends on α , such that the ﬁnal iterate w T +1 satisﬁes [Equation]

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: According to Theorem 1,
Revised: According to Theorem 1,

Original: [Equation]
Revised: [Equation]

Original: The key terms here are C 1 (cid:44) (cid:16)(cid:80)
Revised: The key terms here are C 1 (cid:44) (cid:16)(cid:80)

Original: I max − 1 i =
Revised: I max − 1 i =

Original: √
Revised: √

Original: s i (cid:17) 2 /s 0 and C 2 (cid:44) 15 (cid:16)(cid:80)
Revised: s i (cid:17) 2 /s 0 and C 2 (cid:44) 15 (cid:16)(cid:80)

Original: I max − 1 i =
Revised: I max − 1 i =

Original: √
Revised: √

Original: s i (cid:17) 2 /d .
Revised: s i (cid:17) 2 /d .

Original: As long as we can bound both terms with a constant C ( α ) , the corollary will be directly proved.
Revised: As long as we can bound both terms with a constant C ( α ) , the corollary will be directly proved.

Original: 1) If κ < 2 , then there is only one interval with s 0 = d .
Revised: 1) If κ < 2 , then there is only one interval with s 0 = d .

Original: By setting C ( α ) =
Revised: By setting C ( α ) =

Original: max( C1 , C 2 ) = 15 , this completes the proof.
Revised: max( C , C 2 ) = 15 , this completes the proof.

Original: If κ ≥ 2 , then bounding C 1 and C 2 be done by computing the value of s i under power power law.
Revised: If κ ≥ 2 , then bounding C 1 and C 2 be done by computing the value of s i under power power law.

Original: For all interval i except the last interval, we have,
Revised: For all interval i except the last interval, we have,

Original: [Equation]
Revised: [Equation]

Original: Therefore [Equation] we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: ) holds for all interval i except the last interval i (cid:48)
Revised: holds for all interval i except the last interval i (cid:48)

Original: =
Revised: =

Original: I max − 1 = log 2 κ − 1 > 0 .
Revised: I max − 1 = log 2 κ − 1 > 0 .

Original: This last interval may not completely covers [ µ · 2 i (cid:48) , µ · 2 i (cid:48)
Revised: This last interval may not completely covers [ µ · 2 i (cid:48) , µ · 2 i (cid:48)

Original: +1 )
Revised: +1 )

Original: due to the boundary truncated by L 
Revised: due to the boundary truncated by L ,

Original: [Equation]

Original: but we still have
Revised: but we still have

Original: [Equation]
Revised: [Equation]

Original: It follows,
Revised: It follows,

Original: [Equation]
Revised: [Equation]

Original: i = √

Original: s i (cid:33)≤ d · 2 1 − α −κ 1 − α − 1 · (cid:32)

Original: I max − 1 (cid:88) i =

Original: [-(cid:112) 
-]
Revised: [-(cid:112) 
-]

Original: i (1 − α ) (cid:33)= d · 2 1 − α −κ 1 − α − 1 · (cid:32)

Original: I max − 1 (cid:88) i

Original: [-= 
-]
Revised: [-= 
-]

Original: i (1 − α ) /(cid:33)=

Original: d · 2 1 − α −κ 1 − α − 1 · (cid:32)

Original: I max − 1 (cid:88) i = (cid:18) (cid:19) i ( α − 1) / 2 (cid:33)≤ d · 2 1 − α −κ 1 − α − 1 · (cid:32) ∞ (cid:88)

Original: i = (cid:18) (cid:19)

Original: i ( α − 1) / 2 (cid:33)= d · 2 1 − α −κ 1 − α − 1 · (cid:32) ∞ (cid:88)

Original: i = (cid:18) 2 ( α − 1) /(cid:19)

Original: i (cid:33)= d · 2 1 − α −κ 1 − α − 1 · (cid:32) 1 − 1 2 ( α − 1) /(cid:33)=

Original: d · 2 1 − α −κ 1 − α − 1 · (cid:18) 1 − 2 (1 − α ) /(cid:19).

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: Here the last inequality for C 2 is entailed by κ ≥ 2 and α > 
Revised: Here the last inequality for C 2 is entailed by κ ≥ 2 and α > 1 .

Original: [Equation]

Original: By setting C ( α ) =
Revised: By setting C ( α ) =

Original: max( C [Equation] , C 
Revised: max( C 1 , C 2 ) = 15 · (cid:16)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Revised: (cid:17)

Original: [Equation]
Revised: [Equation]

Original: (cid:17) 2 , we obtain 
Revised: we obtain E

Original: [Equation]
Revised: [Equation]

Original: G. P ROOF OF T HEOREM
Revised: G. P ROOF OF T HEOREM

Original: Theorem 7.
Revised: Theorem 4.

Original: Let objective function f ( x ) be quadratic.
Revised: Let objective function f ( x ) be quadratic.

Original: We run SGD for T -steps starting from w 0 and a step decay learning rate sequence { η t } Tt =1 deﬁned in Algorithm 1 of Ge et al.
Revised: We run SGD for T -steps starting from w 0 and a step decay learning rate sequence { η t } T t =1 deﬁned in Algorithm 1 of Ge et al.

Original: (2019) with η 1 = 1 /L .
Revised: (2019) with η 1 = 1 /L .

Original: As long as (1) H is diagonal, (2) The equality in Assumption (1.7) holds, i.e. E ξ (cid:2) n t n (cid:62) t (cid:3) =
Revised: As long as (1) H is diagonal, (2) The equality in Assumption (1.7) holds, i.e. E ξ (cid:2) n t n (cid:62) t (cid:3) =

Original: σ 2 H and (3) T/
Revised: σ 2 H and (3) T/

Original: log T ≥ max( κ, 2 16 , 16) , the ﬁnal iterate w T +1 satisﬁes,
Revised: log T ≥ max( κ, 2 16 , 16) , the ﬁnal iterate w T +1 satisﬁes,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: In Ge et al.
Revised: In Ge et al.

Original: (2019), step decay has following learning rate sequence:
Revised: (2019), step decay has following learning rate sequence:

Original: [Equation]
Revised: [Equation]

Original: where (cid:96) = 0 , 1 , . .
Revised: where (cid:96) = 0 , 1 , . .

Original: , log T − 1 .
Revised: , log T − 1 .

Original: We ﬁrst translate the general notations to diagonal cases so that the idea of the proof can be clearer.
Revised: We ﬁrst translate the general notations to diagonal cases so that the idea of the proof can be clearer.

Original: Since f ( x ) is quadratic, according to  Lemma 15,
Revised: Since f ( x ) is quadratic, according to the proof in Lemma 1,

Original: [Equation]
Revised: [Equation]

Original: Furthermore, according to Lemma 4, where P t = I − η t H ,
Revised: Furthermore, according to Lemma 4, where P t = I − η t H ,

Original: [Equation]
Revised: [Equation]

Original: Here the second equality is entailed by the fact that H and P t are diagonal, and the third equality comes from E ξ (cid:2) n t n (cid:62) t (cid:3) =
Revised: Here the second equality is entailed by the fact that H and P t are diagonal, and the third equality comes from E ξ (cid:2) n t n (cid:62) t (cid:3) =

Original: σ 2 H .
Revised: σ 2 H .

Original: Thus, by denoting v j (cid:44) (cid:80) {+Tτ =
+}
Revised: Thus, by denoting v j (cid:44) (cid:80) {+Tτ =
+}

Revised: η 2 τ (cid:81) Tk = τ +

Original: [Equation]
Revised: [Equation]

Original:  we have, 
Revised: 1 − η k λ j ) 2 , we have, [Equation] G [Equation]

Original: [Equation]
Revised: [Equation]

Original: For each dimension λ j , step decay’s bottleneck actually occurs at the ﬁrst interval (cid:96) that satisﬁes
Revised: For each dimension λ j , step decay’s bottleneck actually occurs at the ﬁrst interval (cid:96) that satisﬁes

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: We ﬁrst show that interval (cid:96) is well-deﬁned for any dimension j .
Revised: We ﬁrst show that interval (cid:96) is well-deﬁned for any dimension j .

Original: Since we assume T/ log T ≥ κ , i.e. 1 /κ ≥ log T/T , it follows
Revised: Since we assume T/ log T ≥ κ , i.e. 1 /κ ≥ log T/T , it follows

Original: [Equation]
Revised: [Equation]

Original: Since interval (cid:96) is the ﬁrst interval satisiﬁes Eqn.
Revised: Since interval (cid:96) is the ﬁrst interval satisiﬁes Eqn.

Original: (G.22), we also have
Revised: (G.21), we also have

Original: [Equation]
Revised: [Equation]

Original:  which follows,
Revised: 22) which follows,

Original: [Equation]
Revised: [Equation]

Original: The last inequality is because T ≥ T/ log T ≥ 2 16 ⇒ log T ≥ 16 .
Revised: The last inequality is because T ≥ T/ log T ≥ 2 16 ⇒ log T ≥ 16 .

Original: Since T ≥ 16 ⇒ log T ≥ 4 > 3 , thus such (cid:96) is guaranteed to be well-deﬁned.
Revised: Since T ≥ 16 ⇒ log T ≥ 4 > 3 , thus such (cid:96) is guaranteed to be well-deﬁned.

Original: Back to our main analysis, by focusing on interval (cid:96) only, we have,
Revised: Back to our main analysis, by focusing on interval (cid:96) only, we have,

Original: [Equation]
Revised: [Equation]

Original: k

Original: = τ + (1 − η k λ j )

Original: [Table]

Original: >
Revised: >

Original: T log T ·

Original: [Table]
Revised: [Equation]

Original: L

Original: [Table]
Revised: [Equation]

Original: = [Table] [Table] log 
Revised: T + (1 − η k λ j )≥256 · log T

Original: [Table]
Revised: T ·λ 2 j ·   1 −

Original: T [Table]
Revised: T (cid:88) k

Original: ≥ [Table] log 
Revised: = (cid:96) · T log T + 2 η k λ j   = 1 256 · log T T ·λ 2 j ·   1 − 2 λ j

Original: [Table] T 
Revised: T (cid:88) k

Original: [Table] [Table]  =256 · log TT ·
Revised: = (cid:96) · T log T + η k   =256 · log T

Original: [Table]
Revised: T ·λ 2 j ·   1 − 2 λ

Original:  log T  (cid:88) [Equation]
Revised: j log T − 1 (cid:88) i

Original: = 
Revised: = (cid:96) ( i +1) · T log T (cid:88) k = i · T log

Original: [Equation]
Revised: [Table]

Original:  k [Equation]  =256 · log T
Revised: + η k   =256 · log T

Original: T 
Revised: T ·λ 2 j ·   1 − 2 λ j log T − [Table] i =

Original: [Equation]
Revised: [Table]

Original:  i
Revised: ( i

Original: [Equation]
Revised: [Table] [Table]

Original: i
Revised: T

Original: [Equation]
Revised: [Table]

Original: [Equation]

Original: k = i · T log T +[-L 
 i-]
Revised: k = i · T log T +[-L 
 i-]

Original: [Equation]  =256 · log T
Revised: L 2 i   =256 · log T

Original: [Equation]
Revised: T ·λ 2 j · (cid:32) 1 − 2 λ j log T − 1 (cid:88) i = (cid:96) T log T ·

Original: [Equation] ≥256 · log T
Revised: L 2 i (cid:33) ≥256 · log T

Original: [Equation]
Revised: T ·λ 2 j · (cid:18) 1 − 2 λ j ·

Original: ) ≥ 256 · log 
Revised: T log T · L 2 (cid:96) −(cid:19) (G.21) ≥ 256 · log T

Original: [Equation]
Revised: T ·λ 2 j · (cid:32) 1 − 4 λ j · T log T ·

Original:  =256 · log [Equation] [Equation]
Revised: L · λ j L · 8 T log T (cid:33) =256 · log T

Original:  log 
Revised: T ·λ 2 j ·  =512 · log T

Original: [Equation]
Revised: T ·λ 2 j

Original: Here the ﬁrst inequality is obtained by focusing variance generated in interval (cid:96) only.
Revised: Here the ﬁrst inequality is obtained by focusing variance generated in interval (cid:96) only.

Original: The second inequality utilizes τ + 1 ≥ (cid:96) · T/ log T .
Revised: The second inequality utilizes τ + 1 ≥ (cid:96) · T/ log T .

Original: The fourth inequality is entailed by (1 − a 1 )(1 − a 2 ) = 1 − a 1 − a 2 + a 1 a 2 ≥ 1 − a 1 − a 2 for ∀ a 1 , a 2 ∈ [0 , 1] , where by mathematical induction, we can extend this inequality for more terms (cid:81) ni
Revised: The fourth inequality is entailed by (1 − a 1 )(1 − a 2 ) = 1 − a 1 − a 2 + a 1 a 2 ≥ 1 − a 1 − a 2 for ∀ a 1 , a 2 ∈ [0 , 1] , where by mathematical induction, we can extend this inequality for more terms (cid:81) ni

Original: =1 (1 − a i ) ≥ 1 − (cid:80) ni
Revised: =1 (1 − a i ) ≥ 1 − (cid:80) ni

Original: =1 a i as long as (cid:80) ni =1 a i ≤ 1 .
Revised: =1 a i as long as (cid:80) ni =1 a i ≤ 1 .

Original: The ﬁfth inequality comes from (cid:80) log T − 1 i = (cid:96) 
Revised: The ﬁfth inequality comes from (cid:80) log T − 1 i = (cid:96) 1 / 2 i ≤ (cid:80) ∞

Original: [Equation]
Revised: i = (cid:96) 1 / 2 i = 1 / 2 (cid:96) − 1 .

Original: By combining with Eqn.
Revised: By combining with Eqn.

Original: (G.21) [Equation] we have
Revised: (G.20), we have

Original: [Equation]
Revised: [Equation]

Original: G. T HE R EASON OF U SING A SSUMPTION (1.7)
Revised: G. T HE R EASON OF U SING A SSUMPTION (1.7)

Original: In all of our analysis, we employ assumption (1.7)
Revised: In all of our analysis, we employ assumption (1.7)

Original: [Equation]
Revised: [Equation]

Original: where n t =
Revised: where n t =

Original: Hw t − b − ( H ( ξ ) w t − b ( ξ )) which is the same as the one in Appendix C, Theorem 13 of Ge et al.
Revised: Hw t − b − ( H ( ξ ) w t − b ( ξ )) which is the same as the one in Appendix C, Theorem 13 of Ge et al.

Original: This key theorem is the major difference between our work and Ge et al.
Revised: This key theorem is the major difference between our work and Ge et al.

Original: (2019), which directly entails its main theorem by instantiating σ with speciﬁc values in its assumptions.
Revised: (2019), which directly entails its main theorem by instantiating σ with speciﬁc values in its assumptions.

Original: On the other hand, it is possible to use the assumptions in Ge et al.
Revised: On the other hand, it is possible to use the assumptions in Ge et al.

Original: (2019); Bach & Moulines (2013);
Revised: (2019); Bach & Moulines (2013);

Original: Jain et al.
Revised: Jain et al.

Original: (2016) instead of our assumption (1.7) for least square regression: min  wf ( w ) where f ( w ) (cid:44) 1 2 E ( x,y ) ∼D (cid:2) ( y − w (cid:62) x ) 2 (cid:3) (G.24) y
Revised: (2016) instead of our assumption (1.7) for least square regression: min [Equation] w

Original: = w (cid:62)∗ x + (cid:15) with (cid:15) satisfying E ( x,y ) ∼D (cid:2) (cid:15) 2 xx (cid:62) (cid:3) (cid:22) σ 2 H for ∀ ( x, y ) ∼ D (G.25)
Revised: [Equation] where f ( w ) [Equation] [Equation]

Original: E
Revised: y [Equation] w [Equation]

Revised: G.

Original: [Equation]
Revised: [Equation]

Original: H (G.26)
Revised: y [Equation] (cid:62)∗ x + (cid:15) with (cid:15) satisfying E [Equation]

Original:  By combining our Lemma 1 and assumption (1.7) with Lemma 5, Lemma 8 and Lemma 9 in Ge et al.
Revised: for [Equation] [Equation] By combining our Lemma 1 and assumption (1.7) with Lemma 5, Lemma 8 and Lemma 9 in Ge et al.

Original: (2019), one can obtain similar results in this paper with their assumptions.
Revised: (2019), one can obtain similar results in this paper with their assumptions.

Original: For simplicity, we just use assumption (1.7) here.
Revised: For simplicity, we just use assumption (1.7) here.