Original: A DA DQH O PTIMIZER : E VOLVING FROM S TOCHASTIC TO A DAPTIVE BY A UTO S WITCH OF P RECONDITION M ATRIX
Revised: A DA DQH O PTIMIZER : E VOLVING FROM S TOCHASTIC TO A DAPTIVE BY A UTO S WITCH OF P RECONDITION M ATRIX

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Adaptive optimizers (e.g., Adam) have achieved tremendous success in deep learning.
Revised: Adaptive optimizers (e.g., Adam) have achieved tremendous success in deep learning.

Original: The key component of the optimizer is the precondition matrix, which provides more gradient information and adjusts the step size of each gradient direction.
Revised: The key component of the optimizer is the precondition matrix, which provides more gradient information and adjusts the step size of each gradient direction.

Original: Intuitively, the closer the precondition matrix approximates the Hessian, the faster convergence and better generalization the optimizer can achieve in terms of iterations.
Revised: Intuitively, the closer the precondition matrix approximates the Hessian, the faster convergence and better generalization the optimizer can achieve in terms of iterations.

Original: However, this performance improvement is usually accompanied by a huge increase in the amount of computation.
Revised: However, this performance improvement is usually accompanied by a huge increase in the amount of computation.

Original: In this paper, we propose a new optimizer called AdaDQH to achieve better generalization with acceptable computational overhead.
Revised: In this paper, we propose a new optimizer called AdaDQH to achieve better generalization with acceptable computational overhead.

Original: The intuitions are the trade-off of the precondition matrix between computation time and approximation of Hessian, and the auto switch of the precondition matrix from Stochastic Gradient Descent (SGD) to the adaptive optimizer.
Revised: The intuitions are the trade-off of the precondition matrix between computation time and approximation of Hessian and the auto switch of the precondition matrix from Stochastic Gradient Descent (SGD) to the adaptive optimizer.

Original: We evaluate AdaDQH on public datasets of Computer Vision (CV), Natural Language
Revised: We evaluate AdaDQH on public datasets of Computer Vision (CV) Natural Language

Original: Processing (NLP) and Recommendation Systems (RecSys).
Revised: Processing (NLP) and Recommendation Systems RecSys).

Revised: The experimental results reveal that, compared to the

Revised: State

Revised: Of

Original: The experimental results reveal that, compared to the State-Of-The-Art (SOTA) optimizers, AdaDQH can achieve signiﬁcantly better or highly competitive performance.
Revised: -The- Art (SOTA) optimizers, AdaDQH can achieve signiﬁcantly better or highly competitive performance

Original: Furthermore we analyze how AdaDQH is able to auto switch from stochastic to adaptive and the actual effects in different scenes.
Revised: Furthermore, we analyze how AdaDQH is able to auto switch from stochastic to adaptive and the actual effects in different scenes

Original: The code is available in the supplemental material.
Revised: The code is available in the supplemental material

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Consider the following empirical risk minimization problems:
Revised: Consider the following empirical risk minimization problems:

Original: [Equation]
Revised: [Equation]

Original: where w ∈ R
Revised: where w ∈ R

Original: n is a vector of parameters to be optimized, { x 1 , . .
Revised: n is a vector of parameters to be optimized, { x 1 , . .

Original: , x M } is a training set, and (cid:96) ( w ; x ) is a loss function measuring the performance of the parameter w on the example x .
Revised: , x M } is a training set, and (cid:96) ( w ; x ) is a loss function measuring the performance of the parameter w on the example x .

Original: Since it is ineffective to calculate the exact gradient in each optimization iteration when M is large, we usually adopt a mini-batched stochastic gradient, which is
Revised: Since it is ineffective to calculate the exact gradient in each optimization iteration when M is large, we usually adopt a mini-batched stochastic gradient, which is

Original: [Equation]
Revised: [Equation]

Original: where B ⊂ { 1 , . . .
Revised: where B ⊂ { 1 , . . .

Original: , M } is the sample set of size |B| (cid:28) M .
Revised: , M } is the sample set of size |B| (cid:28) M .

Original: Obviously, we have E p ( x ) [ g ( w )]
Revised: Obviously, we have E p ( x ) [ g ( w )]

Original: = ∇ f ( w ) where p ( x ) is the distribution of the training data.
Revised: = ∇ f ( w ) where p ( x ) is the distribution of the training data.

Original: Equation 1 is usually solved iteratively.
Revised: Equation 1 is usually solved iteratively

Original: Assume w t is already known and let ∆ w = w t +1 − w t , then

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: Duchi et al., 2011)

Original: [Equation]
Revised: [Equation]

Original: (Zeiler, 2012), [Equation] (Kingma & Ba, 2015) [Equation]
Revised: Assume w t is already known and let ∆ w = w t +1 − w t , then

Original: (Loshchilov & Hutter, 2019),

Original: [Equation]

Original: (Reddi et al., 2018)

Original: [Equation]

Original: Zhuang et al., 2020)

Original: [Equation]
Revised: [Equation]

Original: (Robbins & Monro, 1951)

Original: [Equation]

Original: (Polyak, 1964)

Original: [Table]

Original: where the ﬁrst approximation is from Taylor expansion.
Revised: where the ﬁrst approximation is from Taylor expansion.

Original: By solving Equation 2, the general update formula is
Revised: By solving Equation 2 and using m t to replace ∇ f ( w t ) , the general update formula is

Original: [Equation]
Revised: [Equation]

Original: where α t is the step size for avoiding divergence, m t  is the ﬁrst moment term which is the weighted average of gradient g t and B t is the so-called precondition matrix that incorporates additional information and adjusts
Revised: where α t is the step size for avoiding divergence, m t ≈ E p ( x ) [ g t ] is the ﬁrst moment term which is the weighted average of gradient g t and B t is the so-called precondition matrix that incorporates additional information and adjusts

Original: updated velocity of variable w t in each direction.
Revised: updated velocity of variable w t in each direction.

Original: Most of gradient descent algorithms can be summarized with Equation 3 such as SGD (Robbins & Monro, 1951), M OMENTUM (Polyak, 1964), A DAGRAD (Duchi et al., 2011), A DA D ELTA (Zeiler, 2012), A DAM (Kingma & Ba, 2015), AMSG RAD (Reddi et al., 2018), A DA B ELIEF (Zhuang et al., 2020) and A DA H ESSIAN (Yao et al., 2020).
Revised: Most of gradient descent algorithms can be summarized with Equation 3 such as SGD (Robbins & Monro, 1951), M OMENTUM (Polyak, 1964), A DAGRAD (Duchi et al., 2011), A DA D ELTA (Zeiler, 2012), A DAM (Kingma & Ba, 2015), AMSG RAD (Reddi et al., 2018), A DA B ELIEF (Zhuang et al., 2020) and A DA H ESSIAN (Yao et al., 2020).

Original: Intuitively, the closer B t approximates the Hessian, the closer h (∆ w ) approximates f ( w t +1 ) .
Revised: Intuitively, the closer B t approximates the Hessian, the closer h (∆ w ) approximates f ( w t +1 ) .

Original: Consequently, we can achieve a more accurate solution in terms of iterations.
Revised: Consequently, we can achieve a more accurate solution in terms of iterations.

Original: However, it is usually untrue in terms of runtime.
Revised: However, it is usually untrue in terms of runtime.

Original: For instance, A DA H ESSIAN that approximates the diagonal Hessian consumes 2 .
Revised: For instance, A DA H ESSIAN that approximates the diagonal Hessian consumes 2 .

Original: 91 × more computation time than A DAM for ResNet32 on Cifar10 (Yao et al., 2020).
Revised: 91 × more computation time than A DAM for ResNet32 on Cifar10 (Yao et al., 2020).

Original: Therefore, the key factor of designing the precondition matrix is how to trade off the approximation degree of the Hessian and the computation complexity.
Revised: Therefore, the key factor of designing the precondition matrix is how to trade off the approximation degree of the Hessian and the computation complexity.

Original: In this paper, we propose AdaDQH ( Ada ptive optimizer with D iagonal Q uasi- H essian), whose precondition matrix is closely related to the Hessian but computationally efﬁcient.
Revised: In this paper, we propose AdaDQH ( Ada ptive optimizer with D iagonal Q uasi- H essian), whose precondition matrix is closely related to the Hessian but computationally efﬁcient.

Original: Furthermore,
Revised: Furthermore,

Original: AdaDQH can auto switch the precondition matrix from SGD to the adaptive optimizer through the hyperparameter threshold δ .
Revised: AdaDQH can auto switch the precondition matrix from SGD to the adaptive optimizer through the hyperparameter threshold δ .

Original: Our contributions can be summarized as follows
Revised: Our contributions can be summarized as follows.

Original: • We propose AdaDQH, which originates the new design of the precondition matrix.
Revised: • We propose AdaDQH, which originates the new design of the precondition matrix.

Original: We establish theoretically proven convergence guarantees in both convex and non-convex stochastic settings.
Revised: We establish theoretically proven convergence guarantees in both convex and non-convex stochastic settings.

Original: • We validate AdaDQH on a total of six public datasets: two from CV (Cifar10 (Krizhevsky et al., 2009) and ImageNet (Russakovsky et al., 2015)), two from NLP (IWSLT14 (Cettolo et al., 2014) and PTB (Marcus et al., 1993)) and the rest from RecSys (Criteo (Criteo, 2014) and Avazu (Avazu, 2015)).
Revised: • We validate AdaDQH on a total of six public datasets: two from CV (Cifar10 (Krizhevsky et al., 2009) and ImageNet (Russakovsky et al., 2015)), two from NLP (IWSLT14 (Cettolo et al., 2014) and PTB (Marcus et al., 1993)) and the rest from RecSys (Criteo (Criteo, 2014) and Avazu (Avazu, 2015)).

Original: The experimental results reveal that AdaDQH can outperform or be on a par with the SOTA optimizers.
Revised: The experimental results reveal that AdaDQH can outperform or be on a par with the SOTA optimizers.

Original: • We analyze how AdaDQH is able to auto switch from stochastic to adaptive, and assess the rigorous effect of the hyperparameter δ which controls the auto-switch process in different scenes.
Revised: • We analyze how AdaDQH is able to auto switch from stochastic to adaptive, and assess the rigorous effect of the hyperparameter δ which controls the auto-switch process in different scenes.

Original: R ELATED WORK
Revised: R ELATED WORK

Original: By choosing different B t and m t of Equation 3, different optimizers are invented from the standard second order optimizer, i.e., Gauss-Newton method to the standard ﬁrst order optimizer, i.e., SGD(Robbins & Monro, 1951) where m t is usually designed for noise reduction and B t for solving the ill-conditioned problems.
Revised: By choosing different B t and m t of Equation 3, different optimizers are invented from the standard second order optimizer, i.e., Gauss-Newton method to the standard ﬁrst order optimizer, i.e., SGD where m t is usually designed for noise reduction and B t for solving the ill-conditioned problems.

Original: See Table 1.
Revised: See Table 1.

Original: Kunstner et al.
Revised: Kunstner et al.

Original: (2019) shows that the Fisher information matrix can be the reasonable approximation of the Hessian whereas the empirical Fisher can’t.
Revised: (2019) shows that the Fisher information matrix can be the reasonable approximation of the Hessian whereas the empirical Fisher can’t.

Original: Furthermore, they propose the concept of variance adaption to explain the practical success of the empirical Fisher preconditioning.
Revised: Furthermore, they propose the concept of variance adaption to explain the practical success of the empirical Fisher preconditioning.

Revised: The hybrid optimization methods of switching an adaptive optimizer to SGD have been proposed for improving the generalization performance, such as A DA B OUND (Luo et al., 2019) and SWATS (Keskar & Socher, 2017).

Revised: Luo et al.

Revised: (2019) adopts clipping on the learning rate of A DAM , whose upper and lower bounds are a non-increasing and non-decreasing functions, respectively, which would converge to the learning rate of SGD.

Revised: The clipping method is also mentioned in Keskar &

Revised: Socher (2017), whose upper and lower bounds are constants.

Original: N OTATION
Revised: N OTATION

Original: We use lowercase letters to denote scalars and vectors, and uppercase letters to denote matrices.
Revised: We use lowercase letters to denote scalars, boldface lowercase to denote vectors, and uppercase letters to denote matrices.

Original: We denote a sequence of vectors by subscripts, that is, x 1 , . . .
Revised: We denote a sequence of vectors by subscripts, that is, x 1 , . . .

Original: , x t where t ∈ [ T ] := { 1 , 2 , . . .
Revised: , x t where t ∈ [ T ] := { 1 , 2 , . . .

Original: , T } , and entries of each vector by an additional subscript,
Revised: , T } , and entries of each vector by an additional subscript,

Original: e.g., x t,i .
Revised: e.g., x t,i .

Original: For any vectors x, y ∈ R n , we write x T y or x · y for the standard inner product, xy for element-wise multiplication, x/y for elementwise division, √ x for element-wise square root, x 2 for element-wise square.
Revised: For any vectors x , y ∈ R n , we write x T y or x · y for the standard inner product, xy for element-wise multiplication, x / y for element-wise division, √ x for element-wise square root, x 2 for element-wise square.

Original: For the standard Euclidean norm, (cid:107) x (cid:107) =
Revised: For the standard Euclidean norm, (cid:107) x (cid:107) =

Original: (cid:107) x (cid:107) 2 = (cid:112) (cid:104) x, x (cid:105) and max( x, y ) for element-wise maximum.
Revised: (cid:107) x (cid:107) 2 = (cid:112) (cid:104) x , x (cid:105) and max( x , y ) for element-wise maximum.

Original: We also use (cid:107) x (cid:107) ∞ =
Revised: We also use (cid:107) x (cid:107) ∞ =

Original: max i | x ( i ) | to denote (cid:96) ∞ -norm, where x ( i ) is the i -th element of x .
Revised: max i | x ( i ) | to denote (cid:96) ∞ -norm, where x ( i ) is the i -th element of x .

Original: Let e i denote the unit vector where the i -th element is one and ∇ i f denote the i -th element of ∇ f . 
Revised: Let e i denote the unit vector where the i -th element is one and ∇ i f denote the i -th element of ∇ f . [Table] AUSS [Table]

Revised: ≈ H

Revised: LBFGS (Byrd et al., 1995)

Original: Let f t ( w ) be the loss function of the model at t -step where w ∈ R n .
Revised: [Table] [Equation] H ) [Table]

Revised: ESSIAN (

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: =

Revised: [Table]

Revised: N ATURAL

Revised: [Table]

Revised: [Table]

Revised: S

Revised: [Table]

Revised: (Gupta

Revised: [Table]

Revised: [Table]

Revised: A

Revised: [Table]

Revised: (Duchi et al., 2011),

Revised: A

Revised: [Table]

Revised: D ELTA (Zeiler, 2012),

Revised: A DAM (Kingma & Ba, 2015), A DAM W (Loshchilov & Hutter, 2019),

Revised: [Table]

Revised: (Reddi et al., 2018)

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: et al., 2020)

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: SGD (Robbins & Monro, 1951), M

Revised: [Table]

Revised: (Polyak, 1964)

Revised: [Table]

Revised: Let f t ( w ) be the loss function of the model at t -step where w ∈ R n .

Original: We consider m t as Exponential
Revised: We consider m t as Exponential

Original: Moving Averages (EMA) of g t throughout this paper, i.e.,
Revised: Moving Averages (EMA) of g t throughout this paper, i.e.,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where β 1 ∈ [0 , 1) is the exponential decay rate.
Revised: where β 1 ∈ [0 , 1) is the exponential decay rate.

Original:  A LGORITHM 2. D ETAILS AND INTUITIONS OF A DA DQH OPTIMIZER
Revised:  A LGORITHM 2. D ETAILS AND INTUITIONS OF A DA DQH OPTIMIZER

Original: The algorithm is listed in Algorithm 1.
Revised: The algorithm is listed in Algorithm 1.

Original: The design of AdaDQH comes from two intuitions: Hessian
Revised: The design of AdaDQH comes from two intuitions: Hessian

Original: Algorithm 1 AdaDQH
Revised: Algorithm 1 AdaDQH

Original: 1: Input: parameters β 1 , β 2 , δ , w 1 ∈ R n , step size
Revised: 1: Input  2: for t = 1 to T do 3: [Equation]

Original: α t , initialize m 0 = 0 , b 0 =2: for t = 1 to T do 3: g t = ∇ f t ( w t ) 4: m t ← β 1 m t − 1 + (1 − β 1 ) g t 5: s t = (cid:26) m 1 / (1 − β 1 ) t
Revised: [Equation] ← [Equation]

Original: =m t / (1 − β t 1 ) − m t − 1 / (1 − β t − 1) t >6: b t ← β 2 b t − 1 + (1 − β 2 ) s 2 t 7: w t
Revised: [Equation] 6: b t ← β [Equation] t − 1 + (

Original: +1 =
Revised: [Equation] β [Equation]

Original: w t − α t
Revised: t 7: w t

Original: [Equation]
Revised: [Equation]

Revised: =

Original: − β t1 − β tm t max( √ b t ,δ √ 1 − β t 2 ) 8: end for approximation and auto switch for fast convergence and good generalization across tasks.
Revised: [Equation] t [Equation] 8: end for approximation and auto switch for fast convergence and good generalization across tasks.

Original: H ESSIAN APPROXIMATION
Revised: H ESSIAN APPROXIMATION

Original: Let ∆ w = − α t B − 1 t m t of Equation 3, then we have
Revised: Let ∆ w = − α t B − 1 t m t of Equation 3, then we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the second equality above follows from the mean value theorem and in the second approximation we assume that w t is not updated except for the i -th direction.
Revised: where the second equality above follows from the mean value theorem and in the second approximation we assume that w t is not updated except for the i -th direction.

Original: Therefore, we can see that
Revised: Therefore, we can see that

Original: E [ g ( w t ) − g ( w t − 1 )] is closely related to diag ( H ( w t )) .
Revised: E [ g ( w t ) − g ( w t − 1 )] is closely related to diag ( H ( w t )) .

Original: Similar to Kingma & Ba (2015), we use m t / (1 − β t 1 ) to approximate E [ g ( w t )] where the denominator is for bias correction.
Revised: Similar to Kingma & Ba (2015), we use m t / (1 − β t 1 ) to approximate E [ g ( w t )] where the denominator is for bias correction.

Original: Denote
Revised: Denote

Original: [Equation]
Revised: [Equation]

Original: Therefore, we choose the precondition matrix B t satisfying
Revised: Therefore, we choose the precondition matrix B t satisfying

Original: [Equation]
Revised: [Equation]

Original: where β 2 is the parameter of EMA and the denominator is also for bias correction.
Revised: where β 2 is the parameter of EMA and the denominator is also for bias correction.

Original: A UTO SWITCH
Revised: A UTO SWITCH

Original: Normally, a small value is added to B t for numerical stability, becoming B t + (cid:15) I .
Revised: Normally, a small value is added to B t for numerical stability, becoming B t + (cid:15) I .

Original: However, we replace it with max( B t , δ I ) , where we use a different notation δ to indicate its essential role in auto switch.
Revised: However, we replace it with max( B t , δ I ) , where we use a different notation δ to indicate its essential role in auto switch.

Original: When ˆ b t := (cid:112) b t / (1 − β t 2 ) is relatively larger than δ , AdaDQH takes a conﬁdent step in the adaptive way.
Revised: When ˆ b t := (cid:112) b t / (1 − β t 2 ) is relatively larger than δ , AdaDQH takes a conﬁdent step in the adaptive way.

Original: Otherwise, the update is EMA, i.e. m t , with a constant scale α t / (1 − β t 1 ) , similar to SGD with momentum.
Revised: Otherwise, the update is EMA, i.e. m t , with a constant scale α t / (1 − β t 1 ) , similar to SGD with momentum.

Original: Moreover, AdaDQH can auto switch modes in a per parameter manner as training progresses.
Revised: Moreover, AdaDQH can auto switch modes in a per parameter manner as training progresses.

Original: Compared to the additive method, AdaDQH can eliminate the noise caused by (cid:15) in adaptive updates.
Revised: Compared to the additive method, AdaDQH can eliminate the noise caused by (cid:15) in adaptive updates.

Original: Another major beneﬁt is that AdaDQH has the ability to generalize in different tasks by tuning δ , without choosing from candidates of obscure optimizers empirically.
Revised: Another major beneﬁt is that AdaDQH has the ability to generalize in different tasks by tuning δ , without choosing from candidates of obscure optimizers empirically.

Original: The effect of δ is discussed in
Revised: The effect of δ is discussed in

Original: Sec.3.5, experimentally.
Revised: Sec.3.5, experimentally.

Original: 2. C ONVERGENCE ANALYSIS
Revised: 2. C ONVERGENCE ANALYSIS

Original: Using the framework developed in Reddi et al.
Revised: Using the framework developed in Reddi et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: ; Zhou et al.
Revised: ; Zhou et al.

Original: (2018), we have the following theorems that provide the convergence in convex and non-convex settings.
Revised: (2018), we have the following theorems that provide the convergence in convex and non-convex settings.

Original: Particularly, we use β 1 ,t to replace β 1 where β 1 ,t is non-increasing and lim t →∞ β 1 ,t = 0 .
Revised: Particularly, we use β 1 ,t to replace β 1 where β 1 ,t is non-increasing.

Original: Theorem 1.
Revised: Theorem 1.

Original: (Convergence in convex settings) Let { w t } be the sequence obtained by AdaDQH (Algorithm 1), α t = α/ √ t , β 1 ,t ≤ β 1 ∈ [0 , 1) , β 2 ∈ [0 , 1) , b t,i
Revised: (Convergence in convex settings) Let { w t } be the sequence obtained by AdaDQH (Algorithm 1), α t = α/ √ t , β 1 ,t ≤ β 1 ∈ [0 , 1) , β 2 ∈ [0 , 1) , b t,i

Original: ≤ b t
Revised: ≤ b t

Original: +1 ,i ∀ i ∈
Revised: +1 ,i ∀ i ∈

Original: [ n ] and (cid:107) g t (cid:107) ∞ ≤
Revised: [ n ] and (cid:107) g t (cid:107) ∞ ≤

Original: G ∞ , ∀ t ∈ [ T ] .
Revised: G ∞ , ∀ t ∈ [ T ] .

Original: Suppose f t ( w ) is convex for all t ∈
Revised: Suppose f t ( w ) is convex for all t ∈

Original: [ T ] , w ∗ is an optimal solution of (cid:80)
Revised: [ T ] , w ∗ is an optimal solution of (cid:80)

Original: Tt =1 f t ( w ) , i.e., w ∗ =
Revised: Tt =1 f t ( w ) , i [Equation]

Original:  arg min w ∈ R n (cid:80) Tt =1 f t ( w ) and there exists the constant D ∞ such that max t ∈
Revised: e., w ∗ = arg min w ∈ R n (cid:80) Tt =1 f t ( w ) and there exists the constant D ∞ such that max t ∈

Original: [ T ] (cid:107) w t − w ∗ (cid:107) ∞
Revised: [ T ] (cid:107) w t − w ∗ (cid:107) ∞

Original: ≤ D ∞ .
Revised: ≤ D ∞ .

Original: Then we have the following bound on the regret
Revised: Then we have the following bound on the regret

Original: T 
Revised: T (cid:88)

Original: [Equation]
Revised: [Equation]

Original: The proof of Theorem 1 is given in Appendix A.
Revised: The proof of Theorem 1 is given in Appendix A.

Original: Corollary 1.
Revised: Corollary 1.

Original: Suppose β 1 ,t = β 1 /t , then we have
Revised: Suppose β 1 ,t = β 1 /t , then we have

Original: [Equation]
Revised: [Equation]

Original: The proof of Corollary 1 is given in Appendix B. Corollary 1 implies the regret is O ( √ T ) and can achieve the convergence rate O (1 / √ T ) in convex settings.
Revised: The proof of Corollary 1 is given in Appendix B. Corollary 1 implies the regret is O ( √ T ) and can achieve the convergence rate O (1 / √ T ) in convex settings.

Original: Theorem 2.
Revised: Theorem 2.

Original: (Convergence in non-convex settings) Suppose that the following assumptions are satisﬁed:
Revised: (Convergence in non-convex settings) Suppose that the following assumptions are satisﬁed:

Original: f is differential and lower bounded, i.e., f ( w ∗ ) >
Revised: f is differential and lower bounded, i.e., f ( w ∗ ) >

Original: −∞ where w ∗ is an optimal solution.
Revised: −∞ where w ∗ is an optimal solution.

Original: f is also L -smooth, i.e., ∀ u, v ∈ R n , we have
Revised: f is also L -smooth, i e., ∀ u , v ∈ R n , we have

Original: [Equation]
Revised: [Equation]

Original: f ( v ) , u − v (cid:105) + L 2 (cid:107) u − v (cid:107) 2 . 2.

Original: At step t , the algorithm can access a bounded noisy gradient and the true gradient is bounded, i.e., (cid:107) g t (cid:107) ∞ ≤ G ∞ , (cid:107)∇ f ( w t ) (cid:107) ∞ ≤ G ∞ , ∀ t ∈
Revised: At step t , the algorithm can access a bounded noisy gradient and the true gradient is bounded, i.e., (cid:107) g t (cid:107) ∞ ≤ G ∞ , (cid:107)∇ f ( w t ) (cid:107) ∞ ≤ G ∞ , ∀ t ∈

Original: [ T ] .
Revised: [ T ] .

Original: Without loss of generality, we assume G ∞ ≥ δ .
Revised: Without loss of generality, we assume G ∞ ≥ δ .

Original: The noisy gradient is unbiased and the noise is independent, i.e., g t = ∇ f ( w t ) + ζ t , E [ ζ t ] =and ζ i is independent of ζ j if i (cid:54) =
Revised: The noisy gradient is unbiased and the noise is independent, i.e., g t = ∇ f ( w t ) + ζ t , E [ ζ t ] =and ζ i is independent of ζ j if i (cid:54) =

Original: j .4.
Revised: j .

Original: α t = α/ √ t , β 1 ,t ≤ β 1 ∈ [0 , 1) , β 2 ∈ [0 , 1) and b t,i ≤ b t +1 ,i ∀ i ∈ [ n ] .

Original: [Figure] [Figure]
Revised: [Equation] [Equation]

Original: [Figure] [Figure]
Revised: [Equation] n ] .

Original: Then Algorithm 1 yields
Revised: Then Algorithm 1 yields

Original: [Equation]
Revised: [Equation]

Original: where C 1 , C 2 and C 3 are deﬁned as follows:
Revised: where C 1 , C 2 and C 3 are deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: The proof of Theorem 2 is given in Appendix C. Note that we can let b t +1 = max( b t +1 , b t ) , which is usually called AMSGrad condition (Reddi et al., 2018), to make sure the assumption b t,i
Revised: The proof of Theorem 2 is given in Appendix C. Note that we can let b t +1 = max( b t +1 , b t ) , which is usually called AMSGrad condition (Reddi et al., 2018), to make sure the assumption b t,i

Original: ≤ b t
Revised: ≤ b t

Original: +1 ,i ∀ i ∈
Revised: +1 ,i ∀ i ∈

Original: [ n ] always true, though it could degenerate the effect in practice.
Revised: [ n ] always true, though it could degenerate the effect in practice.

Original: The more detailed analysis is given in Appendix E.3.
Revised: The more detailed analysis is given in Appendix E.3.

Revised: From Theorem 2, we have the following corollaries.

Original: Corollary 2.
Revised: Corollary 2.

Original: Suppose β 1 ,t = β 1 / √ t , then we have
Revised: Suppose β 1 ,t = β 1 / √ t , then we have

Original: [Equation]
Revised: [Equation]

Original: where C 4 and C  5 are deﬁned as follows:
Revised: where C 2 is the same with Theorem 2 and C 4 , C 5 are deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Revised: The proof of Corollary 2 is given in Appendix D.

Revised: Corollary 3.

Revised: Suppose β 1 ,t = β 1 , ∀ t ∈

Original: The proof of Corollary 2 is given in Appendix D. Corollary 2 implies the convergence (to the stationary point) rate for AdaDQH is O (log T/ √ T ) in non-convex settings.
Revised: [ T ] , then we have [Equation] where C 1 and C 2 are the same with Theorem 2. Corollaries 2, 3 imply the convergence (to the stationary point) rate for AdaDQH is O (log T/ √ T ) in non-convex settings.

Original: 2. N UMERICAL ANALYSIS
Revised: 2. N UMERICAL ANALYSIS

Original: In this section, we compare AdaDQH against several SOTA optimizers in three test funtions.
Revised: In this section, we compare AdaDQH against several SOTA optimizers in three test funtions.

Original: We adopt the parameter settings from Zhuang et al.
Revised: We adopt the parameter settings from Zhuang et al.

Original: The learning rate is set to 1e-3 for all adaptive optimizers, along with the same epsilon/delta (1e-8), betas ( β 1 = 0 .
Revised: The learning rate is set to 1e-3 for all adaptive optimizers, along with the same epsilon/delta (1e-8), betas ( β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: For SGD, momentum is set to 0.9 and learning rate is 1e-6 for numerical stability.
Revised: For SGD, momentum is set to 0.9 and learning rate is 1e-6 for numerical stability.

Original: AdaDQH shows promising results as it reaches the optimal points across all of the experiments, shown in Figure 1.
Revised: AdaDQH shows promising results as it reaches the optimal points across all of the experiments, shown in Figure 1.

Original: Furthermore, we search the best learing rate for each optimizer with regard to Beale function.
Revised: Furthermore, we search the best learing rate for each optimizer with regard to Beale function.

Original: AdaDQH is still the strongest competitor.
Revised: AdaDQH is still the strongest competitor.

Original: The details are provided in E.1.
Revised: The details are provided in E.1.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: OptimalAdaDQHSGD

Original: [Table]
Revised: + [Figure]

Revised: AdamAdaBeliefAdaHessian

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Original:  E XPERIMENTS 3. E XPERIMENT SETUP
Revised:  E XPERIMENTS 3. E XPERIMENT SETUP

Original: We experimentally compare the performance of different optimizers on a wide range of learning tasks, including CV, NLP and RecSys.
Revised: We experimentally compare the performance of different optimizers on a wide range of learning tasks, including CV, NLP and RecSys.

Original: The details of the tasks are as follows.
Revised: The details of the tasks are as follows.

Original: CV:

Original: We

Original: experiment

Original:  with ResNet and ResNet32 on Cifar10 (Krizhevsky et al., 2009) dataset, and
Revised: CV: We experiment with ResNet20 and ResNet32 on Cifar10 (Krizhevsky et al., 2009) dataset, and

Original: ResNet18 on ImageNet (Russakovsky et al., 2015) dataset.
Revised: ResNet18 on ImageNet (Russakovsky et al., 2015) dataset.

Original: The details of the datasets are listed in
Revised: The details of the datasets are listed in

Original: Table 2.
Revised: Table 2.

Original: We train 160 epochs and decay the learning rate by a factor of 10 at epoch 80 andfor Cifar10, and train 90 epochs and decay the learning rate by a factor of 10 every 30 epochs for
Revised: We train 160 epochs and decay the learning rate by a factor of 10 at epoch 80 andfor Cifar10, and train 90 epochs and decay the learning rate by a factor of 10 every 30 epochs for

Original: ImageNet
Revised: ImageNet.

Original: The batch size is 256 for both datasets.
Revised: The batch size is 256 for both datasets.

Original: NLP: We experiment with Neural

Original: Machine

Original: Translation (NMT

Original:  on IWSLT14 German-to-English (De-En) (Cettolo et al., 2014), and Language Modeling (LM) on Penn TreeBank (Marcus et al., 1993).
Revised: NLP: We experiment with Neural Machine Translation (NMT) on IWSLT14 German-to-English (De-En) (Cettolo et al., 2014), and Language Modeling (LM) on Penn TreeBank (Marcus et al., 1993).

Original: For NMT task, transformer small architecture is adopted.
Revised: For NMT task, transformer small architecture is adopted.

Original: We use the same setting and pre-processing method in Yao et al.
Revised: We use the same setting and pre-processing method in Yao et al.

Original: (2020), as well as the same length penalty (1.0), beam size (5) and max tokens (4096).
Revised: (2020), as well as the same length penalty (1.0), beam size (5) and max tokens (4096).

Original: We train 55 epochs and average the last 5 checkpoints for inference.
Revised: We train 55 epochs and average the last 5 checkpoints for inference.

Original: For LM task, we train 1,2,3-layer LSTM with batch size of 20 for 200 epochs.
Revised: For LM task, we train 1,2,3-layer LSTM with batch size of 20 for 200 epochs.

Original: The details are listed in Table 2.
Revised: The details are listed in Table 2.

Original: Additionally, we keep settings like learning rate scheduler and warm-up steps identical for the same task.
Revised: Additionally, we keep settings like learning rate scheduler and warm-up steps identical for the same task.

Original: RecSys: We experiment on two common datasets including Avazu (Avazu, 2015) and Criteo (Criteo, 2014) which are various display ads logs for the purpose of predicting the Click Through Rate (CTR).
Revised: RecSys: We experiment on two common datasets including Avazu (Avazu, 2015) and Criteo (Criteo, 2014) which are various display ads logs for the purpose of predicting the Click Through Rate (CTR).

Original: For Avazu, the samples from the ﬁrst nine days are used for training, while the rest are for testing.
Revised: For Avazu, the samples from the ﬁrst nine days are used for training, while the rest are for testing.

Original: We use the basic Multilayer Perceptron (MLP) structure of most deep CTR models.
Revised: We use the basic Multilayer Perceptron (MLP) structure of most deep CTR models.

Original: Speciﬁcally, the model maps each categorical feature as a 16-dimensional embedding vector, following up with 4 fully connected layer of dimension in 64,32,16,1, respectively.
Revised: Speciﬁcally, the model maps each categorical feature as a 16-dimensional embedding vector, following up with 4 fully connected layer of dimension in 64,32,16,1, respectively.

Original: For Criteo, we take the early 6/ part of all samples as the train set.
Revised: For Criteo, we take the early 6/ part of all samples as the train set.

Original: We adopt Deep & Cross Network (DCN) (Wang et al., 2017) with embedding size set to 8, along with 2 deep layers of size 64 and 2 cross layers.
Revised: We adopt Deep & Cross Network (DCN) (Wang et al., 2017) with embedding size set to 8, along with 2 deep layers of size 64 and 2 cross layers.

Original: The details are listed in Table 2.
Revised: The details are listed in Table 2.

Original: We train 1 epoch with a batch size of 512 for both datasets.
Revised: We train 1 epoch with a batch size of 512 for both datasets.

Original: Optimizers to be compared include SGD (Robbins & Monro, 1951), Adam (Kingma & Ba, 2015),
Revised: Optimizers to be compared include SGD (Robbins & Monro, 1951), Adam (Kingma & Ba, 2015),

Original: AdamW (Loshchilov & Hutter, 2019), AdaBelief (Zhuang et al., 2020) and AdaHessian (Yao et al., 2020).
Revised: AdamW (Loshchilov & Hutter, 2019), AdaBelief (Zhuang et al., 2020) and AdaHessian (Yao et al., 2020).

Original: The choices of the hyperparameters are given in Appendix E.2.
Revised: The choices of the hyperparameters are given in Appendix E.2.

Original: The experiments of CV and
Revised: The experiments of CV and

Revised: [Table]

Original:  NLP are conducted with GPUs in the PyTorch framework (Paszke et al., 2019), and the experiments of RecSys are conducted with 3 parameter servers and 5 workers in the TensorFlow framework (Abadi et al., 2016).
Revised: [Table] NLP are conducted with GPUs in the PyTorch framework (Paszke et al., 2019) [Table] and the experiments of RecSys are conducted with 3 parameter servers and 5 workers in the TensorFlow framework (Abadi et al., 2016).

Original: We run all the experiments 5 times with random seeds and calculate the statistical results.
Revised: We run all the experiments 5 times with random seeds and calculate the statistical results.

Original: 3. CV
Revised: 3. CV

Original: Table 3 reports the top-1 accuracy for different optimizers when trained on Cifar10 and ImageNet.
Revised: Table 3 reports the top-1 accuracy for different optimizers when trained on Cifar10 and ImageNet.

Original: It is remarkable that AdaDQH outperforms other optimizers on both Cifar10 and ImageNet.
Revised: It is remarkable that AdaDQH outperforms other optimizers on both Cifar10 and ImageNet.

Original: The testing accuracy ( [ µ ± σ ] ) curves of different optimizers for ResNet20/32 on Cifar10 and ResNet on ImageNet are plotted in Figure 2.
Revised: The testing accuracy ( [ µ ± σ ] ) curves of different optimizers for ResNet20/32 on Cifar10 and ResNet on ImageNet are plotted in Figure 2.

Original: Note that the results of SGD, AdaBelief and AdaHessian on
Revised: Note that the results of SGD, AdaBelief and AdaHessian on

Original: ImageNet are lower than the number reported in original papers (Chen et al., 2020; Zhuang et al., 2020; Yao et al., 2020) which are run single time.
Revised: ImageNet are lower than the number reported in original papers (Chen et al., 2020; Zhuang et al., 2020; Yao et al., 2020) which are run single time.

Original: More discussions are given in Appendix E.4.
Revised: More discussions are given in Appendix E.4.

Original: We also report the accuracy of AdaDQH for ResNet18 on Cifar10 for comparing with the SOTA[Table] [Table] results 1 , which is listed in Appendix E.5.
Revised: We also report the accuracy of AdaDQH for ResNet18 on Cifar10 for comparing with the SOTA results 1 , which is listed in Appendix E.5.

Original: In addition, it is worth mentioning that although AdaDQH is considered as quasi-Hessian, its runtime cost is comparable with SGD and much lower than
Revised: In addition, it is worth mentioning that although AdaDQH is considered as quasi-Hessian, its runtime cost is comparable with SGD and much lower than

Original: AdaHessian, which is shown in Table 4.
Revised: AdaHessian, which is shown in Table 4.

Original: 3. NLP
Revised: 3. NLP

Original: We report case-insensitive BiLingual Evaluation Understudy (BLEU, higher is better) score and the perplexity (PPL, lower is better) on test set for NMT and LM tasks respectively.
Revised: We report case-insensitive BiLingual Evaluation Understudy (BLEU, higher is better) score and the perplexity (PPL, lower is better) on test set for NMT and LM tasks respectively.

Original: The results are shown in Table 5.
Revised: The results are shown in Table 5.

Original: For the NMT task on IWSLT14, AdaDQH achieves a similar result to AdaBelief, outperforming the other optimizers.
Revised: For the NMT task on IWSLT14, AdaDQH achieves a similar result to AdaBelief, outperforming the other optimizers.

Original: For LM task on PTB, AdaDQH obtains lowest PPL in all 1,2,3-layer LSTM experiments, as demonstrated in Figure 3.
Revised: For LM task on PTB, AdaDQH obtains lowest PPL in all 1,2,3-layer LSTM experiments, as demonstrated in Figure 3.

Original: Furthermore, we report the relative training time in Table 4 which is similar to CV.
Revised: Furthermore, we report the relative training time in Table 4 which is similar to CV.

Original: 3. R EC S YS
Revised: 3. R EC S YS

Original: We adopt Area Under the receiver-operator Curve (AUC) as the evaluation criterion which is a good measurement in CTR estimation (Graepel et al., 2010).
Revised: We adopt Area Under the receiver-operator Curve (AUC) as the evaluation criterion which is a good measurement in CTR estimation (Graepel et al., 2010).

Original: Table 6 shows that compared to other optimizers, AdaDQH can achieve signiﬁcantly better or highly competitive performance on the AUC metric.
Revised: Table 6 shows that compared to other [Table] [Figure] [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Revised: [Table]

Revised: optimizers, AdaDQH can achieve signiﬁcantly better or highly competitive performance on the AUC metric.

Original: 3. T HE EFFECT OF δ
Revised: 3. T HE EFFECT OF δ

Original: In this section, we analyze the rigorous effect of δ , i.e., what exact percentage of ˆ b t is truncated by δ in Algorithm 1.
Revised: In this section, we analyze the rigorous effect of δ , i.e., what exact percentage of ˆ b t is truncated by δ in Algorithm 1.

Original: Figure 4 depicts the distribution of ˆ b t during the training process on different tasks in the best conﬁguration we found.
Revised: Figure 4 depicts the distribution of ˆ b t during the training process on different tasks in the best conﬁguration we found.

Original: The black dot gives the exact percentage of ˆ b t that is truncated by δ in the task.
Revised: The black dot gives the exact percentage of ˆ b t that is truncated by δ in the task.

Original: Lower percentage means more SGD-like updates than adaptive steps, which is controlled by the choice of δ .
Revised: Lower percentage means more SGD-like updates than adaptive steps, which is controlled by the choice of δ .

Original: Figure 4 reveals how auto switch in AdaDQH works in different tasks.
Revised: Figure 4 reveals how auto switch in AdaDQH works in different tasks.

Original: As shown in Figure 4a,
Revised: As shown in Figure 4a,

Original: AdaDQH behaves more like SGD in early stage of training (before the ﬁrst learning rate decay at the 30th epoch) and switches to the adaptive for ﬁne-tuning, since SGD generally outperforms the adaptive optimizers in CNN tasks.Figure 4b indicates that the parameters taking adaptive updates [Figure] [Figure]
Revised: AdaDQH behaves more like SGD in early stage of training (before the ﬁrst learning rate decay at the 30th epoch) and switches to the adaptive for ﬁne-tuning, since SGD generally outperforms the adaptive optimizers in CNN tasks.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure] [Table] [Table] are dominant, which is expected because the adaptive optimizers like AdamW are preferred in transformer.
Revised: Figure 4b indicates that the parameters taking adaptive updates are dominant, which is expected because the adaptive optimizers like AdamW are preferred in transformer.

Original: As indicated in Figure 4c, most parameters are updating stochastically, which explains why AdaDQH has a similar curve to SGD in Figure 3b before the 100th epoch.
Revised: As indicated in Figure 4c, most parameters are updating stochastically, which explains why AdaDQH has a similar curve to SGD in Figure 3b before the 100th epoch.

Original: The ratio grows from 3% to 5% afterwards, resulting in a better PPL in the ﬁne-tuning stage.
Revised: The ratio grows from 3% to 5% afterwards, resulting in a better PPL in the ﬁne-tuning stage.

Original: As for Figure 4d, the model of the RecSys task is training for only one epoch, and AdaDQH gradually switch to the adaptive updates for a better ﬁt to the data.
Revised: As for Figure 4d, the model of the RecSys task is training for only one epoch, and AdaDQH gradually switch to the adaptive updates for a better ﬁt to the data.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we propose the AdaDQH optimizer, which can evolve from stochastic to adaptive by auto switch of the precondition matrix and has better generalization compared to the SOTA optimizers.
Revised: In this paper, we propose the AdaDQH optimizer, which can evolve from stochastic to adaptive by auto switch of the precondition matrix and has better generalization compared to the SOTA optimizers.

Original: We theoretically prove the convergence rate in both convex and non-convex stochastic settings and conduct empirical evaluation in real-world datasets of different scenes.
Revised: We theoretically prove the convergence rate in both convex and non-convex stochastic settings and conduct empirical evaluation in real-world datasets of different scenes.

Original: The results clearly demonstrate the advantages of our optimizer in getting signiﬁcantly better performance.
Revised: The results clearly demonstrate the advantages of our optimizer in getting signiﬁcantly better performance.

Original: Finally, we analyze how it is able to auto switch from stochastic to adaptive, and the rigorous effect of the hyperparameter δ which controls the auto-switch process
Revised: Finally, we analyze how it is able to auto switch from stochastic to adaptive, and the rigorous effect of the hyperparameter δ which controls the auto-switch process.