Original: O PTIMISTIC A CCELERATION FOR O PTIMIZATION
Revised: O PTIMISTIC A CCELERATION FOR O PTIMIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We consider new variants of optimization algorithms.
Revised: We consider new variants of optimization algorithms.

Original: Our algorithms are based on the observation that mini-batch of stochastic gradients in consecutive iterations do not change drastically and consequently may be predictable.
Revised: Our algorithms are based on the observation that mini-batch of stochastic gradients in consecutive iterations do not change drastically and consequently may be predictable.

Original: Inspired by the similar setting in online learning literature called Optimistic Online learning , we propose two new algorithms, O PTIMISTIC
Revised: Inspired by the similar setting in online learning literature called O PTIMISTIC O NLINE LEARNING , we propose two new algorithms, O PTIMISTIC

Original: -AMSG RAD and O PTIMISTIC -A DAM that exploit the predictability of gradients.
Revised: -AMSG RAD and O PTIMISTIC -A DAM that exploit the predictability of gradients.

Original: O PTIMISTIC
Revised: O PTIMISTIC

Original: -AMSG RAD and
Revised: -AMSG RAD and

Original: O PTIMISTIC -A DAM combine the idea of momentum method, adaptive gradient method, and algorithms in Optimistic Online learning , which leads to speed up in training deep neural nets in practice.
Revised: O PTIMISTIC -A DAM combine the idea of momentum method, adaptive gradient method, and algorithms in O PTIMISTIC O NLINE LEARNING , which leads to speed up in training deep neural nets in practice.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Nowadays deep learning has been shown to be very effective in several tasks, from robotics (e.g. Levine et al.
Revised: Nowadays deep learning has been shown to be very effective in several tasks, from robotics (e.g. Levine et al.

Original: computer vision (e.g. He et al.
Revised: computer vision (e.g. He et al.

Original: ; Goodfellow et al.
Revised: ; Goodfellow et al.

Original: , reinforcement learning (e.g. Mnih et al.
Revised: , reinforcement learning (e.g. Mnih et al.

Original: (2013), to natural language processing (e.g. Graves et al.
Revised: (2013), to natural language processing (e.g. Graves et al.

Original: Typically, the model parameters of a state-of-the-art deep neural net is very high-dimensional and the required training data is also in huge size.
Revised: Typically, the model parameters of a state-of-the-art deep neural net is very high-dimensional and the required training data is also in huge size.

Original: Therefore, fast algorithms are necessary for training a deep neural net.
Revised: Therefore, fast algorithms are necessary for training a deep neural net.

Original: To achieve this, there are number of algorithms proposed in recent years, such as AMSG RAD (Reddi et al.
Revised: To achieve this, there are number of algorithms proposed in recent years, such as AMSG RAD (Reddi et al.

Original: , ADAM (Kingma & Ba (2015)), RMSPROP (Tieleman & Hinton (2012)),
Revised: , ADAM (Kingma & Ba (2015)), RMSPROP (Tieleman & Hinton (2012)),

Original: ADADELTA (Zeiler (2012)), and NADAM (Dozat (2016)), etc.
Revised: ADADELTA (Zeiler (2012)), and NADAM (Dozat (2016)), etc.

Original: All the prevalent algorithms for training deep nets mentioned above combines two ideas: the idea of adaptivity in
Revised: All the prevalent algorithms for training deep nets mentioned above combines two ideas: the idea of adaptivity in

Original: A DA G RAD (Duchi et al.
Revised: A DA G RAD (Duchi et al.

Original: ; McMahan & Streeter (2010)) and the idea of momentum as N ESTEROV ’ S M ETHOD (Nesterov (2004)) or the H EAVY BALL method (Polyak (1964)).
Revised: ; McMahan & Streeter (2010)) and the idea of momentum as N ESTEROV ’ S M ETHOD (Nesterov (2004)) or the H EAVY BALL method (Polyak (1964)).

Original: A DA G RAD is an online learning algorithm that works well compared to the standard online gradient descent when the gradient is sparse.
Revised: A DA G RAD is an online learning algorithm that works well compared to the standard online gradient descent when the gradient is sparse.

Original: The update of A DA G RAD has a notable feature: the learning rate is different for different dimensions, depending on the magnitude of gradient in each dimension, which exploits the geometry of data and performs informative update.
Revised: The update of A DA G RAD has a notable feature: the learning rate is different for different dimensions, depending on the magnitude of gradient in each dimension, which might help in exploiting the geometry of data and leading to a better update.

Original: On the other hand, N ESTEROV ’ S M ETHOD or the Momentum Method (Polyak (1964)) is an accelerated optimization algorithm whose update not only depends on the current iterate and current gradient but also depends on the past gradients (i.e. momentum).
Revised: On the other hand, N ESTEROV ’ S M ETHOD or the Momentum Method (Polyak (1964)) is an accelerated optimization algorithm whose update not only depends on the current iterate and current gradient but also depends on the past gradients (i.e. momentum).

Original: State-of-the-art algorithms like AMSG RAD (Reddi et al.
Revised: State-of-the-art algorithms like AMSG RAD (Reddi et al.

Original: and ADAM (Kingma & Ba (2015)) leverages these two ideas to get fast training for neural nets.
Revised: and ADAM (Kingma & Ba (2015)) leverages these two ideas to get fast training for neural nets.

Original: In this paper, we propose an algorithm that goes further than the hybrid of the adaptivity and momentum approach.
Revised: In this paper, we propose an algorithm that goes further than the hybrid of the adaptivity and momentum approach.

Original: Our algorithm is inspired by Optimistic Online learning (Chiang et al.
Revised: Our algorithm is inspired by O PTIMISTIC O NLINE LEARNING (Chiang et al.

Original: ; Rakhlin & Sridharan (2013); Syrgkanis et al. (2015); Abernethy et al.
Revised: ; Rakhlin & Sridharan (2013); Syrgkanis et al. ; Abernethy et al.

Original: Optimistic Online learning considers that a good guess of the loss function in the current round of online learning is available and plays an action by exploiting the good guess.
Revised: O PTIMISTIC O NLINE LEARNING considers that a good guess of the loss function in the current round of online learning is available and plays an action by exploiting the good guess.

Original: By exploiting the guess, those algorithms in Optimistic Online learning have regret in the form of O ( (cid:113)(cid:80)
Revised: By exploiting the guess, those algorithms in O PTIMISTIC O NLINE LEARNING have regret in the form of O ( (cid:113)(cid:80)

Original: Tt =1 (cid:107) g t − m t (cid:107) ) , where g t is the gradient of loss function in round t and m t is the ”guess” of g t before seeing the loss function in round t (i.e. before getting g t ).
Revised: Tt =1 (cid:107) g t − m t (cid:107) ) , where g t is the gradient of loss function in round t and m t is the ”guess” of g t before seeing the loss function in round t (i.e. before getting g t ).

Original: This kind of regret can be much smaller than O ( √ T ) when one has a good guess m t of g t .
Revised: This kind of regret can be much smaller than O ( √ T ) when one has a good guess m t of g t .

Original: We combinethe
Revised: We combine

Original: Optimistic Online learning idea with the adaptivity and the momentum ideas to design new algorithms in training deep neural nets, which leads to O PTIMISTIC
Revised: the O PTIMISTIC O NLINE LEARNING idea with the adaptivity and the momentum ideas to design new algorithms in training deep neural nets, which leads to O PTIMISTIC

Original: -AMSG RAD and O PTIMISTIC -A DAM .
Revised: -AMSG RAD and O PTIMISTIC -A DAM .

Original: We also provide theoretical analysis of O PTIMISTIC
Revised: We also provide theoretical analysis of O PTIMISTIC

Original: -AMSG RAD.
Revised: -AMSG RAD

Original: The proposed O PTIMISTIC - algorithms not only adapt to the informative dimensions and exhibit momentums but also take advantage of a good guess of the next gradient to facilitate acceleration.
Revised: The proposed O PTIMISTICalgorithms not only adapt to the informative dimensions and exhibit momentums but also take advantage of a good guess of the next gradient to facilitate acceleration.

Original: We evaluate our algorithms with (Kingma & Ba (2015)), (Reddi et al.
Revised: We evaluate our algorithms with (Kingma & Ba (2015)), (Reddi et al.

Original: and (Daskalakis et al.
Revised: and (Daskalakis et al.

Original: Experiments show that our O PTIMISTIC -algorithms are faster than the baselines.
Revised: Experiments show that our O PTIMISTIC -algorithms are faster than the baselines.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Both AMSG RAD (Reddi et al.
Revised: Both AMSG RAD (Reddi et al.

Original: and ADAM (Kingma & Ba (2015)) are actually Online learning algorithms.
Revised: and ADAM (Kingma & Ba (2015)) are actually O NLINE LEARNING algorithms.

Original: Their theoretical analysis are the regret analysis in online learning.
Revised: They use REGRET ANALYSIS to provide some theoretical guarantees of the algorithms.

Original: Since one can convert an online learning algorithm to an ofﬂine optimization algorithm by online-to-batch conversion (Cesa-Bianchi et al.
Revised: Since one can convert an online learning algorithm to an ofﬂine optimization algorithm by online-to-batch conversion (Cesa-Bianchi et al.

Original: , one can design an ofﬂine optimization algorithm by designing and analyzing its counterpart in online learning.
Revised: (2004)), one can design an ofﬂine optimization algorithm by designing and analyzing its counterpart in online learning.

Original: Therefore, we would like to give a brief review of Online learning and Optimistic-Online learning .
Revised: Therefore, we would like to give a brief review of O NLINE LEARNING and O PTIMISTIC -O NLINE LEARNING .

Original: 2. B RIEF REVIEW OF ONLINE LEARNING
Revised: 2. B RIEF REVIEW OF ONLINE LEARNING

Original: In the typical setting of online learning, there is a learner playing an action and then receiving a loss function in each round t .
Revised: In the typical setting of online learning, there is a LEARNER playing an action and then receiving a loss function in each round t .

Original: Speciﬁcally, the learner plays an action w t ∈ K in round t , where w t is chosen in a compact and convex set K ⊆ R n , known as the decision space .
Revised: Speciﬁcally, the learner plays an action w t ∈ K in round t , where w t is chosen in a compact and convex set K ⊆ R n , known as the DECISION SPACE .

Original: Then, the learner sees the loss function (cid:96) t ( · ) and suffers loss (cid:96) t ( w t ) for the choice.
Revised: Then, the learner sees the LOSS FUNCTION (cid:96) t ( · ) and suffers loss (cid:96) t ( w t ) for the choice.

Original: No distributional assumption is made on the loss functions sequence { (cid:96) 1 ( · ) , (cid:96) 2 ( · ) , . . .
Revised: No distributional assumption is made on the loss functions sequence { (cid:96) 1 ( · ) , (cid:96) 2 ( · ) , . . .

Original: , (cid:96) T ( · ) } in online learning .
Revised: , (cid:96) T ( · ) } in ONLINE LEARNING .

Original: Namely, the loss functions can be adversarial.
Revised: Namely, the loss functions can be adversarial.

Original: The goal of an online learner is minimizing its regret , which is
Revised: The goal of an online learner is minimizing its REGRET , which is

Original: [Equation]
Revised: [Equation]

Original: We can also deﬁne average regret as Regret T :=
Revised: We can also deﬁne AVERAGE REGRET as Regret T :=

Original: Regret T T , which is regret divided by number of rounds T .
Revised: Regret T T , which is REGRET divided by number of rounds T .

Original: In Online learning literature, No-regret algorithms means online learning algorithms satisfying Regret T → 0 as T → ∞ .
Revised: In O NLINE LEARNING literature, N O - REGRET ALGORITHMS means online learning algorithms satisfying Regret T →as T → ∞ .

Original: 2. B RIEF REVIEW OF O PTIMISTIC O NLINE LEARNING
Revised: 2. B RIEF REVIEW OF O PTIMISTIC O NLINE LEARNING

Original: In recent years, there is a branch of works in the paradigm of Optimistic Online learning (e.g. Chiang et al.
Revised: In recent years, there is a branch of works in the paradigm of O PTIMISTIC O NLINE LEARNING (e.g. Chiang et al.

Original:  Rakhlin & Sridharan (2013); Syrgkanis et al.
Revised: (2012); Rakhlin & Sridharan (2013); Syrgkanis et al.

Original: ; Abernethy et al.
Revised: ; Abernethy et al.

Original: The idea of Optimistic Online learning is as follows.
Revised: The idea of O PTIMISTIC O NLINE LEARNING is as follows.

Original: Suppose that, in each round t , the learner has a good guess m t ( · ) of the loss function (cid:96) t ( · ) before playing an action w t .
Revised: Suppose that, in each round t , the learner has a good guess m t ( · ) of the loss function (cid:96) t ( · ) before playing an action w t .

Original: (Recall that the learner receives the loss function after the learner commits an action!)
Revised: (Recall that the learner receives the loss function after the learner commits an action!)

Original: Then, the learner should exploit the guess m t ( · ) to choose an action w t , as m t ( · ) is close to the true loss function (cid:96) t ( · ) .
Revised: Then, the learner should exploit the guess m t ( · ) to choose an action w t , as m t ( · ) is close to the true loss function (cid:96) t ( · ) .

Original: For example, (Syrgkanis et al.
Revised: For example, (Syrgkanis et al.

Original: (2015)) proposed an optimistic-variant of the so called F OLLOW - THE -R EGULARIZEDL EADER (FTRL).
Revised: (2015)) proposed an optimistic-variant of the so called F OLLOW - THE -R EGULARIZEDL EADER (FTRL).

Original: The update rule of FTRL (see e.g. Hazan (2016)) is
Revised: The update rule of FTRL (see e.g. Hazan (2016)) is

Original: [Equation]
Revised: [Equation]

Original:  where R ( · ) is a 1 -strong convex function with respect to a norm ( (cid:107) · (cid:107) ) on the constraint set K , L t − 1 := (cid:80) t − 1 s =1 g s is the cumulative sum of gradient vectors of the  loss functions (i.e. g s :=
Revised: [Equation] ) where R ( · ) is a 1 -strong convex function with respect to a norm ( (cid:107) · (cid:107) ) on the constraint set K , L t − 1 := (cid:80) t − 1 s =1 g s is the cumulative sum of gradient vectors of the convex loss functions (i.e. g s :=

Original: ∇ (cid:96) s ( w s ) )
Revised: ∇ (cid:96) s ( w s ) )

Original: up to but not including t , and η is a parameter.
Revised: up to but not including t , and η is a parameter.

Original: The O PTIMISTIC -FTRL of (Syrgkanis et al.
Revised: The O PTIMISTIC -FTRL of (Syrgkanis et al.

Original: has update
Revised: has update

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where m t is the learner’s guess of the gradient vector g t := ∇ (cid:96) t ( w t ) .
Revised: ) where m t is the learner’s guess of the gradient vector g t : = ∇ (cid:96) t ( w t ) .

Original: Under the assumption that loss functions are convex, the regret of O PTIMISTIC -FTRL satisﬁes Regret T ≤ O ( (cid:113)(cid:80)
Revised: Under the assumption that loss functions are convex, the regret of O PTIMISTIC -FTRL satisﬁes Regret T ≤ O ( (cid:113)(cid:80)

Original: Tt =1 (cid:107) g t − m t (cid:107) ∗ ) , which can be much smaller than  √ T  of FTRL if one has a good guess.
Revised: Tt =1 (cid:107) g t − m t (cid:107) ∗ ) , which can be much smaller than O ( √ T ) of FTRL if m t is close to g t .

Original: That is, if m t is close to g t , then O PTIMISTIC -FTRL will have much smaller regret thanFTRL.
Revised: Consequently, O PTIMISTIC -FTRL will have much smaller regret than

Original: The concern is that how to get good m t .
Revised: FTRL.

Original: Apparently, if m t is far from g t , then the regret of O PTIMISTIC -FTRL would be a constant factor worse than that of FTRL without optimistic update.
Revised: On the other hand, if m t is far from g t , then the regret of O PTIMISTIC -FTRL would be a constant factor worse than that of FTRL without optimistic update.

Original: In the later section, we’ll respond to the concern and provide a way to get m t .
Revised: In the later section, we provide a way to get m t .

Original: Here, we just want to emphasize the importance of leveraging a good guess m t for updating w t to get a fast convergence rate (or equivalently, small regret).
Revised: Here, we just want to emphasize the importance of leveraging a good guess m t for updating w t to get a fast convergence rate (or equivalently, small regret).

Original: We also note that the works of Optimistic Online learning Chiang et al.
Revised: We also note that the works of O PTIMISTIC O NLINE LEARNING Chiang et al.

Original: ; Rakhlin & Sridharan (2013); Syrgkanis et al.
Revised: Rakhlin & Sridharan (2013); Syrgkanis et al.

Original: (2015)) has been shown to accelerate the convergence of some zero-sum games.
Revised: (2015)) has been shown to accelerate the convergence of some zero-sum games.

Revised: 1: Required: do 4:

Revised: Get mini-batch stochastic gradient vector g t ∈ R d at w t .

Revised: 5: θ t = β 1 θ t − 1 + (1 − β 1 ) g t .

Original: 2. A DAM AND AMSG RAD
Revised: 6: v t = β 2 v t − 1 + (1 − β 2 ) g 2 t . 7: ˆ v t = max(ˆ v t − 1 , v t ) . 8: w t +1 =

Revised: w t − η t θ t √ ˆ v t .

Original:  A DAM (Kingma & Ba (2015)) is a very popular algorithm for training deep nets.
Revised: 9: end A DAM AND AMSG RAD A DAM (Kingma & Ba (2015)) is a very popular algorithm for training deep nets.

Original: It combines the  idea of A DA G RAD (Duchi et al.
Revised: It combines the momentum idea Polyak (1964) with the idea of A DA G RAD (Duchi et al.

Original: (2011)), which has individual learning rate for different dimensions.
Revised: (2011)), which has individual learning rate for different dimensions.

Original: The learning rate of A DA G RAD in iteration t for a dimension j is proportional to the inverse of (cid:112) Σ t s =1 g s
Revised: The learning rate of A DA G RAD in iteration t for a dimension j is proportional to the inverse of (cid:112) Σ ts =

Original:  [ j ] 2 , where g s [ j ] is the j th element of the gradient vector g s in time s .
Revised: 1 g s [ j ] 2 , where g s [ j ] is the j th element of the gradient vector g s in time s .

Original: This adaptive learning rate may help for accelerating the convergence when the gradient vector is sparse (Duchi et al.
Revised: This adaptive learning rate may help for accelerating the convergence when the gradient vector is sparse (Duchi et al.

Original: However, when applying A DA G RAD to train deep nets, it is observed that the learning rate might decay too fast (Kingma & Ba (2015)).
Revised: However, when applying A DA G RAD to train deep nets, it is observed that the learning rate might decay too fast (Kingma & Ba (2015)).

Original: Therefore, (Kingma & Ba (2015)) proposes using a moving average of gradients (element-wise) divided by the root of the second moment of the moving average to update model parameter w (i.e. line 5,6 and line 8 of Algorithm 1).
Revised: Therefore, (Kingma & Ba (2015)) proposes using a moving average of gradients (element-wise) divided by the root of the second moment of the moving average to update model parameter w (i.e. line 5,6 and line 8 of Algorithm 1).

Original: Yet, A DAM (Kingma & Ba (2015)) does not converge to some speciﬁc convex functions.
Revised: Yet, A DAM (Kingma & Ba (2015)) fails at some online convex optmization problems.

Original: AMSG RAD (Reddi et al.
Revised: AMSG RAD (Reddi et al.

Original: (2018)) ﬁxes the issue.
Revised: (2018)) ﬁxes the issue.

Original: The algorithm of AMSG RAD is shown in Algorithm 1.
Revised: The algorithm of AMSG RAD is shown in Algorithm 1.

Original: The difference between A DAM and AMSG RAD lies on line 7 of Algorithm 1.
Revised: The difference between A DAM and AMSG RAD lies on line 7 of Algorithm 1.

Original: A DAM does not have the update of line 7.
Revised: A DAM does not have the update of line 7.

Original: (Reddi et al.
Revised: (Reddi et al.

Original: (2018)) adds the step to guarantee a non-increasing learning rate, η t √ ˆ v t .
Revised: (2018)) adds the step to guarantee a nonincreasing learning rate, η t √ ˆ v t .

Original: which helps for the convergence (i.e. average regret Regret T → 0 .)
Revised: which helps for the convergence (i.e. average regret Regret T → 0 .)

Original: For the parameters of AMSG RAD , it is suggested that β 1 = 0 .
Revised: For the parameters of AMSG RAD , it is suggested that β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 99 and η t = η/ √ t for a number η .
Revised: 99 and η t = η/ √ t for a number η .

Original:  O PTIMISTIC -AMSG RAD
Revised:  O PTIMISTIC -AMSG RAD

Original: In this section, we propose a new algorithm for training deep nets: O PTIMISTIC -AMSG RAD , shown in Algorithm 2.
Revised: In this section, we propose a new algorithm for training deep nets: O PTIMISTIC -AMSG RAD , shown in Algorithm 2.

Original: O PTIMISTIC -AMSG RAD has an optimistic update, which is line 9 of Algorithm 2.
Revised: O PTIMISTIC -AMSG RAD has an optimistic update, which is line 9 of Algorithm 2.

Original: It exploits the guess m t +1 of g t +1 to get w t +1 , since the vector h t +1 uses m t +1 .
Revised: It exploits the guess m t +1 of g t + to get w t +1 , since the vector h t +1 uses m t +1 .

Original: Notice that the gradient vector is computed at w t instead of w t − 12 and the moving average of gradients is used to update w t
Revised: Notice that the gradient vector is computed at w t instead of w t − 12 and the moving average of gradients is used to update w t

Original: + 12 .
Revised: + 12 .

Revised: One might want to combining line 8 and line 9 and getting a single line: w t +1 =

Revised: w t − 12 − η t θ t √ ˆ v t − η t +1 4 1 − β 1 h t +1 √ ˆ v t .

Original: Also, w t + 12 is updated from w t − 12 instead of w t .
Revised: From this, we see that w t +1 is updated from w t − 12 instead of w t .

Original: Therefore, while O PTIMISTIC -AMSG RAD looks like  doing an additionaloptimistic update compared to AMSG RAD , the difference of update is subtle.
Revised: Therefore, while O PTIMISTIC -AMSG RAD looks like just doing an additional update compared to AMSG RAD , the difference of update is subtle.

Original: We also want to emphasize that although the learning rate on line 9 contains 4 1 − β 1 factor.
Revised: We also want to emphasize that although the learning rate on line 9 contains 4 1 − β 1 factor.

Original: We suspect that it is due to the artifact of our theoretical analysis.
Revised: We suspect that it is due to the artifact of our theoretical analysis.

Original: In our experiments, the learning rate on line 9 does not have the factor of 4 1 − β 1 .
Revised: In our experiments, the learning rate on line 9 does not have the factor of 4 1 − β 1 .

Original: That is, in practice, we implement line 9 as w t
Revised: That is, in practice, we implement line 9 as w t

Original: +1 =
Revised: +1 =

Original: w t + 12 − η t
Revised: w t + 12 − η t

Original: +1 h t +1 √ ˆ v t .
Revised: +1 h t +1 √ ˆ v t .

Original: We leave closing the gap between theory and practice as the future work.
Revised: We leave closing the gap between theory and practice as a future work.

Original: We see that O PTIMISTIC
Revised: We see that O PTIMISTIC

Original: -AMSG RAD inherits three properties
Revised: -AMSG RAD inherits three properties

Original: • Adaptive learning rate of each dimension as A DA G RAD (Duchi et al.
Revised: • Adaptive learning rate of each dimension as A DA G RAD (Duchi et al.

Original: (line 8) • Exponentially moving average of the past gradients as N ESTROV ’ S METHOD (Nesterov (2004)) and the H EAVY -B
Revised: (line 8) • Exponentially moving average of the past gradients as N ESTEROV ’ S METHOD (Nesterov (2004)) and the H EAVY -B

Original: ALL method (Polyak (1964)).
Revised: ALL method (Polyak (1964)).

Original: (line 5) • Optimistic update that exploits a good guess of the next gradient vector as optimistic online learning algorithms (Chiang et al.
Revised: (line 5) • Optimistic update that exploits a good guess of the next gradient vector as optimistic online learning algorithms (Chiang et al.

Original: ; Rakhlin & Sridharan (2013); Syrgkanis et al. (2015); Abernethy et al.
Revised: ; Rakhlin & Sridharan (2013); Syrgkanis et al. (2015); Abernethy et al.

Original: (line 9)
Revised: (line 9)

Original: The ﬁrst property helps acceleration when the gradient has sparse structure.
Revised: The ﬁrst property helps acceleration when the gradient has sparse structure.

Original: The second one is the well-recognized idea of momentum which can achieve acceleration.
Revised: The second one is the well-recognized idea of momentum which can achieve acceleration.

Original: The last one, perhaps less known outside the Online learning community, can actually achieve acceleration when the prediction of the next gradient is good.
Revised: The last one, perhaps less known outside the O NLINE LEARNING community, can actually achieve acceleration when the prediction of the next gradient is good.

Original: We are going to elaborate this property in the later section where we give the theoretical analysis of O PTIMISTIC
Revised: We are going to elaborate this property in the later section where we give the theoretical analysis of O PTIMISTIC

Original: -AMSG RAD .
Revised: -AMSG RAD .

Original: To obtain m t , we use the extrapolation algorithm of (Scieur et al.

Original: Extrapolation studies estimating the limit of sequence using the last few iterates (Brezinski & Zaglia (2013)).

Original: Some classical works include Anderson acceleration (Walker & Ni.

Original: (2011)), minimal polynomial extrapolation (Cabay & Jackson (1976)), reduced rank extrapolation (Eddy (1979)).

Original: These method typically assumes that the sequence { x t } ∈ R d has a linear relation

Original: [Equation]

Original: for an unknown matrix A ∈ R d × d (not necessarily symmetric).

Original: The goal is to use the last few iterates { x t } to estimate the ﬁxed point x ∗ .

Original: (Scieur et al.

Original: (2016)) adapt the classical extrapolation methods to the iterates/updates of an optimization algorithm and propose an algorithm that produces a solution that is better than the last iterate of the

Original: 1: Required: parameter β 1 , β 2 , and η t .
Revised: 1: Required: parameter β 1 , β 2 , and η t .

Original: 2: Init: w 1 .
Revised: 2: Init: w 1 .

Original: 3: for  do 4:
Revised: 3: for t = 1 to T do 4:

Original: Get mini-batch stochastic gradient vector g t ∈ R d at w t .
Revised: Get mini-batch stochastic gradient vector g t ∈ R d at w t .

Original: 5: θ t = β 1 θ t − 1 + (1 − β 1 ) g t .
Revised: 5: θ t = β 1 θ t − 1 + (1 − β 1 ) g t .

Original: 6: v t = β 2 v t − 1 + (1 − β 2 ) g 2 t .
Revised: 6: v t = β 2 v t − 1 + (1 − β 2 ) g 2 t .

Original: 7: ˆ v t =
Revised: 7: ˆ v t =

Original: max(ˆ v t − 1 , v t ) .
Revised: max(ˆ v t − 1 , v t ) .

Original: 8: w t + 12 =
Revised: 8: w t + 12 =

Original: w t − 12 − η t θ t √ ˆ v t .
Revised: w t − 12 − η t θ t √ ˆ v t .

Original: (element-wise division) 9: w t +1 =
Revised: (element-wise division) 9: w t +1 =

Original: w t + 12 − η t +1 4 1 − β 1 h t +1 √ ˆ v t , where h t +1 :
Revised: w t + 12 − η t +1 4 1 − β 1 h t +1 √ ˆ v t , where h t +1 :

Original: = β 1 θ t − 1 + (1 − β 1 ) m t + and m t +1 is the guess of g t +1 .
Revised: = β 1 θ t − 1 + (1 − β 1 ) m t + and m t +1 is the guess of g t +1 .

Original: (In practice, use w t +1 =
Revised: (In practice, use w t +1 =

Original: w t + 12 − η t
Revised: w t + 12 − η t

Original: +1 h t +1 √ ˆ v t .)
Revised: +1 h t +1 √ ˆ v t .)

Original: 10: end for
Revised: 10: end for

Original:  1: Input:  sequence 
Revised: (Scieur 1: Input: some sequence { x s ∈ R d } s = r s =0 , parameter λ >2:

Original: Compute matrix U =
Revised: Compute matrix U =

Original: [ x 1 − x 0 , . . .
Revised: [ x 1 − x 0 , . . .

Original: , x r − x r − 1 ] ∈ R d × r 3:
Revised: , x r − x r − 1 ] ∈ R d × r 3:

Original: Obtain z by solving ( U (cid:62) U + λI ) z
Revised: Obtain z by solving ( U (cid:62) U + λI ) z

Original: = 1 .
Revised: = 1 .

Original: Get c = z/ ( z (cid:62) 1 ) 5: Output: Σ r − [-
-]{+1 i =
+}
Revised: Get c = z/ ( z (cid:62) 1 ) 5: Output: Σ r − [-
-]{+1 i =
+}

Original: i 0 c i x i , the approximation  the ﬁxed  x ∗ .
Revised: c i x i , the approximation of the ﬁxed point x ∗ .

Revised: To obtain m t , we use the extrapolation algorithm of (Scieur et al.

Revised: Extrapolation studies estimating the limit of sequence using the last few iterates (Brezinski & Zaglia (2013)).

Original: underlying optimization algorithm in practice.
Revised: Some classical works include Anderson acceleration (Walker & Ni. (2011)), minimal polynomial extrapolation (Cabay & Jackson (1976)), reduced rank extrapolation (Eddy (1979)). These method typically assumes that the sequence { x t } ∈ R d has a linear relation [Equation]

Revised: for an unknown matrix A ∈ R d × d (not necessarily symmetric).

Revised: The goal is to use the last few iterates { x t } to estimate the ﬁxed point x ∗ on (4).

Revised: (Scieur et al.

Revised: (2016)) adapt the classical extrapolation methods to the iterates/updates of an optimization algorithm and propose an algorithm that produces a solution that is better than the last iterate of the underlying optimization algorithm in practice.

Original: The algorithm of (Scieur et al.
Revised: The algorithm of (Scieur et al.

Original: (shown in Algorithm 3) allows the iterates { x t } to be nonlinear
Revised: (shown in Algorithm 3) allows the iterates { x t } to be nonlinear

Original: [Equation]
Revised: [Equation]

Original: 5) where e t is a second order term (namely, satisfying (cid:107) e t (cid:107) 

Revised: where e t is a second order term (namely, satisfying (cid:107) e t (cid:107)

Original: = O ( (cid:107) x
Revised: = O ( (cid:107) x

Original: t − 1 − x ∗ (cid:107) 22 ) .
Revised: t − 1 − x ∗ (cid:107) 22 ) .

Original: Some theoretical guarantees regarding the distance between the output and x ∗ is provided in (Scieur et al.
Revised: Some theoretical guarantees regarding the distance between the output and x ∗ is provided in (Scieur et al.

Original: In O PTIMISTIC -AMSG RAD , we use Algorithm 3 to get m t .
Revised: In O PTIMISTIC -AMSG RAD , we use Algorithm 3 to get m t .

Original: Speciﬁcally, m t is obtained by
Revised: Speciﬁcally, m t is obtained by

Original: • Call Algorithm 3 with input consists of some past r + 1 gradients, { g t , g t − 1 , g t − 2 , . .
Revised: • Call Algorithm 3 with input being a sequence of some past r + 1 gradients, { g t , g t − 1 , g t − 2 , . .

Original: . , g t − r } to obtain m t , where r is a parameter.
Revised: . , g t − r } to obtain m t , where r is a parameter.

Original: • Set
Revised: • Set

Original: m t := Σ r − 1 i =

Revised: m t := Σ r − 1 i =

Original: c i g t − r
Revised: c i g t − r

Original: + i from the output of Algorithm 3.
Revised: + i from the output of Algorithm 3.

Original: For this extrapolation method  to work well in predictingg t +1 , the gradient vectors at a speciﬁc time span is assumed to be captured by (5). If the gradient does not change signiﬁcantly, this will be a mild condition.
Revised: If the past few gradients can be modeled by (4) approximately, the extrapolation method should be expected to work well in predicting the gradient. In practice, it helps to achieve faster convergence.

Original: O PTIMISTIC -A DAM
Revised: O PTIMISTIC -A DAM

Original: By removing line 7 in Algorithm 2, the step of making monotone weighted second moment, we obtain an algorithm which we call it O PTIMISTIC -A DAM , as the resulting algorithm can be viewed as an O PTIMISTICvariant of A DAM .
Revised: By removing line 7 in Algorithm 2, the step of making monotone weighted second moment, we obtain an algorithm which we call it O PTIMISTIC -A DAM , as the resulting algorithm can be viewed as an O PTIMISTICvariant of A DAM .

Original: 3. T HEORETICAL ANALYSIS OF O PTIMISTIC -AMSG RAD
Revised: 3. T HEORETICAL ANALYSIS OF O PTIMISTIC -AMSG RAD

Original: We provide the regret analysis here.
Revised: We provide the regret analysis here.

Original: We denote the Mahalanobis norm (cid:107) · (cid:107) H = (cid:112) (cid:104)· , H ·(cid:105) for some PSD matrix
Revised: We denote the Mahalanobis norm (cid:107) · (cid:107) H = (cid:112) (cid:104)· , H ·(cid:105) for some PSD matrix

Original: H .
Revised: H .

Original: For the PSD matrix diag { ˆ v t } , where diag { ˆ v t } represents the diagonal matrix such that its i th diagonal element is ˆ v t
Revised: For the PSD matrix diag { ˆ v t } , where diag { ˆ v t } represents the diagonal matrix such that its i th diagonal element is ˆ v t

Original: [ i ] in Algorithm 2, we deﬁne the the corresponding Mahalanobis norm (cid:107) · (cid:107) ψ t :=
Revised: [ i ] in Algorithm 2, we deﬁne the the corresponding Mahalanobis norm (cid:107) · (cid:107) ψ t :=

Original: (cid:112) (cid:104)· , diag { ˆ v t } 1 / 2 ·(cid:105) , where we use the notation ψ t to represent the matrix diag { ˆ v t }− 1 / 2 .
Revised: (cid:112) (cid:104)· , diag { ˆ v t } 1 / 2 ·(cid:105) , where we use the notation ψ t to represent the matrix diag { ˆ v t } 1 / 2 .

Original: We can also deﬁne the the corresponding dual norm (cid:107) · (cid:107) ψ ∗ t :=
Revised: We can also deﬁne the the corresponding dual norm (cid:107) · (cid:107) ψ ∗ t :=

Original: (cid:112) (cid:104)· , diag { ˆ v t } − 1 / 2 ·(cid:105) .
Revised: (cid:112) (cid:104)· , diag { ˆ v t } − 1 / 2 ·(cid:105) .

Original: We assume that the model parameter w is in d -dimensional space.
Revised: We assume that the model parameter w is in d -dimensional space.

Original: That is, w ∈ R d .
Revised: That is, w ∈ R d .

Original: Also, the analysis of O PTIMISTICAMSG RAD is for unconstrained optimization.
Revised: Also, the analysis of O PTIMISTICAMSG RAD is for unconstrained optimization.

Original: Thus, we assume that the constraint K of the benchmark in the regret deﬁnition, min w ∈K (cid:80) Tt =1 (cid:96) t ( w ) , is a ﬁnite norm ball that contains the optimal solutions to the underlying ofﬂine unconstrained optimization problem.
Revised: Thus, we assume that the constraint K of the benchmark in the regret deﬁnition, min w ∈K (cid:80) Tt =1 (cid:96) t ( w ) , is a ﬁnite norm ball that contains the optimal solutions to the underlying ofﬂine unconstrained optimization problem.

Original: Now we can conduct our analysis.
Revised: Now we can conduct our analysis.

Original: First of all, we can decompose the regret as follows.
Revised: First of all, we can decompose the regret as follows.

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality is by assuming that the loss function (cid:96) t [Equation] ) is convex and that we use the notation g t :=  (cid:96) t ( w t ) which we adopt throughout the following proof for brevity.
Revised: where the ﬁrst inequality is by assuming that the loss function (cid:96) t ( · ) is convex and that we use the notation g t := ∇ (cid:96) t ( w t ) which we adopt throughout the following proof for brevity.

Original: Given the decomposition, let us analyze the ﬁrst term (cid:80)
Revised: Given the decomposition, let us analyze the ﬁrst term (cid:80)

Original: Tt =1 (cid:104) w t − 12 − w ∗ , g t (cid:105) in (6).
Revised: Tt =1 (cid:104) w t − 12 − w ∗ , g t (cid:105) in (6).

Original: Lemma 1.
Revised: Lemma 1.

Original: Denote D ∞ =
Revised: Denote D ∞ =

Original: max t (cid:107) w t − 12 − w ∗ (cid:107) ∞ .
Revised: max t (cid:107) w t − 12 − w ∗ (cid:107) ∞ .

Original: We have that (cid:80)
Revised: We have that (cid:80)

Original: Tt =1 (cid:104) w t − 12 − w ∗

Original: [Equation]

Original: g

Original: [Equation]

Original: ≤

Original: [Equation]
Revised: [Equation]

Original: Astute readers may realize that the bound in Lemma 1 is actually the bound of AMS GRAD .
Revised: Astute readers may realize that the bound in Lemma 1 is actually the bound of AMS GRAD .

Original: Indeed, since in online learning setting the loss vectors g t come adversarially, it does matter how g t is generated.
Revised: Indeed, since in online learning setting the loss vectors g t come adversarially, it does matter how g t is generated.

Original: Therefore, the regret of (cid:80)
Revised: Therefore, the regret of (cid:80)

Original: Tt =1 (cid:104) w t − 12 − w ∗ , g t (cid:105) can be bounded in the same way as AMS GRAD .
Revised: Tt =1 (cid:104) w t − 12 − w ∗ , g t (cid:105) can be bounded in the same way as AMS GRAD .

Original: In Appendix B, we provide the detail proof of Lemma 1.
Revised: In Appendix B, we provide the detail proof of Lemma 1.

Original: Now we switch to bound the other sums (cid:80)
Revised: Now we switch to bound the other sums (cid:80)

Original: Tt =1 (cid:104) w t − w t − 12 , g t − h t (cid:105) + (cid:80)
Revised: Tt =1 (cid:104) w t − w t − 12 , g t − h t (cid:105) + (cid:80)

Original: Tt =1 (cid:104) w t − w t − 12 , h t (cid:105) in (6).
Revised: Tt =1 (cid:104) w t − w t − 12 , h t (cid:105) in (6).

Original: The proof is available in Appendix C.
Revised: The proof is available in Appendix C.

Original: Lemma 2.
Revised: Lemma 2.

Original: [Equation] w t − 12 (cid:107) [Equation]
Revised: (cid:80) Tt =1 (cid:104) w t − w t − 12 , g

Original: −

Original: [Equation]
Revised: [Equation]

Original: Combining (6) and Lemma 1 and 2 leads to
Revised: Combining (6) and Lemma 1 and 2 leads to

Original: Regret

Original: [Equation]
Revised: [Equation]

Original: Now we can conclude the following theorem.
Revised: Now we can conclude the following theorem.

Original: The proof is in Appendix D.
Revised: The proof is in Appendix D.

Original: Theorem 1.
Revised: Theorem 1.

Original: Denote γ :=
Revised: Denote γ :=

Original: β 1 / √ β 2 < 1 and D ∞ =
Revised: β 1 / √ β 2 < 1 and D ∞ =

Original: max t (cid:107) w t − 12 − w ∗ (cid:107) ∞ .
Revised: max t (cid:107) w t − 12 − w ∗ (cid:107) ∞ .

Original: Then,
Revised: Then,

Original: Regret

Original: [Equation]
Revised: [Equation]

Original: One should compare the bound with that of AMSG RAD (Reddi et al.
Revised: One should compare the bound with that of AMSG RAD (Reddi et al.

Original: (2018)), which is
Revised: (2018)), which is

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where η t = η/ √ t in their setting.
Revised: where η t = η/ √ t in their setting.

Original: We see that the  term (cid:80) Tt =1 2 η t (1 − β 1 ) (cid:107) g t − m t (cid:107) 2 ψ ∗ t of O PTIMISTIC -AMSG RAD can be much smaller than the term [Equation]
Revised: We need to compare the last two terms in (8) with the last term in (9). We are going to show that, under certain conditions, the bound is smaller than that of AMSG RAD .

Original: of [Equation] if g t and m t are close.
Revised: Let us suppose that g t is close to m t so that (cid:80) Tt

Original: Yet,
Revised: =1 2 η t (1 − β 1 ) (cid:107) g t − m t (cid:107) 2 ψ ∗ t is much smaller than the last term of (8).

Original: O PTIMISTIC -AMSG RAD also has an additional term  (cid:80) Tt =1 η t 2 (cid:107) g t − h t (cid:107) ( 1 − β 1 8 ψ t − 1 ) ∗ . We claimed that it is smaller than O ( √ T ) so that we are good here.
Revised: Yet, the last term of (8), (cid:80) Tt =1 η t 2 (cid:107) g t − h t (cid:107) ( 1 − β 1 8 ψ t − 1 ) ∗ , might be actually o ( √ T ) and consequently might also be smaller than the last term of (9).

Original: To see this, let us rewrite (cid:80)
Revised: To see this, let us rewrite (cid:80)

Original: Tt =1 η t 2 (cid:107) g t − h t (cid:107) ( 1 − β 1 8 ψ t − 1 ) ∗ and set η t =
Revised: Tt =1 η t 2 (cid:107) g t − h t (cid:107) ( 1 − [Equation] 8 ψ t [Equation]

Revised: 1 ) ∗ and set η t =

Original: η/ √ t as (Reddi et al.
Revised: η/ √ t as (Reddi et al.

Original: ; it is
Revised: ; it is

Revised: O (

Original: [Equation]
Revised: [Equation]

Original: [Equation] (cid:88) 
Revised: T (cid:88) t = d (cid:88) i =

Original: [Equation]
Revised: η √ t ( g t [ i ] 2 − h t

Original:  (cid:112)[Equation]
Revised: [ i ] 2 ) (cid:112)

Original:  [ i ] ) .
Revised: ˆ v t − 1 [ i ] ) .

Original: Assume that if each ( g t [ i ] 2 − h t
Revised: Assume that if each ( g t [ i ] 2 − h t

Original: [ i ] 2 ) √ ˆ v t − 1 [ i ] in the inner sum is bounded by a constant c , we will get (cid:80) Tt
Revised: [ i ] 2 ) √ ˆ v t − 1 [ i ] in the inner sum is bounded by a constant c , we will get (cid:80) Tt

Original: =1 O ( 1 √ t )
Revised: =1 O ( 1 √ t )

Original: =
Revised: =

Original: O ( √ T ) .
Revised: O ( √ T ) .

Original: Yet, the denominator (cid:112) ˆ v t − 1 is non-decreasing so that we can actually have a smaller bound.
Revised: Yet, the denominator (cid:112) ˆ v t − 1 is non-decreasing so that we can actually have a smaller bound.

Revised: That is, in practice, (cid:112)

Revised: ˆ v t − 1 [ i ] might grow over time, and the growth rate is different for different dimension i .

Revised: If (cid:112) ˆ v t − 1 [ i ] in the denominator grows like O ( √ t ) then the last term is just O

Revised: (log T ) , which might be better than that of the last term on (9).

Revised: One can also get a data dependent bound of the last term of (8).

Original: To summarize, when m t is close to g t , O PTIMISTIC -AMSG RAD can have a smaller regret (thus better convergence rate) than A DAM (Kingma & Ba (2015)) and AMSG RAD (Reddi et al.
Revised: To summarize, when m t is close to g t , O PTIMISTIC -AMSG RAD can have a smaller regret (thus better convergence rate) than A DAM (Kingma & Ba (2015)) and AMSG RAD (Reddi et al.

Original: 3. C OMPARISON OF A DAM -DISZ IN D ASKALAKIS ET AL . (2018)
Revised: 3. C OMPARISON OF A DAM -DISZ IN D ASKALAKIS ET AL . (2018)

Original: do 4: β  θ t − 1 + 1 − β 1 ) g t 
Revised: : θ t = β 1 θ t − 1 + (1 − β 1 ) g t .

Original: 6: v 
Revised: 6: v t = β [Equation] v [Equation]

Original: = β 2 v t − 1 + (1 − β 2 ) g 2 t .
Revised: − [Equation] + (1 − β 2 ) g 2 t .

Original: 7: w t +1 =
Revised: 7: w t +1 =

Original: Π k [ w t − 2 η t θ t √ v t + η t θ t − 1 √ v t − 1 ]
Revised: Π k [ w t − 2 η t θ t √ v t + η t [Table]

Original: 8: endfor
Revised: 8: end

Original: We are aware of Algorithm 1 in Daskalakis et al.
Revised: We are aware of Algorithm 1 in Daskalakis et al.

Original: (2018), which was also motivated by Optimistic Online learning 2 .
Revised: (2018), which was also motivated by O PTIMISTIC O NLINE LEARNING 2 .

Original: For comparison, we replicate A DAM -DISZ in Algorithm 4.
Revised: For comparison, we replicate A DAM -DISZ in Algorithm 4.

Original: We are going to describe the differences of the algorithms and the differences of the contributions between our work and their work.
Revised: We are going to describe the differences of the algorithms and the differences of the contributions between our work and their work.

Original: First of all, by comparing O PTIMISTICAMSG RAD (Algorithm 2) or O PTIMISTIC -A DAM (Algorithm 2 without line 7) with A DAM -DISZ
Revised: First of all, by comparing O PTIMISTIC -AMSG RAD (Algorithm 2) or O PTIMISTIC -A DAM (Algorithm 2 without line 7) with A DAM -DISZ

Original: (Algorithm 4), we see that the updates are indeed different.
Revised: (Algorithm 4), we see that the updates are indeed different.

Original: The update cannot be written into the same form as our O PTIMISTICAMSG RAD (and vise versa).
Revised: The update cannot be written into the same form as our O PTIMISTIC -AMSG RAD (and vise versa).

Original: O PTIMISTIC
Revised: O PTIMISTIC

Original: -AMSG RAD
Revised: -AMSG RAD

Original: (Algorithm 2) actually uses two interleaving sequences of updates { w t } Tt =1 , { w t − 12 } Tt =1 [-. Furthermore, by rewriting the update as w t +
 =-]
Revised: (Algorithm 2) actually uses two interleaving sequences of updates { w t } Tt =1 , { w t − 12 } Tt =1 [-. Furthermore, by rewriting the update as w t +
 =-], Also, Daskalakis et al.

Original: Π k [ w t − η t θ t √ v t − η t ( θ t √ v t − θ t − 1 √ v t − 1 )] , we see that it is unclear that the term θ t √ v t − θ t − 1 √ v t − 1 would be a good guess of the next gradient at t

Original: + 1 .

Original: On the other hand, our algorithm uses the extrapolation method to get a good estimate of the next gradient.

Original: Also, Daskalakis et al.

Original: (2018) does not have any theoretical analysis of A DAM -DISZ.
Revised: (2018) does not have any theoretical analysis of A DAM -DISZ.

Original: The theoretical analysis in Daskalakis et al.
Revised: The theoretical analysis in Daskalakis et al.

Original: (2018) is for other algorithms.
Revised: (2018) is for other algorithms.

Original: Second, the contributions are very different.
Revised: Second, the contributions are very different.

Original: Daskalakis et al.
Revised: Daskalakis et al.

Original: (2018) focus on training GANs Goodfellow et al.
Revised: (2018) focus on training GANs Goodfellow et al.

Original: GANs is a two-player zero-sum game.
Revised: GANs is a two-player zero-sum game.

Original: There have been some related works in Optimistic Online learning like Chiang et al.
Revised: There have been some related works in O PTIMISTIC O NLINE LEARNING like Chiang et al.

Original: ; Rakhlin & Sridharan (2013); Syrgkanis et al.(2015)) showing that if both players use some kinds of O PTIMISTIC -update, then acceleration to the convergence of the minimax value of the game is possible.
Revised: Rakhlin & Sridharan (2013); Syrgkanis et al. showing that if both players use some kinds of O PTIMISTICupdate, then acceleration to the convergence of the minimax value of the game is possible.

Original: Daskalakis et al.
Revised: Daskalakis et al.

Original: (2018) was inspired by these related works and showed that O PTIMISTIC -M IRRORD ESCENT can avoid the cycle behavior in a bilinear zero-sum game, which accelerates the convergence.
Revised: (2018) was inspired by these related works and showed that O PTIMISTIC -M IRROR -D ESCENT can avoid the cycle behavior in a bilinear zero-sum game, which accelerates the convergence.

Original: Our work is about solving min x f ( x ) (e.g. empirical risk) quickly.
Revised: Our work is about solving min x f ( x ) (e.g. empirical risk) quickly.

Original: The goals are different.

Original: We also show that ADAM-DISZ suffers the non-convergence issue as A DAM .
Revised: We also show that ADAM-DISZ suffers the non-convergence issue as A DAM .

Original: The proof is available in Appendix E.
Revised: The proof is available in Appendix E.

Original: Theorem 2.
Revised: Theorem 2.

Original: There exists a convex online learning problem such that ADAM-DISZ has nonzero average regret (i.e.
Revised: There exists a convex online learning problem such that ADAM-DISZ has nonzero average regret (i.e.

Original: RegretT (cid:54) =
Revised: RegretT (cid:54) =

Original: o ( T ) )
Revised: o ( T ) )

Original: One might wonder if the non-convergence issue can be avoided if one let the weighted second moment of ADAMDISZ be monotone by adding the step ˆ v t =
Revised: One might wonder if the non-convergence issue can be avoided if one let the weighted second moment of ADAMDISZ be monotone by adding the step ˆ v t =

Original: max(ˆ v t − 1 , v t ) as AMSG RAD , which we call it AMSG RAD -DISZ.
Revised: max(ˆ v t − 1 , v t ) as AMSG RAD , which we call it AMSG RAD -DISZ.

Original: Unfortunately, we are unable to prove if the step guarantees convergence or not.
Revised: Unfortunately, we are unable to prove if the step guarantees convergence or not.

Original: Yet, even if we assume that the step for monotone weighted second moment is adopted and that the regret is like O ( (cid:112) Σ t (cid:107) g t − m t (cid:107) 2 ) , as one would expect for an O PTIMISTIC - ONLINE LEARNING algorithm, the update implies that the algorithm achieves acceleration when m t :=

Original: g t − 1 − g t − 2 is close to g t .

Original: The condition may not hold.

Original: et al.

Original:  EXPERIMENTS
Revised:  EXPERIMENTS

Original: To demonstrate the effectiveness of our proposed method, we test its performance with various neural network architectures, including fully-connected neural networks, convolutional neural networks (CNN’s) and recurrent neural networks (RNN’s).
Revised: To demonstrate the effectiveness of our proposed method, we test its performance with various neural network architectures, including fully-connected neural networks, convolutional neural networks (CNN’s) and recurrent neural networks (RNN’s).

Original: The results illustrate that O PTIMISTIC
Revised: The results illustrate that O PTIMISTIC

Original: -AMSG RAD is able to speed up the convergence of stateof-art AMSG RAD algorithm, making the learning process more efﬁcient.
Revised: -AMSG RAD is able to speed up the convergence of stateof-art AMSG RAD algorithm, making the learning process more efﬁcient.

Original: For AMSG RAD algorithm, we set the parameter β 1 and β 2 , respectively, to be 0.9 and 0.999, as recommended in Reddi et al.
Revised: For AMSG RAD algorithm, we set the parameter β 1 and β 2 , respectively, to be 0.9 and 0.999, as recommended in Reddi et al.

Original: We tune the learning rate η over a ﬁne grid and report the results under best-tuned parameter setting.
Revised: We tune the learning rate η over a ﬁne grid and report the results under best-tuned parameter setting.

Original: For O PTIMISTIC -AMSG RAD and AMSG RAD -DISZ, we use same β 1 , β 2 and learning rate as those for AMSG RAD to make a fair comparison of the enhancement brought by the optimistic step.
Revised: For O PTIMISTIC -AMSG RAD and AMSG RAD -DISZ, we use same β 1 , β 2 and learning rate as those for AMSG RAD to make a fair comparison of the enhancement brought by the optimistic step.

Original: We use the same weight initialization for all algorithms.
Revised: We use the same weight initialization for all algorithms.

Original: The remaining tuning parameter of O PTIMISTIC -AMSG RAD is r , the number of previous gradients that we use to predict the next move.
Revised: The remaining tuning parameter of O PTIMISTIC -AMSG RAD is r , the number of previous gradients that we use to predict the next move.

Original: We conduct O PTIMISTIC
Revised: We conduct O PTIMISTIC

Original: -AMSG RAD with different values of r and observe similar performance.
Revised: -AMSG RAD with different values of r and observe similar performance (See Appendix A).

Original: Hence, we report r = 15 for all experiments for tidiness of plots.
Revised: Hence, we report r = 15 for all experiments for tidiness of plots.

Original: To follow previous works of Reddi et al.
Revised: To follow previous works of Reddi et al.

Original: and Kingma & Ba (2015), we compare different methods on MNIST, CIFAR and IMDB datasets in our experiment.
Revised: and Kingma & Ba (2015), we compare different methods on MNIST, CIFAR and IMDB datasets in our experiment.

Original: For MNIST, we use a noisy version named as MNIST-back-rand in Larochelle et al.
Revised: For MNIST, we use a noisy version named as MNIST-back-rand in Larochelle et al.

Original: (2007) to increase the training difﬁculty.
Revised: (2007) to increase the training difﬁculty.

Original: 4. MNIST- BACK -R AND + M ULTI - LAYER N EURAL N ETWORKS
Revised: 4. MNIST- BACK -R AND + M ULTI - LAYER N EURAL N ETWORKS

Original: Our experiments start with fully connected neural network for multi-classiﬁcation problems.
Revised: Our experiments start with fully connected neural network for multi-class classiﬁcation problems.

Original: MNIST-back-rand dataset consists of 12000 training samples and 50000 test samples, where random background is inserted in original MNIST hand written digit images.
Revised: MNIST-back-rand dataset consists of 12000 training samples and 50000 test samples, where random background is inserted in original MNIST hand written digit images.

Original: The input dimension is 784 ( 28 × 28 ) and the number of classes is 10.
Revised: The input dimension is 784 ( 28 × 28 ) and the number of classes is 10.

Original: We investigate a multi-layer neural networks with input layer followed by a hidden layer with 200 cells, which is then connected to a layer with 100 neurons before the output layer.
Revised: We investigate a multi-layer neural networks with input layer followed by a hidden layer with 200 cells, which is then connected to a layer with 100 neurons before the output layer.

Original: All hidden layer cells are rectiﬁer linear units (ReLu’s).
Revised: All hidden layer cells are rectiﬁer linear units (ReLu’s).

Original: We use mini-batch size 128 to calculate stochastic gradient in each iteration.
Revised: We use mini-batch size 128 to calculate stochastic gradient in each iteration.

Original: Model performance is evaluated by multi-class cross entropy loss.
Revised: Model performance is evaluated by multi-class cross entropy loss.

Original: The training loss with respect to number of iterations is reported in ﬁgure 1.
Revised: The training loss with respect to number of iterations is reported in ﬁgure 1.

Original: [Figure]
Revised: [Figure]

Original: On this dataset, we empirically observe obvious improvement of O PTIMISTIC
Revised: On this dataset, we empirically observe obvious improvement of O PTIMISTIC

Original: -AMSG RAD in terms of both convergence speed and training loss.
Revised: -AMSG RAD in terms of both convergence speed and training loss.

Original: On the other hand, AMSG RAD -DISZ performs similarly to AMSG RAD in general.
Revised: On the other hand, AMSG RAD -DISZ performs similarly to AMSG RAD in general.

Original: 4. CIFAR10 + CONVOLUTIONAL NEURAL NETWORKS (CNN)
Revised: 4. CIFAR10 + CONVOLUTIONAL NEURAL NETWORKS (CNN)

Original: Convolutional Neural Networks (CNN) have been widely studied and playing an importantrole in various deep learning applications such as computer vision and natural language processing.
Revised: Convolutional Neural Networks (CNN) have been widely studied and is important in various deep learning applications such as computer vision and natural language processing.

Original: We test the effectiveness of O PTIMISTICAMSG RAD in deep CNN’s with dropout.
Revised: We test the effectiveness of O PTIMISTIC -AMSG RAD in deep CNN’s with dropout.

Original: We use the CIFAR10 dataset, which includes 60,000 images (50,000 for training and 10,000 for testing) of size 32 × 32 in 10 different classes.
Revised: We use the CIFAR10 dataset, which includes 60,000 images (50,000 for training and 10,000 for testing) of size 32 × 32 in 10 different classes.

Original: ALL-CNN architecture proposed in Springenberg et al.
Revised: ALL-CNN architecture proposed in Springenberg et al.

Original: (2015) is implemented with two blocks of 3 × 3 convolutional ﬁlter, 3 × 3 convolutional layer with stride 2 and dropout layer with keep probability 0.5.
Revised: (2015) is implemented with two blocks of 3 × 3 convolutional ﬁlter, 3 × 3 convolutional layer with stride 2 and dropout layer with keep probability 0.5.

Original: Another block of 3 × 3 , 1 × 1 convolutional layer and a 6 × 6 global averaging pooling is added before the output layer.
Revised: Another block of 3 × 3 , 1 × 1 convolutional layer and a 6 × 6 global averaging pooling is added before the output layer.

Original: We apply another dropout with keep probability 0.8 on the input layer.
Revised: We apply another dropout with keep probability 0.8 on the input layer.

Original: The cost function is multi-class cross entropy.
Revised: The cost function is multi-class cross entropy.

Original: The batch size is 128.
Revised: The batch size is 128.

Original: The images are all whitened.
Revised: The images are all whitened.

Original: The training loss is provided in ﬁgure 2.
Revised: The training loss is provided in ﬁgure 2.

Original: The result shows that O PTIMISTIC -AMSG RAD accelerates the learning process signiﬁcantly and gives lowest training cost after 10000 iterations.
Revised: The result shows that O PTIMISTIC -AMSG RAD accelerates the learning process signiﬁcantly and gives lowest training cost after 10000 iterations.

Original: For this dataset, the performance of AMSG RAD -DISZ is worse than original AMSG RAD .
Revised: For this dataset, the performance of AMSG RAD -DISZ is worse than original AMSG RAD .

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: OPTIMISTIC AMSGrad

Original: 4. IMDB + L ONG -S HORT T ERM M EMORY (LSTM)
Revised: 4. IMDB + L ONG -S HORT T ERM M EMORY (LSTM)

Original: As another important application of deep learning, natural language processing tasks often beneﬁt from considering sequence dependency in the models.
Revised: As another important application of deep learning, natural language processing tasks often beneﬁt from considering sequence dependency in the models.

Original: Recurrent Neural Networks (RNN’s) achieves this goal by adding hidden state units that act as ”memory”.
Revised: Recurrent Neural Networks (RNN’s) achieves this goal by adding hidden state units that act as ”memory”.

Original: Long-Short Term Memory (LSTM) is the most popular structure in building RNN’s.
Revised: Long-Short Term Memory (LSTM) is the most popular structure in building RNN’s.

Original: We use IMDB movie review dataset from Maas et al.
Revised: We use IMDB movie review dataset from Maas et al.

Original: (2011) to test the performance of O PTIMISTIC
Revised: (2011) to test the performance of O PTIMISTIC

Original: -AMSG RAD in RNN’s under the circumstance of high data sparsity.
Revised: -AMSG RAD in RNN’s under the circumstance of high data sparsity.

Original: IMDB is a binary classiﬁcation dataset with 25000 training and test samples respectively.
Revised: IMDB is a binary classiﬁcation dataset with 25000 training and test samples respectively.

Original: Our model includes a word embedding layer with 5000 input entries representing most frequent words in the dataset and each word is embedded into a 32 dimensional space.
Revised: Our model includes a word embedding layer with 5000 input entries representing most frequent words in the dataset and each word is embedded into a 32 dimensional space.

Original: The output of embedding layer is passed to 100 LSTM units, which is then connected to 128 fully connected ReLu’s before reaching the output layer.
Revised: The output of embedding layer is passed to 100 LSTM units, which is then connected to 100 fully connected ReLu’s before reaching the output layer.

Original: Binary cross-entropy loss is used and the batch size is 100.
Revised: Binary cross-entropy loss is used and the batch size is 128.

Original: We provide the results in ﬁgure 3.
Revised: We provide the results in ﬁgure 3.

Original: [Figure]
Revised: [Figure]

Original: We observe a considerable improvement in convergence speed.
Revised: We observe a considerable improvement in convergence speed.

Original: In the ﬁrst epoch, the result is more exciting.
Revised: In the ﬁrst epoch, the result is more exciting.

Original: At epoch 0.5, O PTIMISTIC -AMSG RAD already achieves the training loss that vanilla AMSG RAD can produce with more than 1 epoch.
Revised: At epoch 0.5, O PTIMISTIC -AMSG RAD already achieves the training loss that vanilla AMSG RAD can produce with more than 1 epoch.

Original: The sample efﬁciency is signiﬁcantly improved.
Revised: The sample efﬁciency is signiﬁcantly improved.

Original: On this dataset, AMSG RAD -DISZ performs less effectively and may be trapped in local minimum.
Revised: On this dataset, AMSG RAD -DISZ performs less effectively and may be trapped in local minimum.

Original: We remark that in each iteration, only a small portion of gradients in embedding layer is non-zero.
Revised: We remark that in each iteration, only a small portion of gradients in embedding layer is non-zero.

Original: Thus, this experiment demonstrates that O PTIMISTIC -AMSGR AD could also perform well with sparse gradient.
Revised: Thus, this experiment demonstrates that O PTIMISTIC -AMSGR AD could also perform well with sparse gradient.

Original: ( See additional experiments in Appendix A.)
Revised: ( See additional experiments in Appendix A

Original:  E XTENSIONS AND F UTURE W ORK
Revised:  E XTENSIONS AND F UTURE W ORK

Original: In this paper, we propose O PTIMISTIC
Revised: In this paper we propose O PTIMISTIC

Original: -AMSG RAD , which combines optimistic learning and AMSG RAD to strengthen the learning process of optimization problems, in particular, deep neural networks.
Revised: -AMSG RAD , which combines optimistic learning and AMSG RAD to strengthen the learning process of optimization problems, in particular, deep neural networks.

Original: The idea of adding optimistic step can be easily extended to other optimization algorithms, e.g A DAM and A DA G RAD .
Revised: The idea of adding optimistic step can be easily extended to other optimization algorithms, e g A DAM and A DA G RAD .

Original: We provide
Revised: We provide

Original: O PTIMISTIC -A DA G RAD algorithm and theoretical results in Appendix F. A potential directions based on this work is to improve the method for predicting next gradient.
Revised: O PTIMISTIC -A DA G RAD algorithm and theoretical results in Appendix F. A potential direction based on this work is to improve the method for predicting next gradient.

Original: We expect that optimistic acceleration strategy could be widely used in various optimization problems.
Revised: We expect that optimistic acceleration strategy could be widely used in various optimization problems.

Original: A M ORE E XPERIMENTAL R ESULTS A. CHOOSING PARAMETER r
Revised: A M ORE E XPERIMENTAL R ESULTS A. CHOOSING PARAMETER r

Original: In O PTIMISTIC -AMSG RAD , parameter r  the number of previous gradients used, may affect the performance dramatically.
Revised: In O PTIMISTIC -AMSG RAD , parameter r , the number of previous gradients used, may affect the performance dramatically.

Original: If we choose r too small ( e g r < 10 ), the optimistic updates will start at very early stage, but the information we collect from the past is limited.
Revised: If we choose r too small ( e.g r < 10 ), the optimistic updates will start at very early stage, but the information we collect from the past is limited.

Original: This may make our training process off-track at ﬁrst several iterations.
Revised: This may make our training process off-track at ﬁrst several iterations.

Original: On the other hand, if r is chosen to be too large ( e.g r > = 30 ), although we may get a better prediction of next gradient, the optimistic step will start late so O PTIMISTIC
Revised: On the other hand, if r is chosen to be too large ( e.g r > = 30 ), although we may get a better prediction of next gradient, the optimistic step will start late so O PTIMISTIC

Original: -AMSG RAD may miss great chances to improve the learning performance at early stages.
Revised: -AMSG RAD may miss great chances to improve the learning performance at early stages.

Original: Additionally, we need more room to store past gradients, hence the operational time will increase.
Revised: Additionally, we need more room to store past gradients, hence the operational time will increase.

Original: [Figure]

Original: The empirical impact of different r value is reported in ﬁgure 4.
Revised: The empirical impact of different r value is reported in ﬁgure 4.

Original: We suggest 10 ≤ r ≤ 20 as an ideal range, and r =tend to perform well in most training tasks.
Revised: We suggest 10 ≤ r ≤ 20 as an ideal range, and r =tend to perform well in most training tasks.

Original: Actually, we may make the algorithm more ﬂexible by ”early start”.
Revised: Actually, we may make the algorithm more ﬂexible by ”early start”.

Original: For example, when we set r = 20 , we can instead start adding optimistic step at iteration 10, and gradually increase the number of past gradients we use to predict next move from 10 to 20 in next 10 iterations.
Revised: For example, when we set r = 20 , we can instead start adding optimistic step at iteration 10, and gradually increase the number of past gradients we use to predict next move from 10 to 20 in next 10 iterations.

Original: After iteration 21, we ﬁx the moving window size for optimistic prediction as 20.
Revised: After iteration 21, we ﬁx the moving window size for optimistic prediction as 20.

Original: This may bring enhancement to O PTIMISTIC
Revised: This may bring enhancement to O PTIMISTIC

Original: -AMSG RAD because it can seek opportunities to accelerate learning in ﬁrst several iterations, which is critical for indicating a better direction towards the minimum loss. 
Revised: -AMSG RAD because it can seek opportunities to accelerate learning in ﬁrst several iterations, which is critical for indicating a better direction towards the minimum loss. [Figure]

Original: A. O PTIMISTIC ADAM
Revised: A. O PTIMISTIC ADAM

Original: We also conduct experiments on O PTIMISTIC -A DAM and A DAM -DISZ.
Revised: We also conduct experiments on O PTIMISTIC -A DAM and A DAM -DISZ.

Original: The experiment setting is similar to
Revised: The experiment setting is similar to

Original: O PTIMISTIC -AMSG RAD , where we ﬁx β 1 , β 2 and compare the performance using the best learning rate with respect to A DAM.
Revised: O PTIMISTIC -AMSG RAD [Figure] where we ﬁx β 1 , β 2 and compare the performance using the best learning rate with respect to A DAM

Original: We provide a brief summary of the results in ﬁgure 5, with r = 15 .
Revised: We provide a brief summary of the results in ﬁgure [Figure] with r = 15 .

Original: The improvement brought by
Revised: The improvement brought by

Original: O PTIMISTIC -A DAM is obvious, indicating that adding optimistic step could also enhance the performance of A DAM optimizer.
Revised: O PTIMISTIC -A DAM is obvious, indicating that adding optimistic step could also enhance the performance of A DAM optimizer

Original: [Figure]
Revised: [Figure]

Original: B P ROOF OF LEMMA
Revised: B P ROOF OF LEMMA

Original: Lemma 1 Denote [Equation]
Revised: Lemma 1 Denote D ∞ =

Original: [Figure] [Equation] [Figure] [Equation]
Revised: max t (cid:107) w t − 12 − w ∗ (cid:107) ∞ .

Original: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof of this lemma basically follows that of (Reddi et al.
Revised: The proof of this lemma basically follows that of (Reddi et al.

Original: We have that
Revised: We have that

Original: [Equation]
Revised: [Equation]

Original: By rearranging the terms above and summing it from t = 1 , . . .
Revised: By rearranging the terms above and summing it from t = 1 , . . .

Original: , T ,T
Revised: , T ,

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality is due to Young’s inequality and the second one is due to the constraint 0 < β 1 ≤ 1 so that η t 
Revised: where the ﬁrst inequality is due to Young’s inequality and the second one is due to the constraint 0 < β 1 ≤ 1 so that η t 2(1 − β

Original: [Equation]
Revised: [Equation]

Revised: +

Original: [Equation]
Revised: [Equation]

Original: Now we can bound (cid:80) Tt =
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: w
Revised: ≤ η t (1 − β

Original: [Equation]
Revised: [Equation]

Original:  in (6) as
Revised: Now we can bound (cid:80) Tt =1 (cid:104) w t − 12 − w ∗ , g t (cid:105) in (6) as

Original: T (cid:88) 
Revised: T (cid:88) t =

Original: [Equation]
Revised: [Equation]

Original:  where the ﬁrst inequality is due to (12) [Equation]
Revised: [Equation] where the ﬁrst inequality is due to (12).

Original: Continue the analysis, we have
Revised: Continue the analysis, we have

Original: [Equation]
Revised: [Equation]

Original: where the last equality is by telescoping sum
Revised: where the last equality is by telescoping sum.

Original: [Equation]

Original: C P ROOF OF LEMMA
Revised: C P ROOF OF LEMMA

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From the update,
Revised: From the update,

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have that
Revised: Therefore, we have that

Original: [Equation]
Revised: [Equation]

Original: where (a) is by H ¨ older’s inequality and (b) is by Young’s inequality.
Revised: where (a) is by H ¨ older’s inequality and (b) is by Young’s inequality.

Original: D P ROOF OF T HEOREM
Revised: D P ROOF OF T HEOREM

Original: Theorem 1 Denote γ := β 1 / √ β 2 < 1 and D ∞ =
Revised: Theorem 1 Denote γ := β 1 / √ β 2 < 1 and D ∞ =

Original: max t (cid:107) w t − 12 − w ∗ (cid:107) ∞ .
Revised: max t (cid:107) w t − 12 − w ∗ (cid:107) ∞ .

Original: Then,
Revised: Then,

Original: Regret T ≤ 2 η T (1 − β 1 ) D
Revised: Regret T ≤ 2 η T (1 − β 1 ) D

Original: 2 ∞ d (cid:88) i
Revised: 2 ∞ d (cid:88) i

Original: =
Revised: =

Original: ˆ v T [ i ] 2 + D 2 ∞
Revised: ˆ v T [ i ] 2 + D 2 ∞

Original: T (cid:88) t = d (cid:88) i =
Revised: T (cid:88) t = d (cid:88) i =

Original: β 1 ˆ v t
Revised: β 1 ˆ v t

Original: [ i ] 1 /2 η t (1 − β 1 ) +
Revised: [ i ] 1 /2 η t (1 − β 1 ) +

Original: T (cid:88) t = 2 η t (1 − β 1 ) (cid:107) g t − m t (cid:107) 2 ψ ∗ t +
Revised: T (cid:88) t = 2 η t (1 − β 1 ) (cid:107) g t − m t (cid:107) 2 ψ ∗ t +

Original: T (cid:88) t =
Revised: T (cid:88) t =

Original: η t
Revised: η t

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Recall that from (7), we show that
Revised: Recall that from (7), we show that

Original: [Equation]
Revised: [Equation]

Original: To proceed
Revised: To proceed

Original: [Equation]
Revised: [Equation]

Original: let us analyze (cid:80)
Revised: let us analyze (cid:80)

Original: Tt =
Revised: Tt =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: t − 1 above.
Revised: t − 1 above.

Original: Notice that
Revised: Notice that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where we use the update rule
Revised: where we use the update rule

Original: [Equation]
Revised: [Equation]

Original: w t = w t − 12 − η t 4 1 − β
Revised: w t = w t − 12 − η t 4 1 − β

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we have that
Revised: we have that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 21) where the second inequality is due to that the sequence { ˆ v t [ i ] } is non-decreasing and the last equality is because θ t − h t = β 1 θ t − 

Revised: 21) where the second inequality is due to that the sequence { ˆ v t [ i ] } is non-decreasing and the last equality is because θ t − h t = β 1 θ t −

Original: + (1 − β 1 )( g t ) − (cid:0)
Revised: + (1 − β 1 )( g t ) − (cid:0)

Original: [Equation]
Revised: [Equation]

Original: + (1 − β 1 )( m t ) (cid:1) =
Revised: + (1 − β 1 )( m t ) (cid:1) =

Original: (1 − β 1 )( g t − m t ) .
Revised: (1 − β 1 )( g t − m t ) .

Original: To summarize,
Revised: To summarize,

Original: Regret
Revised: Regret

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: E P ROOF OF T HEOREM
Revised: E P ROOF OF T HEOREM

Original: Theorem 2 There exists a convex online learning problem such that ADAM-DISZ has nonzero average regret (i.e.
Revised: Theorem 2 There exists a convex online learning problem such that ADAM-DISZ has nonzero average regret (i.e.

Original: RegretT (cid:54) =
Revised: RegretT (cid:54) =

Original: o ( T ) )
Revised: o ( T ) )

Original: We basically follow the same setting as Theorem 1 of Reddi et al.
Revised: We basically follow the same setting as Theorem 1 of Reddi et al.

Original: In each round, the loss function (cid:96) t ( w ) is linear and the learner’s decision space is K
Revised: In each round, the loss function (cid:96) t ( w ) is linear and the learner’s decision space is K

Original: = [ − 1 , 1] f t ( w ) = (cid:26) Cw , for t mod 3 =− w , otherwise , where C ≥ 4 .
Revised: = [ − 1 , 1] f t ( w ) = (cid:26) Cw , for t mod 3 =− w , otherwise , where C ≥ 4 .

Original: For this loss function sequences, the point w = − 1 achieves the minimum regret, i.e. − 1 = arg min w ∈K (cid:80) Tt
Revised: For this loss function sequences, the point w = − 1 achieves the minimum regret, i.e. − 1 = arg min w ∈K (cid:80) Tt

Original: =1 f t ( w ) when T → ∞ .
Revised: =1 f t ( w ) when T → ∞ .

Original: Consider the execution of ADAM-DISZ with β 1 = 0 , β 2 = 1 1+2 C 2 , η t = η √ t with η < 13 √ 1 − β 2 .
Revised: Consider the execution of ADAM-DISZ with β 1 = 0 , β 2 = 1 1+2 C 2 , η t = η √ t with η < 13 √ 1 − β 2 .

Original: Notice that β 21 / √ β 2 < 1 in this case, which satisﬁes the conditions of Kingma & Ba (2015).
Revised: Notice that β 21 / √ β 2 < 1 in this case, which satisﬁes the conditions of Kingma & Ba (2015).

Original: The goal is to show that for all t , w 3 t +1 = 1 , and w t > 0 .
Revised: The goal is to show that for all t , w 3 t +1 = 1 , and w t > 0 .

Original: Let us denote ˆ w t
Revised: Let us denote ˆ w t

Original: +1 :
Revised: +1 :

Original: = w t − 2 η t θ t √ v t + η t θ t − 1 √ v t − 1 and w t +1 :
Revised: = w t − 2 η t θ t √ v t + η t θ t − 1 √ v t − 1 and w t +1 :

Original: =
Revised: =

Original: Π k
Revised: Π k

Original: [ ˆ w t +1 ] .
Revised: [ ˆ w t +1 ] .

Original: Assume the initial point is 1 .
Revised: Assume the initial point is 1 .

Original: As pointed it out by Reddi et al.
Revised: As pointed it out by Reddi et al.

Original: , the assumption of the initial point is without loss of generality.
Revised: , the assumption of the initial point is without loss of generality.

Original: If the initial point is not 1 , then one can translate the coordinate system to a new coordinate system so that the initial point is 1 and then choose the loss function sequences for the new coordinate system.
Revised: If the initial point is not 1 , then one can translate the coordinate system to a new coordinate system so that the initial point is 1 and then choose the loss function sequences for the new coordinate system.

Original: Therefore, the base case x 1 = 1 is true.
Revised: Therefore, the base case x 1 = 1 is true.

Original: Now assume that for some t > 0 , x 3 t +1 = 1 and x s > 0 for all s ≤ 3 t + 1 .
Revised: Now assume that for some t > 0 , x 3 t +1 = 1 and x s > 0 for all s ≤ 3 t + 1 .

Original: Observe that.
Revised: Observe that.

Original: ∇ f
Revised: ∇ f

Original: [Equation]
Revised: [Equation]

Original: (cid:26)
Revised: (cid:26)

Original: [Equation]
Revised: [Equation]

Original: , for t mod 3 =− 1 , otherwise ,
Revised: , for t mod 3 =− 1 , otherwise ,

Original: According to the update of ADAM-DISZ
Revised: According to the update of ADAM-DISZ

Original: [Equation]
Revised: [Equation]

Original: So, ˆ w 3 t +2 =
Revised: So [Equation] ˆ w 3 t +2 =

Original: w 3 t +
Revised: w 3 t +

Original: [Equation]
Revised: [Equation]

Original: For ˆ w 3 t +4 , let us ﬁrst consider the case that ˆ w 3 t +3 <
Revised: For ˆ w 3 t +4 , let us ﬁrst consider the case that ˆ w 3 t +3 <

Original: [Equation]
Revised: [Equation]

Original: η
Revised: η

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where (a) uses (cid:113) β 2 C2 +
Revised: where (a) uses (cid:113) β 2 C

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: /= √ 1 − β 2 for the choice of β 2 = 1 / (1 + 2 C 2 ) , (b) is because 1 − √ 3 t +1 √ 3 t +2 is a decreasing function of t and has its largest value at t = 1 for any t > 0 , which is 1 − ( √ 4 √ 5 ) ≤ 0 .
Revised: /= √ 1 − β 2 for the choice of β 2 = 1 / (1 + 2 C 2 ) , (b) is because 1 − √ 3 t +1 √ 3 t +2 is a decreasing function of t and has its largest value at t = 1 for any t > 0 , which is 1 − ( √ 4 √ 5 ) ≤ 0 .

Original: To summarize, we have that w 3 t +4 = 1 .
Revised: To summarize, we have that w 3 t +4 = 1 .

Original: Now if it is the case that ˆ w 3 t +3 ≥
Revised: Now if it is the case that ˆ w 3 t +3 ≥

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is because
Revised: where the last inequality is because

Original: [Equation]
Revised: [Equation]

Original: To see this,
Revised: To see this,

Original: [Equation]
Revised: [Equation]

Original: So, the above inequality is equivalent to
Revised: So, the above inequality is equivalent to

Original: [Equation]
Revised: [Equation]

Original: which is true.
Revised: which is true.

Original: This means that w 3 t +4 = 1 .
Revised: This means that w 3 t +4 = 1 .

Original: Therefore, we have completed the induction.
Revised: Therefore, we have completed the induction.

Original: To summarize, we have (cid:96) 3 t +1 ( w 3 t +1 )
Revised: To summarize, we have (cid:96) 3 t +1 ( w 3 t +1 )

Original: + (cid:96) 3 t +2 ( w 3 t +2 )
Revised: + (cid:96) 3 t +2 ( w 3 t +2 )

Original: + (cid:96) 3 t +3 ( w 3 t +3 ) − (cid:96) 3 t +1 ( − 1) − (cid:96) 3 t +2 ( w − 1 ) − (cid:96) 3 t +3 ( − 1) ≥ 2 C − 4 .
Revised: + (cid:96) 3 t +3 ( w 3 t +3 ) − (cid:96) 3 t +1 ( − 1) − (cid:96) 3 t +2 ( w − 1 ) − (cid:96) 3 t +3 ( − 1) ≥ 2 C − 4 .

Original: That is, for every 3 steps, the algorithms suffers regret at least 2 C − 4 , this means that the regret over T rounds would be (2 C − 4) T/ 3 , which is not sublinear to T .
Revised: That is, for every 3 steps, the algorithms suffers regret at least 2 C − 4 , this means that the regret over T rounds would be (2 C − 4) T/ 3 , which is not sublinear to T .

Original: Now we have completed the proof.
Revised: Now we have completed the proof.

Original: One may follow the analysis of Theorem 2 of Reddi et al.
Revised: One may follow the analysis of Theorem 2 of Reddi et al.

Original: (2018) to generalize the result so that ADAM-DISZ does not converge for any β 1 , β 2 ∈ [0 , 1) such that β 1 < √ β 2 .
Revised: (2018) to generalize the result so that ADAM-DISZ does not converge for any β 1 , β 2 ∈ [0 , 1) such that β 1 < √ β 2 .

Original: [Equation]
Revised: [Equation]

Original: The optimistic update step can also be extended to other algorithms as well.
Revised: The optimistic update step can also be extended to other algorithms as well.

Original: For example, based on A DA G RAD , we propose O PTIMISTIC -A DA G RAD by including the optimistic update step.
Revised: For example, based on A DA G RAD , we propose O PTIMISTIC -A DA G RAD by including the optimistic update step.

Original: Algorithm
Revised: Algorithm

Original: G RAD
Revised: G RAD

Original: [Equation]
Revised: [Equation]

Original: where diag ( G t ) :=   (cid:80) ts =1 g t [1]   (cid:80) ts
Revised: where diag ( G t ) :=   (cid:80) ts =1 g t [1]   (cid:80) ts

Original: [Equation]
Revised: [Equation]

Original: g
Revised: g

Original: [Equation]
Revised: [Equation]

Original: [2]   · 
Revised: [2]   · 

Original:  6:
Revised:  6:

Original: [Equation]
Revised: [Equation]

Original: where m t +1 is the guess of g t +1 .
Revised: where m t +1 is the guess of g t +1 .

Original: 7: end for
Revised: 7: end for

Original: F. A NALYSIS :
Revised: F. A NALYSIS :

Original: We provide the regret analysis here.
Revised: We provide the regret analysis here.

Original: Let us recall the notations and assumptions ﬁrst.
Revised: Let us recall the notations and assumptions ﬁrst.

Original: We denote the Mahalanobis norm (cid:107) · (cid:107) H = (cid:112) (cid:104)· , H ·(cid:105) for some PSD matrix H .
Revised: We denote the Mahalanobis norm (cid:107) · (cid:107) H = (cid:112) (cid:104)· , H ·(cid:105) for some PSD matrix H .

Original: We let ψ t ( x ) :
Revised: We let ψ t ( x ) :

Original: = (cid:104) x, diag { ˆ v t } 1 / 2 x (cid:105) for the PSD matrix diag { ˆ v t } , where diag { ˆ v t } represents the diagonal matrix such that its i th diagonal element is ˆ v t
Revised: = (cid:104) x, diag { ˆ v t } 1 / 2 x (cid:105) for the PSD matrix diag { ˆ v t } , where diag { ˆ v t } represents the diagonal matrix such that its i th diagonal element is ˆ v t

Original: [ i ] in Algorithm 2.
Revised: [ i ] in Algorithm 2.

Original: Consequently, ψ t ( · ) is 1-strongly convex with respect to the norm (cid:107) · (cid:107) ψ t := (cid:112) (cid:104)· , diag { ˆ v t } 1 / 2 ·(cid:105) .
Revised: Consequently, ψ t ( · ) is 1-strongly convex with respect to the norm (cid:107) · (cid:107) ψ t := (cid:112) (cid:104)· , diag { ˆ v t } 1 / 2 ·(cid:105) .

Original: Namely, ψ t ( · ) satisﬁes ψ t ( u ) ≥ ψ t ( v ) + (cid:104) ψ t ( v ) , u − v (cid:105) + 1 2 (cid:107) u − v (cid:107) 2 ψ t for any point u, v .
Revised: Namely, ψ t ( · ) satisﬁes ψ t ( u ) ≥ ψ t ( v ) + (cid:104) ψ t ( v ) , u − v (cid:105) + 1 2 (cid:107) u − v (cid:107) 2 ψ t for any point u, v .

Original: A consequence of 1-strongly convexity of ψ t ( · ) is that B ψ t ( u, v ) ≥ ψ t , where the Bregman divergence B ψ t ( u, v ) is deﬁned as B ψ t ( u, v ) :
Revised: A consequence of 1-strongly convexity of ψ t ( · ) is that B ψ t ( u, v ) ≥ ψ t , where the Bregman divergence B ψ t ( u, v ) is deﬁned as B ψ t ( u, v ) :

Original: = ψ t ( u ) − ψ t ( v ) −(cid:104)
Revised: = ψ t ( u ) − ψ t ( v ) −(cid:104)

Original: ψ t ( v ) , u − v (cid:105) and ψ t is called the distance generating function.
Revised: ψ t ( v ) , u − v (cid:105) and ψ t is called the distance generating function.

Original: We can also deﬁne the associate dual norm as ψ ∗ t ( x ) := (cid:104) x, diag { ˆ v t } − 1 / 2 x (cid:105) .
Revised: We can also deﬁne the associate dual norm as ψ ∗ t ( x ) := (cid:104) x, diag { ˆ v t } − 1 / 2 x (cid:105) .

Original: We assume that the model parameter w is in d -dimensional space.
Revised: We assume that the model parameter w is in d -dimensional space.

Original: That is, w ∈ R d .
Revised: That is, w ∈ R d .

Original: It sufﬁces to analyze Algorithm 6, which holds for any convex set K .
Revised: It sufﬁces to analyze Algorithm 6, which holds for any convex set K .

Original: The algorithm reduces to Algorithm 5 when K = R d .
Revised: The algorithm reduces to Algorithm 5 when K = R d .

Original: 1: Required: parameter η t .
Revised: 1: Required: parameter η t .

Original: 2: Init: w 1 .
Revised: 2: Init: w 1 .

Original: 3: for t = 1 to T do 4:
Revised: 3: for t = 1 to T do 4:

Original: Get current gradient g t at w t .
Revised: Get current gradient g t at w t .

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where m t +1 is the guess of g t +1 .
Revised: where m t +1 is the guess of g t +1 .

Original: 7: end for
Revised: 7: end for

Original: By regret decomposition, we have that
Revised: By regret decomposition, we have that

Original: Regret T :=
Revised: Regret T :=

Original: T (cid:88) t = (cid:96) t ( w t ) − min w ∈K
Revised: T (cid:88) t = (cid:96) t ( w t ) − min w ∈K

Original: T (cid:88) t = (cid:96) t ( w ) ≤
Revised: T (cid:88) t = (cid:96) t ( w ) ≤

Original: T (cid:88) t
Revised: T (cid:88) t

Original: = (cid:104) w t − w ∗ , ∇ (cid:96) t ( w t ) (cid:105) =
Revised: = (cid:104) w t − w ∗ , ∇ (cid:96) t ( w t ) (cid:105) =

Original: T (cid:88) t
Revised: T (cid:88) t

Original: = (cid:104)
Revised: = (cid:104)

Original: [Equation]
Revised: [Equation]

Original: (cid:96)
Revised: (cid:96)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: (cid:105)
Revised: (cid:105)

Original: [Equation]
Revised: [Equation]

Original: (cid:104)
Revised: (cid:104)

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:105)
Revised: (cid:105)

Original: [Equation]
Revised: [Equation]

Original: (cid:104) w t + 12 − w ∗ , ∇ (cid:96) t ( w t ) (cid:105) .
Revised: (cid:104) w t + 12 − w ∗ , ∇ (cid:96) t ( w t ) (cid:105) .

Original: Now we are going to exploit a useful inequality: for any update of the form
Revised: Now we are going to exploit a useful inequality: for any update of the form

Original: ˆ w = arg min w ∈K (cid:104) w, θ (cid:105) + B ψ ( w, v ) , we have that (cid:104)
Revised: ˆ w = arg min w ∈K (cid:104) w, θ (cid:105) + B ψ ( w, v ) , we have that (cid:104)

Original: [Equation]
Revised: [Equation]

Original: ψ ( u
Revised: ψ ( u

Original: [Equation]
Revised: [Equation]

Original: ˆ
Revised: ˆ

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: ψ ( ˆ w
Revised: ψ ( ˆ w

Original: [Equation]
Revised: [Equation]

Original: v ) , (31) for any u ∈ K .
Revised: v ) , (31) for any u ∈ K .

Original: Using the above inequality, we have
Revised: Using the above inequality, we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: So
Revised: So

Original: [Equation]
Revised: [Equation]

Original: by (30) (32) and (33)
Revised: by (30) (32) and (33)

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: Regret T ≤
Revised: Regret T ≤

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: +η t (cid:0)
Revised: +η t (cid:0)

Original: [Equation]
Revised: [Equation]

Original: where the third inequality is because
Revised: where the third inequality is because

Original: [Equation]
Revised: [Equation]

Original: and that ψ t − 1 ( · ) is 1-strongly convex with respect to (cid:107) ·
Revised: and that ψ t − 1 ( · ) is 1-strongly convex with respect to (cid:107) ·

Original: [Equation]
Revised: [Equation]

Original: ψ t − 1 .
Revised: ψ t − 1 .

Original: To proceed, notice that, by deﬁnition of Bregman divergence, we also have
Revised: To proceed, notice that, by deﬁnition of Bregman divergence, we also have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where the vector s t ∈ R d is deﬁned as follows: its i th entry s t [ i ] is
Revised: where the vector s t ∈ R d is deﬁned as follows: its i th entry s t [ i ] is

Original: [Equation]
Revised: [Equation]

Original: Now we have
Revised: Now we have

Original: Regret
Revised: Regret

Original: [Equation]
Revised: [Equation]

Original: Therefore, we conclude the following theorem.
Revised: Therefore, we conclude the following theorem.

Original: Theorem 3.
Revised: Theorem 3.

Original: For any learning rate η t := η , O PTIMISTIC -A DA G RAD (Algorithm 6) has regret
Revised: For any learning rate η t := η , O PTIMISTIC -A DA G RAD (Algorithm 6) has regret

Original: [Equation]
Revised: [Equation]

Original: One need to compare it with the bound of A DA G RAD Duchi et al.
Revised: One need to compare it with the bound of A DA G RAD Duchi et al.

Original: (2011), which has regret
Revised: (2011), which has regret

Original: [Equation]
Revised: [Equation]

Original: T
Revised: T

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We see that when m t predicts ∇ f ( w t ) very well, the last term is dominated by the second one.
Revised: We see that when m t predicts ∇ f ( w t ) very well, the last term is dominated by the second one.

Original: One can set η of
Revised: One can set η of

Original: O PTIMISTIC -A DA G RAD to be a large step size so that O PTIMISTIC -A DA G RAD has a smaller regret than A DA G RAD .
Revised: O PTIMISTIC -A DA G RAD to be a large step size so that O PTIMISTIC -A DA G RAD has a smaller regret than A DA G RAD .