Original: O PTIMISTIC A CCELERATION FOR O PTIMIZATION
Revised: O PTIMISTIC A CCELERATION FOR O PTIMIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We consider new variants of optimization algorithms.
Revised: We consider new variants of optimization algorithms.

Original: Our algorithms are based on the observation that mini-batch of stochastic gradients in consecutive iterations do not change drastically and consequently may be predictable.
Revised: Our algorithms are based on the observation that mini-batch of stochastic gradients in consecutive iterations do not change drastically and consequently may be predictable.

Original: Inspired by the similar setting in online learning literature called O PTIMISTIC O NLINE LEARNING , we propose two new algorithms, O PTIMISTIC -AMSG RAD and O PTIMISTIC -A DAM that exploit the predictability of gradients.
Revised: Inspired by the similar setting in online learning literature called O PTIMISTIC O NLINE LEARNING , we propose two new optimistic algorithms for AMSGrad and Adam, respectively, by exploiting the predictability of gradients.

Original: O PTIMISTIC -AMSG RAD and O PTIMISTIC -A DAM combine the idea of momentum method, adaptive gradient method, and algorithms in O PTIMISTIC O NLINE LEARNING , which leads to speed up in training deep neural nets in practice.
Revised: The new algorithms combine the idea of momentum method, adaptive gradient method, and algorithms in O PTIMISTIC O NLINE LEARNING , which leads to speed up in training deep neural nets in practice.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Nowadays deep learning has been shown to be very effective in several tasks, from robotics (e.g. Levine et al.
Revised: Nowadays deep learning has been shown to be very effective in several tasks, from robotics (e.g. Levine et al.

Original: computer vision (e.g. He et al.
Revised: computer vision (e.g. He et al.

Original: ; Goodfellow et al.
Revised: ; Goodfellow et al.

Original: , reinforcement learning (e.g. Mnih et al.
Revised: , reinforcement learning (e.g. Mnih et al.

Original: (2013), to natural language processing (e.g. Graves et al.
Revised: (2013), to natural language processing (e.g. Graves et al.

Original: Typically, the model parameters of a state-of-the-art deep neural net is very high-dimensional and the required training data is also in huge size.
Revised: Typically, the model parameters of a state-of-the-art deep neural net is very high-dimensional and the required training data is also in huge size.

Original: Therefore, fast algorithms are necessary for training a deep neural net.
Revised: Therefore, fast algorithms are necessary for training a deep neural net.

Original: To achieve this, there are number of algorithms proposed in recent years, such as AMSG RAD (Reddi et al.
Revised: To achieve this, there are number of algorithms proposed in recent years, such as AMSG RAD (Reddi et al.

Original: , ADAM (Kingma & Ba (2015)), RMSPROP (Tieleman & Hinton (2012)),
Revised: , ADAM (Kingma & Ba (2015)), RMSPROP (Tieleman & Hinton (2012)),

Original: ADADELTA (Zeiler (2012)), and NADAM (Dozat (2016)), etc.
Revised: ADADELTA (Zeiler (2012)), and NADAM (Dozat (2016)), etc.

Original: All the prevalent algorithms for training deep nets mentioned above combines two ideas: the idea of adaptivity in
Revised: All the prevalent algorithms for training deep nets mentioned above combines two ideas: the idea of adaptivity in

Original: A DA G RAD (Duchi et al.
Revised: A DA G RAD (Duchi et al.

Original: ; McMahan & Streeter (2010)) and the idea of momentum as N ESTEROV ’ S M ETHOD (Nesterov (2004)) or the H EAVY BALL method (Polyak (1964)).
Revised: ; McMahan & Streeter (2010)) and the idea of momentum as N ESTEROV ’ S M ETHOD (Nesterov (2004)) or the H EAVY BALL method (Polyak (1964)).

Original: A DA G RAD is an online learning algorithm that works well compared to the standard online gradient descent when the gradient is sparse.
Revised: A DA G RAD is an online learning algorithm that works well compared to the standard online gradient descent when the gradient is sparse.

Original: The update of A DA G RAD has a notable feature: the learning rate is different for different dimensions, depending on the magnitude of gradient in each dimension, which might help in exploiting the geometry of data and leading to a better update.
Revised: The update of A DA G RAD has a notable feature: the learning rate is different for different dimensions, depending on the magnitude of gradient in each dimension, which might help in exploiting the geometry of data and leading to a better update.

Original: On the other hand, N ESTEROV ’ S M ETHOD or the Momentum Method (Polyak (1964)) is an accelerated optimization algorithm whose update not only depends on the current iterate and current gradient but also depends on the past gradients (i.e. momentum).
Revised: On the other hand, N ESTEROV ’ S M ETHOD or the Momentum Method (Polyak (1964)) is an accelerated optimization algorithm whose update not only depends on the current iterate and current gradient but also depends on the past gradients (i.e. momentum).

Original: State-of-the-art algorithms like AMSG RAD (Reddi et al.
Revised: State-of-the-art algorithms like AMSG RAD (Reddi et al.

Original: and ADAM (Kingma & Ba (2015)) leverages these two ideas to get fast training for neural nets.
Revised: and ADAM (Kingma & Ba (2015)) leverages these two ideas to get fast training for neural nets.

Original: In this paper, we propose an algorithm that goes further than the hybrid of the adaptivity and momentum approach.
Revised: In this paper, we propose an algorithm that goes further than the hybrid of the adaptivity and momentum approach.

Original: Our algorithm is inspired by O PTIMISTIC O NLINE LEARNING (Chiang et al.
Revised: Our algorithm is inspired by O PTIMISTIC O NLINE LEARNING (Chiang et al.

Original: ; Rakhlin & Sridharan (2013);
Revised: ; Rakhlin & Sridharan (2013);

Original: Syrgkanis et al.
Revised: Syrgkanis et al.

Original: ; Abernethy et al.
Revised: ; Abernethy et al.

Original: O PTIMISTIC O NLINE LEARNING considers that a good guess of the loss function in the current round of online learning is available and plays an action by exploiting the good guess.
Revised: O PTIMISTIC O NLINE LEARNING considers that a good guess of the loss function in the current round of online learning is available and plays an action by exploiting the good guess.

Original: By exploiting the guess, those algorithms in O PTIMISTIC
Revised: By exploiting the guess, those algorithms in O PTIMISTIC

Original: O NLINE LEARNING have regret in the form of
Revised: O NLINE LEARNING have regret in the form of

Original: O ( (cid:113)(cid:80)
Revised: O ( (cid:113)(cid:80)

Original: Tt =1 (cid:107) g t − m t (cid:107) ) , where g t is the gradient of loss function in round t and m t is the ”guess” of g t before seeing the loss function in round t (i.e. before getting g t ).
Revised: Tt =1 (cid:107) g t − m t (cid:107) ) , where g t is the gradient of loss function in round t and m t is the ”guess” of g t before seeing the loss function in round t (i.e. before getting g t ).

Original: This kind of regret can be much smaller than O ( √ T ) when one has a good guess m t of g t .
Revised: This kind of regret can be much smaller than O ( √ T ) when one has a good guess m t of g t .

Original: We combine
Revised: We combine

Original: the O PTIMISTIC
Revised: the O PTIMISTIC

Original: O NLINE LEARNING idea with the adaptivity and the momentum ideas to design new algorithms in training deep neural nets, which leads to O PTIMISTIC
Revised: O NLINE LEARNING idea with the adaptivity and the momentum ideas to design new algorithms in training deep neural nets, which leads to O PTIMISTIC

Original: -AMSG RAD and
Revised: -AMSG RAD and

Original: O PTIMISTIC -A DAM .
Revised: O PTIMISTIC -A DAM .

Original: We also provide theoretical analysis of O PTIMISTIC
Revised: We also provide theoretical analysis of O PTIMISTIC

Original: -AMSG RAD 
Revised: -AMSG RAD .

Original: The proposed O PTIMISTICalgorithms not only adapt to the informative dimensions and exhibit momentums but also take advantage of a good guess of the next gradient to facilitate acceleration.
Revised: The proposed O PTIMISTICalgorithms not only adapt to the informative dimensions and exhibit momentums but also take advantage of a good guess of the next gradient to facilitate acceleration

Original: We evaluate our algorithms with (Kingma & Ba (2015)), (Reddi et al.
Revised: We evaluate our algorithms with (Kingma & Ba (

Original:  and (Daskalakis et al.
Revised: , (Reddi et al and (Daskalakis et al

Original: Experiments show that our O PTIMISTIC -algorithms are faster than the baselines.
Revised: Experiments show that our O PTIMISTIC -algorithms are faster than the baselines

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Both AMSG RAD (Reddi et al.
Revised: Both AMSG RAD (Reddi et al.

Original: and ADAM (Kingma & Ba (2015)) are actually O NLINE LEARNING algorithms.
Revised: and ADAM (Kingma & Ba (2015)) are actually O NLINE LEARNING algorithms.

Original: They use REGRET ANALYSIS to provide some theoretical guarantees of the algorithms.
Revised: They use REGRET ANALYSIS to provide some theoretical guarantees of the algorithms.

Original: Since one can convert an online learning algorithm to an ofﬂine optimization algorithm by online-to-batch conversion (Cesa-Bianchi et al.
Revised: Since one can convert an online learning algorithm to an ofﬂine optimization algorithm by online-to-batch conversion (Cesa-Bianchi et al.

Original: (2004)), one can design an ofﬂine optimization algorithm by designing and analyzing its counterpart in online learning.
Revised: (2004)), one can design an ofﬂine optimization algorithm by designing and analyzing its counterpart in online learning.

Original: Therefore, we would like to give a brief review of O NLINE LEARNING and O PTIMISTIC -O NLINE LEARNING .
Revised: Therefore, we would like to give a brief review of O NLINE LEARNING and O PTIMISTIC -O NLINE LEARNING .

Original: 2. B RIEF REVIEW OF ONLINE LEARNING
Revised: 2. B RIEF REVIEW OF ONLINE LEARNING

Original: In the typical setting of online learning, there is a LEARNER playing an action and then receiving a loss function in each round t .
Revised: In the typical setting of online learning, there is a LEARNER playing an action and then receiving a loss function in each round t .

Original: Speciﬁcally, the learner plays an action w t ∈ K in round t , where w t is chosen in a compact and convex set K ⊆ R n , known as the DECISION SPACE .
Revised: Speciﬁcally, the learner plays an action w t ∈ K in round t , where w t is chosen in a compact and convex set K ⊆ R n , known as the DECISION SPACE .

Original: Then, the learner sees the LOSS FUNCTION (cid:96) t ( · ) and suffers loss (cid:96) t ( w t ) for the choice.
Revised: Then, the learner sees the LOSS FUNCTION (cid:96) t ( · ) and suffers loss (cid:96) t ( w t ) for the choice.

Original: No distributional assumption is made on the loss functions sequence { (cid:96) 1 ( · ) , (cid:96) 2 ( · ) , . . .
Revised: No distributional assumption is made on the loss functions sequence { (cid:96) 1 ( · ) , (cid:96) 2 ( · ) , . . .

Original: , (cid:96) T ( · ) } in ONLINE LEARNING .
Revised: , (cid:96) T ( · ) } in ONLINE LEARNING .

Original: Namely, the loss functions can be adversarial.
Revised: Namely, the loss functions can be adversarial.

Original: The goal of an online learner is minimizing its
Revised: The goal of an online learner is minimizing its

Original: REGRET , which is
Revised: REGRET , which is

Revised: Regret T :

Original: [Equation]
Revised: [Equation]

Revised: (cid:88)

Revised: [Equation]

Revised: (cid:96)

Revised: [Equation]

Revised: [Equation]

Revised: (cid:88) t =

Revised: [Equation]

Revised: (cid:96)

Revised: [Equation]

Revised: ( w ) .

Original: We can also deﬁne AVERAGE REGRET as Regret T :=
Revised: We can also deﬁne AVERAGE REGRET as Regret T :=

Original: Regret T T , which is REGRET divided by number of rounds T .
Revised: Regret T T , which is REGRET divided by number of rounds T .

Original: In
Revised: In

Original: O NLINE LEARNING literature, N O - REGRET ALGORITHMS means online learning algorithms satisfying Regret T →as T → ∞ .
Revised: O NLINE LEARNING literature, N O - REGRET ALGORITHMS means online learning algorithms satisfying Regret T →as T → ∞ .

Original: 2. B RIEF REVIEW OF O PTIMISTIC O NLINE LEARNING
Revised: 2. B RIEF REVIEW OF O PTIMISTIC O NLINE LEARNING

Original: In recent years, there is a branch of works in the paradigm of O PTIMISTIC O NLINE LEARNING (e.g. Chiang et al.
Revised: In recent years, there is a branch of works in the paradigm of O PTIMISTIC O NLINE LEARNING (e.g. Chiang et al.

Original: (2012); Rakhlin & Sridharan (2013); Syrgkanis et al.
Revised: (2012); Rakhlin & Sridharan (2013); Syrgkanis et al.

Original: ; Abernethy et al.
Revised: ; Abernethy et al.

Original: The idea of O PTIMISTIC
Revised: The idea of O PTIMISTIC

Original: O NLINE LEARNING is as follows.
Revised: O NLINE LEARNING is as follows.

Original: Suppose that, in each round t , the learner has a good guess m t ( · ) of the loss function (cid:96) t ( · ) before playing an action w t .
Revised: Suppose that, in each round t , the learner has a good guess m t ( · ) of the loss function (cid:96) t ( · ) before playing an action w t .

Original: (Recall that the learner receives the loss function after the learner commits an action!)
Revised: (Recall that the learner receives the loss function after the learner commits an action!)

Original: Then, the learner should exploit the guess m t ( · ) to choose an action w t , as m t ( · ) is close to the true loss function (cid:96) t ( · ) .
Revised: Then, the learner should exploit the guess m t ( · ) to choose an action w t , as m t ( · ) is close to the true loss function (cid:96) t ( · ) .

Original: For example, (Syrgkanis et al.
Revised: For example, (Syrgkanis et al.

Original: (2015)) proposed an optimistic-variant of the so called F OLLOW - THE -R EGULARIZEDL EADER (FTRL).
Revised: (2015)) proposed an optimistic-variant of the so called F OLLOW - THE -R EGULARIZEDL EADER (FTRL).

Original: The update rule of FTRL (see e.g. Hazan (2016)) is
Revised: The update rule of FTRL (see e.g. Hazan (2016)) is

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) where R ( · ) is a 1 -strong convex function with respect to a norm ( (cid:107) · (cid:107) ) on the constraint set K , L t − 1 := (cid:80) t − 1 s =1 g s is the cumulative sum of gradient vectors of the convex loss functions (i.e. g s :=
Revised: where R ( · ) is a 1 -strong convex function with respect to a norm ( (cid:107) · (cid:107) ) on the constraint set K , L t − 1 := (cid:80) t − 1 s =1 g s is the cumulative sum of gradient vectors of the convex loss functions (i.e. g s :=

Original: ∇ (cid:96) s ( w s ) )
Revised: ∇ (cid:96) s ( w s ) )

Original: up to but not including t , and η is a parameter.
Revised: up to but not including t , and η is a parameter.

Original: The O PTIMISTIC -FTRL of (Syrgkanis et al.
Revised: The O PTIMISTIC -FTRL of (Syrgkanis et al.

Original: has update
Revised: has update

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) where m t is the learner’s guess of the gradient vector g t : = ∇ (cid:96) t ( w t ) .
Revised: where m t is the learner’s guess of the gradient vector g t := ∇ (cid:96) t ( w t ) .

Original: Under the assumption that loss functions are convex, the regret of O PTIMISTIC -FTRL satisﬁes Regret T ≤ O ( (cid:113)(cid:80) Tt =1 (cid:107) g t − m t (cid:107) ∗ ) , which can be much smaller than O ( √ T ) of FTRL if m t is close to g t .
Revised: Under the assumption that loss functions are convex, the regret of O PTIMISTIC -FTRL satisﬁes Regret T ≤ O ( (cid:113) (cid:80) Tt =1 (cid:107) g t − m t (cid:107) ∗ ) , which can be much smaller than O ( √ T ) of FTRL if m t is close to g t .

Original: Consequently, O PTIMISTIC -FTRL will have much smaller regret than
Revised: Consequently, O PTIMISTIC -FTRL will have much smaller regret than

Original: FTRL.
Revised: FTRL.

Original: On the other hand, if m t is far from g t , then the regret of O PTIMISTIC -FTRL would be a constant factor worse than that of FTRL without optimistic update.
Revised: On the other hand, if m t is far from g t , then the regret of O PTIMISTIC -FTRL would be a constant factor worse than that of FTRL without optimistic update.

Original: In the later section, we provide a way to get m t .
Revised: In the later section, we provide a way to get m t .

Original: Here, we just want to emphasize the importance of leveraging a good guess m t for updating w t to get a fast convergence rate (or equivalently, small regret).
Revised: Here, we just want to emphasize the importance of leveraging a good guess m t for updating w t to get a fast convergence rate (or equivalently, small regret).

Original: We also note that the works of O PTIMISTIC O NLINE LEARNING Chiang et al.
Revised: We also note that the works of O PTIMISTIC O NLINE LEARNING Chiang et al.

Original: Rakhlin & Sridharan (2013); Syrgkanis et al.
Revised: Rakhlin & Sridharan (2013); Syrgkanis et al.

Original: (2015)) has been shown to accelerate the convergence of some zero-sum games.
Revised: (2015)) has been shown to accelerate the convergence of some zero-sum games.

Original: 1: Required: do 4:
Revised: A DAM (Kingma & Ba (2015)) is a very popular algorithm for training deep nets. It combines the momentum idea Polyak (1964) with the idea of A DA G RAD (Duchi et al.

Original: Get mini-batch stochastic gradient vector g t ∈ R d at w t .
Revised: (2011)), which has individual learning rate for different dimensions. The learning rate of A DA G RAD in iteration t for a dimension j is proportional to the inverse of (cid:112) Σ ts = 1 g s [ j ] 2 ,

Original: 5: θ t = β 1 θ t − 1 + (1 − β 1 ) g t .
Revised: 5: θ t = β 1 θ t − 1 + (1 − β 1 ) g t .

Original: 6: v t = β 2 v t − 1 + (1 − β 2 ) g 2 t .
Revised: 6: v t = β 2 v t − 1 + (1 − β 2 ) g 2 t .

Original: 7: ˆ v t =
Revised: 7: ˆ v t =

Original: max(ˆ v t − 1 , v t ) .
Revised: max(ˆ v t − 1 , v t ) .

Original: 8: w t +1 =
Revised: 8: w t +1 =

Original: w t − η t θ t √ ˆ v t .
Revised: w t − η t θ t √ ˆ v t .

Original: 9: end

Original: A DAM AND AMSG RAD

Original: A DAM (Kingma & Ba (2015)) is a very popular algorithm for training deep nets.

Original: It combines the momentum idea

Original: Polyak (1964) with the idea of A DA G RAD (Duchi et al.

Original: (2011)), which has individual learning rate for different dimensions.

Original: The learning rate of A DA G RAD in iteration t for a dimension j is proportional to the inverse of (cid:112) Σ ts =

Original: 1 g s [ j ] 2 , where g s [ j ] is the j th element of the gradient vector g s in time s .
Revised: 9: end for where g s [ j ] is the j th element of the gradient vector g s in time s .

Original: This adaptive learning rate may help for accelerating the convergence when the gradient vector is sparse (Duchi et al.
Revised: This adaptive learning rate may help for accelerating the convergence when the gradient vector is sparse (Duchi et al.

Original: However, when applying A DA G RAD to train deep nets, it is observed that the learning rate might decay too fast (Kingma & Ba (2015)).
Revised: However, when applying A DA G RAD to train deep nets, it is observed that the learning rate might decay too fast (Kingma & Ba (2015)).

Original: Therefore, (Kingma &
Revised: Therefore, (Kingma &

Original: Ba (2015)) proposes using a moving average of gradients (element-wise) divided by the root of the second moment of the moving average to update model parameter w (i.e. line 5,6 and line 8 of Algorithm 1).
Revised: Ba (2015)) proposes using a moving average of gradients (element-wise) divided by the root of the second moment of the moving average to update model parameter w (i.e. line 5,6 and line 8 of Algorithm 1).

Original: Yet, A DAM (Kingma & Ba (2015)) fails at some online convex optmization problems.
Revised: Yet, A DAM (Kingma & Ba (2015)) fails at some online convex optmization problems

Original: AMSGRAD (Reddi et al. (2018)) ﬁxes the issue.
Revised: AMSG

Original: The algorithm of AMSG RAD is shown in Algorithm 1.
Revised: The algorithm of AMSG RAD is shown in Algorithm 1.

Original: The difference between A DAM and AMSG RAD lies on line 7 of Algorithm 1.
Revised: The difference between A DAM and AMSG RAD lies on line 7 of Algorithm 1.

Original: A DAM does not have the update of line 7.
Revised: A DAM does not have the update of line 7.

Original: (Reddi et al.
Revised: (Reddi et al.

Original: (2018)) adds the step to guarantee a nonincreasing learning rate, η t √ ˆ v t .
Revised: (2018)) adds the step to guarantee a nonincreasing learning rate, η t √ ˆ v t .

Original: which helps for the convergence (i.e. average regret Regret T → 0 .)
Revised: which helps for the convergence (i.e. average regret Regret T → 0 .)

Original: For the parameters of AMSG RAD , it is suggested that β 1 = 0 .
Revised: For the parameters of AMSG RAD , it is suggested that β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 99 and η t = η/ √ t for a number η .
Revised: 99 and η t = η/ √ t for a number η .

Original:  O PTIMISTIC -AMSG RAD
Revised:  O PTIMISTIC -AMSG RAD

Original: In this section, we propose a new algorithm for training deep nets: O PTIMISTIC -AMSG RAD , shown in Algorithm 2.
Revised: In this section, we propose a new algorithm for training deep nets: O PTIMISTIC -AMSG RAD , shown in Algorithm 2.

Original: O PTIMISTIC -AMSG RAD has an optimistic update, which is line 9 of Algorithm 2.
Revised: O PTIMISTIC -AMSG RAD has an optimistic update, which is line 9 of Algorithm 2.

Original: It exploits the guess m t +1 of g t + to get w t +1 , since the vector h t +1 uses m t +1 .
Revised: It exploits the guess m t +1 of g t + to get w t +1 , since the vector h t +1 uses m t +1 .

Original: Notice that the gradient vector is computed at w t instead of w t − 12 and the moving average of gradients is used to update w t
Revised: Notice that the gradient vector is computed at w t instead of w t − 12 and the moving average of gradients is used to update w t

Original: + 12 .
Revised: + 12 .

Original: One might want to combining line 8 and line 9 and getting a single line: w t +1 =
Revised: One might want to combining line 8 and line 9 and getting a single line: w t +1 =

Original: w t − 12 − η t θ t √ ˆ v t − η t +1 4 1 − β 1 h t +1 √ ˆ v t .
Revised: w t − 12 − η t θ t √ ˆ v t − η t +1 4 1 − β 1 h t +1 √ ˆ v t .

Original: From this, we see that w t +1 is updated from w t − 12 instead of w t .
Revised: From this, we see that w t +1 is updated from w t − 12 instead of w t .

Original: Therefore, while O PTIMISTIC -AMSG RAD looks like just doing an additional update compared to AMSG RAD , the difference of update is subtle.
Revised: Therefore, while O PTIMISTIC -AMSG RAD looks like just doing an additional update compared to AMSG RAD , the difference of update is subtle.

Original: We also want to emphasize that although the learning rate on line 9 contains 4 1 − β 1 factor.
Revised: We also want to emphasize that although the learning rate on line 9 contains [Table] factor.

Original: We suspect that it is due to the artifact of our theoretical analysis.
Revised: We suspect that it is due to the artifact of our theoretical analysis.

Original: In our experiments, the learning rate on line 9 does not have the factor of 4 1 − β 1 .
Revised: In our experiments, the learning rate on line 9 does not have the factor of 4 1 − β 1 .

Original: That is, in practice, we implement line 9 as w t
Revised: That is, in practice, we implement line 9 as w t

Original: +1 =
Revised: +1 =

Original: w t + 12 − η t
Revised: w t + 12 − η t

Original: +1 h t +1 √ ˆ v t .
Revised: +1 h t +1 √ ˆ v t .

Original: We leave closing the gap between theory and practice as a future work.
Revised: We leave closing the gap between theory and practice as a future work.

Original: We see that O PTIMISTIC
Revised: We see that O PTIMISTIC

Original: -AMSG RAD inherits three properties
Revised: -AMSG RAD inherits three properties

Original: • Adaptive learning rate of each dimension as A DA G RAD (Duchi et al.
Revised: • Adaptive learning rate of each dimension as A DA G RAD (Duchi et al.

Original: (line 8) • Exponentially moving average of the past gradients as N ESTEROV ’ S METHOD (Nesterov (2004)) and the H EAVY -B
Revised: (line 8) • Exponentially moving average of the past gradients as N ESTEROV ’ S METHOD (Nesterov (2004)) and the H EAVY -B

Original: ALL method (Polyak (1964)).
Revised: ALL method (Polyak (1964)).

Original: (line 5) • Optimistic update that exploits a good guess of the next gradient vector as optimistic online learning algorithms (Chiang et al.
Revised: (line 5) • Optimistic update that exploits a good guess of the next gradient vector as optimistic online learning algorithms (Chiang et al.

Original: ; Rakhlin & Sridharan (2013); Syrgkanis et al. (2015); Abernethy et al.
Revised: ; Rakhlin & Sridharan (2013); Syrgkanis et al. (2015); Abernethy et al.

Original: (line9)
Revised: (line

Original: The ﬁrst property helps acceleration when the gradient has sparse structure.
Revised: The ﬁrst property helps acceleration when the gradient has sparse structure.

Original: The second one is the well-recognized idea of momentum which can achieve acceleration.
Revised: The second one is the well-recognized idea of momentum which can achieve acceleration.

Original: The last one, perhaps less known outside the O NLINE LEARNING community, can actually achieve acceleration when the prediction of the next gradient is good.
Revised: The last one, perhaps less known outside the O NLINE LEARNING community, can actually achieve acceleration when the prediction of the next gradient is good.

Original: We are going to elaborate this property in the later section where we give the theoretical analysis of O PTIMISTIC
Revised: We are going to elaborate this property in the later section where we give the theoretical analysis of O PTIMISTIC

Original: -AMSG RAD .
Revised: -AMSG RAD .

Original: 1: Required: parameter β 1 , β 2 , and η t .
Revised: To obtain m t , we use the extrapolation algorithm of (Scieur et al.

Original: 2: Init: w 1 .
Revised: Extrapolation studies estimating the limit of sequence using the last few iterates (Brezinski & Zaglia (2013)). Some classical works include Anderson acceleration (Walker & Ni. (2011)), minimal polynomial extrapolation (Cabay & Jackson (1976)), reduced rank extrapolation (Eddy (1979)).

Original: 3: for t = 1 to T do 4: Get mini-batch stochastic gradient vector g t  ∈ R d at w t .
Revised: These method typically assumes that the sequence { x t } ∈ R d has a linear relation

Revised: [Equation]

Original: 5: θ t = β 1 θ t − 1 + (1 − β 1 ) g t .
Revised: 5: θ t = β 1 θ t − 1 + (1 − β 1 ) g t .

Original: 6: v t = β 2 v t − 1 + (1 − β 2 ) g 2 t .
Revised: 6: v t = β 2 v t − 1 + (1 − β 2 ) g 2 t .

Original: 7: ˆ v t =
Revised: 7: ˆ v t =

Original: max(ˆ v t − 1 , v t ) .
Revised: max(ˆ v t − 1 , v t ) .

Original: 8: w t + 12 =
Revised: 8: w t + 12 =

Original: w t − 12 − η t θ t √ ˆ v t .
Revised: w t − 12 − η t θ t √ ˆ v t .

Original: (element-wise division) 9: w t +1 =
Revised: (element-wise division) 9: w t +1 =

Original: w t + 12 − η t +1 4 1 − β 1 h t +1 √ ˆ v t , where h t +1 :
Revised: w t + 12 − η t +1 4 1 − β 1 h t +1 √ ˆ v t , where h t +1 :

Original: = β 1 θ t − 1 + (1 − β 1 ) m t + and m t +1 is the guess of g t +1 .
Revised: = β 1 θ t − 1 + (1 − β 1 ) m t + and m t +1 is the guess of g t +1 .

Original: (In practice, use w t +1 =
Revised: (In practice, use w t +1 =

Original: w t + 12 − η t
Revised: w t + 12 − η t

Original: +1 h t +1 √ ˆ v t .)
Revised: +1 h t +1 √ ˆ v t .)

Original: 10: end for
Revised: : end for

Original: (Scieur

Original: 1: Input: some sequence { x s ∈ R d } s = r s =0 , parameter λ >2:

Original: Compute matrix U = [ x 1 − x 0 , . . . ,  x r − x r − 1 ] ∈ R d × r 3: Obtain z by solving ( U (cid:62) U + λI ) z
Revised: ( U (cid:62) U λI ) i x i , the approximation of the ﬁxed point x ∗ for an unknown matrix A ∈ R d × d (not necessarily symmetric).

Original: = 1 .

Original: [-Get c = z/ ( z (cid:62) 1 ) 5: Output: Σ r − 1 i =
-]
Revised: [-Get c = z/ ( z (cid:62) 1 ) 5: Output: Σ r − 1 i =
-]

Original: c i x i , the approximation of the ﬁxed point x ∗ .

Original: To obtain m t , we use the extrapolation algorithm of (Scieur et al.

Original: Extrapolation studies estimating the limit of sequence using the last few iterates (Brezinski & Zaglia (2013)).

Original: Some classical works include Anderson acceleration (Walker & Ni.

Original: (2011)), minimal polynomial extrapolation (Cabay & Jackson (1976)), reduced rank extrapolation (Eddy (1979)).

Original: These method typically assumes that the sequence { x t } ∈ R d has a linear relation

Original: [Equation]

Original: for an unknown matrix A ∈ R d × d (not necessarily symmetric).

Original: The goal is to use the last few iterates { x t } to estimate the ﬁxed point x ∗ on (4).
Revised: The goal is to use the last few iterates { x t } to estimate the ﬁxed point x ∗ on (4).

Original: (Scieur et al.
Revised: (Scieur et al.

Original: (2016)) adapt the classical extrapolation methods to the iterates/updates of an optimization algorithm and propose an algorithm that produces a solution that is better than the last iterate of the underlying optimization algorithm in practice.
Revised: (2016)) adapt the classical extrapolation methods to the iterates/updates of an optimization algorithm and propose an algorithm that produces a solution that is better than the last iterate of the underlying optimization algorithm in practice.

Original: The algorithm of (Scieur et al.
Revised: The algorithm of (Scieur et al.

Original: (shown in Algorithm 3) allows the iterates { x t } to be nonlinear
Revised: (shown in Algorithm 3) allows the iterates { x t } to be nonlinear

Original: [Equation]
Revised: [Equation]

Original: where e t is a second order term (namely, satisfying (cid:107) e t (cid:107) 

Revised: where e t is a second order term (namely, satisfying (cid:107) e t (cid:107)

Original: = O ( (cid:107) x
Revised: = O ( (cid:107) x

Original: t − 1 − x ∗ (cid:107) 22 ) .
Revised: t − 1 − x ∗ (cid:107) 22 ) .

Original: Some theoretical guarantees regarding the distance between the output and x ∗ is provided in (Scieur et al.
Revised: Some theoretical guarantees regarding the distance between the output and x ∗ is provided in (Scieur et al.

Original: In O PTIMISTIC -AMSG RAD , we use Algorithm 3 to get m t .
Revised: In O PTIMISTIC -AMSG RAD , we use Algorithm 3 to get m t .

Original: Speciﬁcally, m t is obtained by
Revised: Speciﬁcally, m t is obtained by

Original: • Call Algorithm 3 with input being a sequence of some past r + 1 gradients, { g t , g t − 1 , g t − 2 , . .
Revised: • Call Algorithm 3 with input being a sequence of some past r + 1 gradients, { g t , g t − 1 , g t − 2 , . .

Original: . , g t − r } to obtain m t , where r is a parameter.
Revised: . , g t − r } to obtain m t , where r is a parameter.

Original: • Set
Revised: • Set

Original: m t := Σ r − 1 i =

Revised: m t := Σ r − 1 i =

Original: c i g t − r
Revised: c i g t − r

Original: + i from the output of Algorithm 3.
Revised: + i from the output of Algorithm 3.

Original: If the past few gradients can be modeled by (4) approximately, the extrapolation method should be expected to work well in predicting the gradient.
Revised: If the past few gradients can be modeled by (4) approximately, the extrapolation method should be expected to work well in predicting the gradient.

Original: In practice, it helps to achieve faster convergence.
Revised: In practice, it helps to achieve faster convergence.

Original: O PTIMISTIC -A DAM
Revised: O PTIMISTIC -A DAM

Original: By removing line 7 in Algorithm 2, the step of making monotone weighted second moment, we obtain an algorithm which we call it O PTIMISTIC -A DAM , as the resulting algorithm can be viewed as an O PTIMISTICvariant of A DAM .
Revised: By removing line 7 in Algorithm 2, the step of making monotone weighted second moment, we obtain an algorithm which we call it O PTIMISTIC -A DAM , as the resulting algorithm can be viewed as an O PTIMISTICvariant of A DAM .

Original: 3. T HEORETICAL ANALYSIS OF O PTIMISTIC -AMSG RAD
Revised: 3. T HEORETICAL ANALYSIS OF O PTIMISTIC -AMSG RAD

Original: We provide the regret analysis here.
Revised: We provide the regret analysis here.

Original: We denote the Mahalanobis norm (cid:107) · (cid:107) H = (cid:112) (cid:104)· , H ·(cid:105) for some PSD matrix
Revised: We denote the Mahalanobis norm (cid:107) · (cid:107) H = (cid:112) (cid:104)· , H ·(cid:105) for some PSD matrix

Original: H .
Revised: H .

Original: For the PSD matrix diag { ˆ v t } , where diag { ˆ v t } represents the diagonal matrix such that its i th diagonal element is ˆ v t
Revised: For the PSD matrix diag { ˆ v t } , where diag { ˆ v t } represents the diagonal matrix such that its i th diagonal element is ˆ v t

Original: [ i ] in Algorithm 2, we deﬁne the the corresponding Mahalanobis norm (cid:107) · (cid:107) ψ t :=
Revised: [ i ] in Algorithm 2, we deﬁne the the corresponding Mahalanobis norm (cid:107) · (cid:107) ψ t :=

Original: (cid:112) (cid:104)· , diag { ˆ v t } 1 / 2 ·(cid:105) , where we use the notation ψ t to represent the matrix diag { ˆ v t } 1 / 2 .
Revised: (cid:112) (cid:104)· , diag { ˆ v t } 1 / 2 ·(cid:105) , where we use the notation ψ t to represent the matrix diag { ˆ v t } 1 / 2 .

Original: We can also deﬁne the the corresponding dual norm (cid:107) · (cid:107) ψ ∗ t :=
Revised: We can also deﬁne the the corresponding dual norm (cid:107) · (cid:107) ψ ∗ t :=

Original: (cid:112) (cid:104)· , diag { ˆ v t } − 1 / 2 ·(cid:105) .
Revised: (cid:112) (cid:104)· , diag { ˆ v t } − 1 / 2 ·(cid:105) .

Original: We assume that the model parameter w is in d -dimensional space.
Revised: We assume that the model parameter w is in d -dimensional space.

Original: That is, w ∈ R d .
Revised: That is, w ∈ R d .

Original: Also, the analysis of O PTIMISTICAMSG RAD is for unconstrained optimization.
Revised: Also, the analysis of O PTIMISTICAMSG RAD is for unconstrained optimization.

Original: Thus, we assume that the constraint K of the benchmark in the regret deﬁnition, min w ∈K (cid:80) Tt =1 (cid:96) t ( w ) , is a ﬁnite norm ball that contains the optimal solutions to the underlying ofﬂine unconstrained optimization problem.
Revised: Thus, we assume that the constraint K of the benchmark in the regret deﬁnition, min w ∈K (cid:80) Tt =1 (cid:96) t ( w ) , is a ﬁnite norm ball that contains the optimal solutions to the underlying ofﬂine unconstrained optimization problem.

Original: Now we can conduct our analysis.
Revised: Now we can conduct our analysis.

Original: First of all, we can decompose the regret as follows.
Revised: First of all, we can decompose the regret as follows.

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality is by assuming that the loss function (cid:96) t ( · ) is convex and that we use the notation g t := ∇ (cid:96) t ( w t ) which we adopt throughout the following proof for brevity.
Revised: where the ﬁrst inequality is by assuming that the loss function (cid:96) [Equation] ) is convex and that we use the notation g t := ∇ (cid:96) t ( w t ) which we adopt throughout the following proof for brevity.

Original: Given the decomposition, let us analyze the ﬁrst term (cid:80)
Revised: Given the decomposition, let us analyze the ﬁrst term (cid:80)

Original: Tt =1 (cid:104) w t − 12 − w ∗ , g t (cid:105) in (6).
Revised: Tt =1 (cid:104) w t − 12 − w ∗ , g t (cid:105) in (6).

Original: Lemma 1.
Revised: Lemma 1.

Original: Denote D ∞ =
Revised: Denote D ∞ =

Original: max t (cid:107) w t − 12 − w ∗ (cid:107) ∞ .
Revised: max t (cid:107) w t − 12 − w ∗ (cid:107) ∞ .

Original: We have that (cid:80)
Revised: We have that (cid:80)

Revised: Tt =1 (cid:104) w t − 12 − w

Original: [Equation]
Revised: [Equation]

Revised: g

Original:  Astute readers may realize that the bound in Lemma 1 is actually the bound of AMS GRAD .
Revised: [Equation] ≤ [Equation] Astute readers may realize that the bound in Lemma 1 is actually the bound of AMS GRAD .

Original: Indeed, since in online learning setting the loss vectors g t come adversarially, it does matter how g t is generated.
Revised: Indeed, since in online learning setting the loss vectors g t come adversarially, it does matter how g t is generated.

Original: Therefore, the regret of (cid:80)
Revised: Therefore, the regret of (cid:80)

Original: Tt =1 (cid:104) w t − 12 − w ∗ , g t (cid:105) can be bounded in the same way as AMS GRAD .
Revised: Tt =1 (cid:104) w t − 12 − w ∗ , g t (cid:105) can be bounded in the same way as AMS GRAD .

Original: In Appendix B, we provide the detail proof of Lemma 1.
Revised: In Appendix B, we provide the detail proof of Lemma 1.

Original: Now we switch to bound the other sums (cid:80)
Revised: Now we switch to bound the other sums (cid:80)

Original: Tt =1 (cid:104) w t − w t − 12 , g t − h t (cid:105) + (cid:80)
Revised: Tt =1 (cid:104) w t − w t − 12 , g t − h t (cid:105) + (cid:80)

Original: Tt =1 (cid:104) w t − w t − 12 , h t (cid:105) in (6).
Revised: Tt =1 (cid:104) w t − w t − 12 , h t (cid:105) in (6).

Original: The proof is available in Appendix C.
Revised: The proof is available in Appendix C.

Original: Lemma 2.
Revised: Lemma 2.

Original: (cid:80) Tt =1 (cid:104) w t − w t − 12 , g [Equation]
Revised: [Equation] − w t − 12 (cid:107) 2 1 − β 1 8 ψ t − 1 .

Original: Combining (6) and Lemma 1 and 2 leads to
Revised: Combining (6) and Lemma 1 and 2 leads to

Revised: Regret T ≤

Revised: [Equation]

Revised: (cid:88)

Revised: [Equation]

Revised: v

Original: [Equation]
Revised: [Equation]

Revised: η

Revised: [Equation]

Revised: [Equation]

Original: Now we can conclude the following theorem.
Revised: Now we can conclude the following theorem.

Original: The proof is in Appendix D.
Revised: The proof is in Appendix D.

Original: Theorem 1.
Revised: Theorem 1.

Original: Denote γ :=
Revised: Denote γ :=

Original: β 1 / √ β 2 < 1 and D ∞ =
Revised: β 1 / √ β 2 < 1 and D ∞ =

Original: max t (cid:107) w t − 12 − w ∗ (cid:107) ∞ .
Revised: max t (cid:107) w t − 12 − w ∗ (cid:107) ∞ .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: One should compare the bound with that of AMSG RAD (Reddi et al.
Revised: One should compare the bound with that of AMSG RAD (Reddi et al.

Original: (2018)), which is
Revised: (2018)), which is

Original: [Equation]
Revised: [Equation]

Original: where η t = η/ √ t in their setting.
Revised: where η t = η/ √ t in their setting.

Original: We need to compare the last two terms in (8) with the last term in (9).
Revised: We need to compare the last two terms in (8) with the last term in (9).

Original: We are going to show that, under certain conditions, the bound is smaller than that of AMSG RAD .
Revised: We are going to show that, under certain conditions, the bound is smaller than that of AMSG RAD .

Original: Let us suppose that g t is close to m t so that (cid:80) Tt
Revised: Let us suppose that g t is close to m t so that (cid:80) Tt

Original: =1 2 η t (1 − β 1 ) (cid:107) g t − m t (cid:107) 2 ψ ∗ t is much smaller than the last term of (8).
Revised: =1 2 η t (1 − β 1 ) (cid:107) g t − m t (cid:107) 2 ψ ∗ t is much smaller than the last term of (8).

Original: Yet, the last term of (8), (cid:80) Tt =1 η t 2 (cid:107) g t − h t (cid:107) ( 1 − β 1 8 ψ t − 1 ) ∗ , might be actually o ( √ T ) and consequently might also be smaller than the last term of (9).
Revised: Yet, the last term of (8), (cid:80) Tt =1 η t 2 (cid:107) g t − h t (cid:107) ( 1 − β 1 8 ψ t − 1 ) ∗ , might be actually o ( √ T ) and consequently might also be smaller than the last term of (9).

Original: To see this, let us rewrite (cid:80)
Revised: To see this, let us rewrite (cid:80)

Original: Tt =1 η t 2 (cid:107) g t − h t (cid:107) ( 1 − [Equation] 8 ψ t [Equation]
Revised: Tt =1 η t 2 (cid:107) g t − h t (cid:107) ( 1 − β 1 8 ψ t − 1 ) ∗ and set η t =

Original: 1 ) ∗ and set η t =

Original: η/ √ t as (Reddi et al.
Revised: η/ √ t as (Reddi et al.

Original: ; it is
Revised: ; it is

Original: O (

Original: [Equation]
Revised: [Equation]

Original: T (cid:88) t = d (cid:88) i =

Original: η √ t ( g t [ i ] 2 − h t

Original: [ i ] 2 ) (cid:112)

Original: ˆ v t − 1 [ i ] ) .

Original: Assume that if each ( g t [ i ] 2 − h t
Revised: Assume that if each ( g t [ i ] 2 − h t

Original: [ i ] 2 ) √ ˆ v t − 1 [ i ] in the inner sum is bounded by a constant c , we will get (cid:80) Tt
Revised: [ i ] 2 ) √ ˆ v t − 1 [ i ] in the inner sum is bounded by a constant c , we will get (cid:80) Tt

Original: =1 O ( 1 √ t )
Revised: =1 O ( 1 √ t )

Original: =
Revised: =

Original: O ( √ T ) .
Revised: O ( √ T ) .

Original: Yet, the denominator (cid:112) ˆ v t − 1 is non-decreasing so that we can actually have a smaller bound.
Revised: Yet, the denominator (cid:112) ˆ v t − 1 is non-decreasing so that we can actually have a smaller bound.

Original: That is, in practice, (cid:112)
Revised: That is, in practice, (cid:112)

Original: ˆ v t − 1 [ i ] might grow over time, and the growth rate is different for different dimension i .
Revised: ˆ v t − 1 [ i ] might grow over time, and the growth rate is different for different dimension i .

Original: If (cid:112) ˆ v t − 1 [ i ] in the denominator grows like O ( √ t ) then the last term is just O
Revised: If (cid:112) ˆ v t − 1 [ i ] in the denominator grows like O ( √ t ) then the last term is just O

Original: (log T ) , which might be better than that of the last term on (9).
Revised: (log T ) , which might be better than that of the last term on (9).

Original: One can also get a data dependent bound of the last term of (8).
Revised: One can also get a data dependent bound of the last term of (8).

Original: To summarize, when m t is close to g t ,
Revised: To summarize, when m t is close to g t ,

Original: O PTIMISTIC -AMSG RAD can have a smaller regret (thus better convergence rate) than A DAM (Kingma & Ba (2015)) and AMSG RAD (Reddi et al.
Revised: O PTIMISTIC -AMSG RAD can have a smaller regret (thus better convergence rate) than A DAM (Kingma & Ba (2015)) and AMSG RAD (Reddi et al.

Original: 3. C OMPARISON OF A DAM -DISZ IN D ASKALAKIS ET AL . (2018)
Revised: 3. C OMPARISON OF A DAM -DISZ IN D ASKALAKIS ET AL . (2018)

Original: : θ t = β 1 θ t − 1 + (1 − β 1 ) g t .

Original: 6: v t = β

Original: [Equation]

Original: v

Original: [Equation]

Original: −

Original: [Equation]

Original: + (1 − β 2 ) g 2 t .

Original: 7: w t +1 =

Original: Π k [ w t − 2 η t θ t √ v t + η t

Original: [Table]

Original: 8: end

Original: We are aware of Algorithm 1 in Daskalakis et al.
Revised: We are aware of Algorithm 1 in Daskalakis et al.

Original: (2018), which was also motivated by O PTIMISTIC O NLINE LEARNING 2 .
Revised: (2018), which was also motivated by O PTIMISTIC O NLINE LEARNING 2 .

Original: For comparison, we replicate A DAM -DISZ in Algorithm 4.
Revised: For comparison, we replicate A DAM -DISZ in Algorithm 4.

Original: We are going to describe the differences of the algorithms and the differences of the contributions between our work and their work.
Revised: We are going to describe the differences of the algorithms and the differences of the contributions between our work and their work.

Original: First of all, by comparing
Revised: First of all, by comparing

Original: O PTIMISTIC -AMSG RAD (Algorithm 2) or O PTIMISTIC -A DAM (Algorithm 2 without line 7) with A DAM -DISZ
Revised: O PTIMISTIC -AMSG RAD (Algorithm 2) or O PTIMISTIC -A DAM (Algorithm 2 without line 7) with A DAM -DISZ

Original: (Algorithm 4), we see that the updates are indeed different.
Revised: (Algorithm 4), we see that the updates are indeed different.

Original: The update cannot be written into the same form as our
Revised: The update cannot be written into the same form as our

Original: O PTIMISTIC -AMSG RAD (and vise versa).
Revised: O PTIMISTIC -AMSG RAD (and vise versa).

Original: O PTIMISTIC
Revised: O PTIMISTIC

Original: -AMSG RAD
Revised: -AMSG RAD

Original: (Algorithm 2) actually uses two interleaving sequences of updates { w t } Tt =1 , { w t − 12 } Tt =1 , Also, Daskalakis et al.
Revised: (Algorithm 2) actually uses two interleaving sequences of updates { w t } Tt =1 , { w t − 12 } Tt =1 , Also, Daskalakis et al.

Original: (2018) does not have any theoretical analysis of
Revised: (2018) does not have any theoretical analysis of

Original: A DAM -DISZ.
Revised: A DAM -DISZ.

Original: The theoretical analysis in Daskalakis et al.
Revised: The theoretical analysis in Daskalakis et al.

Original: (2018) is for other algorithms.
Revised: (2018) is for other algorithms.

Original: Second, the contributions are very different.
Revised: Second, the contributions are very different.

Original: Daskalakis et al.
Revised: Daskalakis et al.

Original: (2018) focus on training GANs Goodfellow et al.
Revised: (2018) focus on training GANs Goodfellow et al.

Original: GANs is a two-player zero-sum game.
Revised: GANs is a two-player zero-sum game.

Original: There have been some related works in O PTIMISTIC O NLINE LEARNING like Chiang et al.
Revised: There have been some related works in O PTIMISTIC O NLINE LEARNING like Chiang et al.

Original: Rakhlin & Sridharan (2013); Syrgkanis et al.
Revised: Rakhlin & Sridharan (2013); Syrgkanis et al.

Original: showing that if both players use some kinds of O PTIMISTICupdate, then acceleration to the convergence of the minimax value of the game is possible.
Revised: showing that if both players use some kinds of O PTIMISTICupdate, then acceleration to the convergence of the minimax value of the game is possible.

Original: Daskalakis et al.
Revised: Daskalakis et al.

Original: (2018) was inspired by these related works and showed that O PTIMISTIC -M
Revised: (2018) was inspired by these related works and showed that O PTIMISTIC -M

Original: IRROR -D ESCENT can avoid the cycle behavior in a bilinear zero-sum game, which accelerates the convergence.
Revised: IRROR -D ESCENT can avoid the cycle behavior in a bilinear zero-sum game, which accelerates the convergence.

Original: Our work is about solving min x f ( x ) (e.g. empirical risk) quickly.
Revised: Our work is about solving min x f ( x ) (e.g. empirical risk) quickly.

Original: We also show that ADAM-DISZ suffers the non-convergence issue as A DAM .
Revised: We also show that ADAM-DISZ suffers the non-convergence issue as A DAM .

Original: The proof is available in Appendix E.
Revised: The proof is available in Appendix E.

Original: Theorem 2.
Revised: Theorem 2.

Original: There exists a convex online learning problem such that ADAM-DISZ has nonzero average regret (i.e.
Revised: There exists a convex online learning problem such that ADAM-DISZ has nonzero average regret (i.e.

Original: RegretT (cid:54) =
Revised: RegretT (cid:54) =

Original: o ( T ) )
Revised: o ( T ) )

Original: One might wonder if the non-convergence issue can be avoided if one let the weighted second moment of ADAMDISZ be monotone by adding the step ˆ v t =
Revised: One might wonder if the non-convergence issue can be avoided if one let the weighted second moment of ADAMDISZ be monotone by adding the step ˆ v t =

Original: max(ˆ v t − 1 , v t ) as AMSG RAD , which we call it AMSG RAD -DISZ.
Revised: max(ˆ v t − 1 , v t ) as AMSG RAD , which we call it AMSG RAD -DISZ.

Original: Unfortunately, we are unable to prove if the step guarantees convergence or not.
Revised: Unfortunately, we are unable to prove if the step guarantees convergence or not.

Original:  EXPERIMENTS
Revised:  EXPERIMENTS

Original: To demonstrate the effectiveness of our proposed method, we test its performance with various neural network architectures, including fully-connected neural networks, convolutional neural networks (CNN’s) and recurrent neural networks (RNN’s).
Revised: To demonstrate the effectiveness of our proposed method, we test its performance with various neural network architectures, including fully-connected neural networks, convolutional neural networks (CNN’s) and recurrent neural networks (RNN’s).

Original: The results illustrate that O PTIMISTIC
Revised: The results illustrate that O PTIMISTIC

Original: -AMSG RAD is able to speed up the convergence of stateof-art AMSG RAD algorithm, making the learning process more efﬁcient.
Revised: -AMSG RAD is able to speed up the convergence of stateof-art AMSG RAD algorithm, making the learning process more efﬁcient.

Original: For AMSG RAD algorithm, we set the parameter β 1 and β 2 , respectively, to be 0.9 and 0.999, as recommended in Reddi et al.
Revised: For AMSG RAD algorithm, we set the parameter β 1 and β 2 , respectively, to be 0.9 and 0.999, as recommended in Reddi et al.

Original: We tune the learning rate η over a ﬁne grid and report the results under best-tuned parameter setting.
Revised: We tune the learning rate η over a ﬁne grid and report the results under best-tuned parameter setting.

Original: For
Revised: For

Original: O PTIMISTIC -AMSG RAD and AMSG
Revised: O PTIMISTIC -AMSG RAD and AMSG

Original: RAD -DISZ, we use same β 1 , β 2 and learning rate as those for AMSG RAD to make a fair comparison of the enhancement brought by the optimistic step.
Revised: RAD -DISZ, we use same β 1 , β 2 and learning rate as those for AMSG RAD to make a fair comparison of the enhancement brought by the optimistic step.

Original: We use the same weight initialization for all algorithms.
Revised: We use the same weight initialization for all algorithms.

Original: The remaining tuning parameter of O PTIMISTIC -AMSG RAD is r , the number of previous gradients that we use to predict the next move.
Revised: The remaining tuning parameter of O PTIMISTIC -AMSG RAD is r , the number of previous gradients that we use to predict the next move.

Original: We conduct O PTIMISTIC
Revised: We conduct O PTIMISTIC

Original: -AMSG RAD with different values of r and observe similar performance (See Appendix A).
Revised: -AMSG RAD with different values of r and observe similar performance (See Appendix A).

Original: Hence, we report r = 15 for all experiments for tidiness of plots.
Revised: Hence, we report r = 15 for all experiments for tidiness of plots.

Original: To follow previous works of Reddi et al.
Revised: To follow previous works of Reddi et al.

Original: and Kingma & Ba (2015), we compare different methods on MNIST, CIFAR and IMDB datasets in our experiment.
Revised: and Kingma & Ba (2015), we compare different methods on MNIST, CIFAR and IMDB datasets in our experiment.

Original: For MNIST, we use a noisy version named as MNIST-back-rand in Larochelle et al.
Revised: For MNIST, we use a noisy version named as MNIST-back-rand in Larochelle et al.

Original: (2007) to increase the training difﬁculty.
Revised: (2007) to increase the training difﬁculty.

Original: 4. MNIST- BACK -R AND + M ULTI - LAYER N EURAL N ETWORKS
Revised: 4. MNIST- BACK -R AND + M ULTI - LAYER N EURAL N ETWORKS

Original: Our experiments start with fully connected neural network for multi-class classiﬁcation problems.
Revised: Our experiments start with fully connected neural network for multi-class classiﬁcation problems.

Original: MNIST-back-rand dataset consists of 12000 training samples and 50000 test samples, where random background is inserted in original
Revised: MNIST-back-rand dataset consists of 12000 training samples and 50000 test samples, where random background is inserted in original

Original: MNIST hand written digit images.
Revised: MNIST hand written digit images.

Original: The input dimension is 784 ( 28 × 28 ) and the number of classes is 10.
Revised: The input dimension is 784 ( 28 × 28 ) and the number of classes is 10.

Original: We investigate a multi-layer neural networks with input layer followed by a hidden layer with 200 cells, which is then connected to a layer with 100 neurons before the output layer.
Revised: We investigate a multi-layer neural networks with input layer followed by a hidden layer with 200 cells, which is then connected to a layer with 100 neurons before the output layer.

Original: All hidden layer cells are rectiﬁer linear units (ReLu’s).
Revised: All hidden layer cells are rectiﬁer linear units (ReLu’s).

Original: We use mini-batch size 128 to calculate stochastic gradient in each iteration.
Revised: We use mini-batch size 128 to calculate stochastic gradient in each iteration.

Original: Model performance is evaluated by multi-class cross entropy loss.
Revised: Model performance is evaluated by multi-class cross entropy loss.

Original: The training loss with respect to number of iterations is reported in ﬁgure 1.
Revised: The training loss with respect to number of iterations is reported in ﬁgure 1.

Original: [Figure]
Revised: [Figure]

Original: On this dataset, we empirically observe obvious improvement of O PTIMISTIC
Revised: On this dataset, we empirically observe obvious improvement of O PTIMISTIC

Original: -AMSG RAD in terms of both convergence speed and training loss.
Revised: -AMSG RAD in terms of both convergence speed and training loss.

Original: On the other hand, AMSG RAD -DISZ performs similarly to AMSG RAD in general.
Revised: On the other hand, AMSG RAD -DISZ performs similarly to AMSG RAD in general.

Original: 4. CIFAR10 + CONVOLUTIONAL NEURAL NETWORKS (CNN)
Revised: 4. CIFAR10 + CONVOLUTIONAL NEURAL NETWORKS (CNN)

Original: Convolutional Neural Networks (CNN) have been widely studied and is important in various deep learning applications such as computer vision and natural language processing.
Revised: Convolutional Neural Networks (CNN) have been widely studied and is important in various deep learning applications such as computer vision and natural language processing.

Original: We test the effectiveness of O PTIMISTIC -AMSG RAD in deep CNN’s with dropout.
Revised: We test the effectiveness of O PTIMISTIC -AMSG RAD in deep CNN’s with dropout.

Original: We use the CIFAR10 dataset, which includes 60,000 images (50,000 for training and 10,000 for testing) of size 32 × 32 in 10 different classes.
Revised: We use the CIFAR10 dataset, which includes 60,000 images (50,000 for training and 10,000 for testing) of size 32 × 32 in 10 different classes.

Original: ALL-CNN architecture proposed in Springenberg et al.
Revised: ALL-CNN architecture proposed in Springenberg et al.

Original: (2015) is implemented with two blocks of 3 × 3 convolutional ﬁlter, 3 × 3 convolutional layer with stride 2 and dropout layer with keep probability 0.5.
Revised: (2015) is implemented with two blocks of 3 × 3 convolutional ﬁlter, 3 × 3 convolutional layer with stride 2 and dropout layer with keep probability 0.5.

Original: Another block of 3 × 3 , 1 × 1 convolutional layer and a 6 × 6 global averaging pooling is added before the output layer.
Revised: Another block of 3 × 3 , 1 × 1 convolutional layer and a 6 × 6 global averaging pooling is added before the output layer.

Original: We apply another dropout with keep probability 0.8 on the input layer.
Revised: We apply another dropout with keep probability 0.8 on the input layer.

Original: The cost function is multi-class cross entropy.
Revised: The cost function is multi-class cross entropy.

Original: The batch size is 128.
Revised: The batch size is 128.

Original: The images are all whitened.
Revised: The images are all whitened.

Original: The training loss is provided in ﬁgure 2.
Revised: The training loss is provided in ﬁgure 2.

Original: The result shows that O PTIMISTIC -AMSG RAD accelerates the learning process signiﬁcantly and gives lowest training cost after 10000 iterations.
Revised: The result shows that O PTIMISTIC -AMSG RAD accelerates the learning process signiﬁcantly and gives lowest training cost after 10000 iterations.

Original: For this dataset, the performance of AMSG RAD -DISZ is worse than original AMSG RAD .
Revised: For this dataset, the performance of AMSG RAD -DISZ is worse than original AMSG RAD .

Original: [Figure]
Revised: [Figure]

Original: 4. IMDB + L ONG -S HORT T ERM M EMORY (LSTM)
Revised: 4. IMDB + L ONG -S HORT T ERM M EMORY (LSTM)

Original: As another important application of deep learning, natural language processing tasks often beneﬁt from considering sequence dependency in the models.
Revised: As another important application of deep learning, natural language processing tasks often beneﬁt from considering sequence dependency in the models.

Original: Recurrent Neural Networks (RNN’s) achieves this goal by adding hidden state units that act as ”memory”.
Revised: Recurrent Neural Networks (RNN’s) achieves this goal by adding hidden state units that act as ”memory”.

Original: Long-Short Term Memory (LSTM) is the most popular structure in building RNN’s.
Revised: Long-Short Term Memory (LSTM) is the most popular structure in building RNN’s.

Original: We use IMDB movie review dataset from Maas et al.
Revised: We use IMDB movie review dataset from Maas et al.

Original: (2011) to test the performance of O PTIMISTIC
Revised: (2011) to test the performance of O PTIMISTIC

Original: -AMSG RAD in RNN’s under the circumstance of high data sparsity.
Revised: -AMSG RAD in RNN’s under the circumstance of high data sparsity.

Original: IMDB is a binary classiﬁcation dataset with 25000 training and test samples respectively.
Revised: IMDB is a binary classiﬁcation dataset with 25000 training and test samples respectively.

Original: Our model includes a word embedding layer with 5000 input entries representing most frequent words in the dataset and each word is embedded into a 32 dimensional space.
Revised: Our model includes a word embedding layer with 5000 input entries representing most frequent words in the dataset and each word is embedded into a 32 dimensional space.

Original: The output of embedding layer is passed to 100 LSTM units, which is then connected to 100 fully connected ReLu’s before reaching the output layer.
Revised: The output of embedding layer is passed to 100 LSTM units, which is then connected to 100 fully connected ReLu’s before reaching the output layer.

Original: Binary cross-entropy loss is used and the batch size is 128.
Revised: Binary cross-entropy loss is used and the batch size is 128.

Original: We provide the results in ﬁgure 3.
Revised: We provide the results in ﬁgure 3.

Original: [Figure]
Revised: [Figure]

Original: We observe a considerable improvement in convergence speed.
Revised: We observe a considerable improvement in convergence speed.

Original: In the ﬁrst epoch, the result is more exciting.
Revised: In the ﬁrst epoch, the result is more exciting.

Original: At epoch 0.5, O PTIMISTIC -AMSG RAD already achieves the training loss that vanilla AMSG RAD can produce with more than 1 epoch.
Revised: At epoch 0.5, O PTIMISTIC -AMSG RAD already achieves the training loss that vanilla AMSG RAD can produce with more than 1 epoch.

Original: The sample efﬁciency is signiﬁcantly improved.
Revised: The sample efﬁciency is signiﬁcantly improved.

Original: On this dataset, AMSG RAD -DISZ performs less effectively and may be trapped in local minimum.
Revised: On this dataset, AMSG RAD -DISZ performs less effectively and may be trapped in local minimum.

Original: We remark that in each iteration, only a small portion of gradients in embedding layer is non-zero.
Revised: We remark that in each iteration, only a small portion of gradients in embedding layer is non-zero.

Original: Thus, this experiment demonstrates that O PTIMISTIC -AMSGR AD could also perform well with sparse gradient.
Revised: Thus, this experiment demonstrates that O PTIMISTIC -AMSGR AD could also perform well with sparse gradient.

Original: ( See additional experiments in Appendix A 
Revised: ( See additional experiments in Appendix A .)

Original:  E XTENSIONS AND F UTURE W ORK
Revised:  E XTENSIONS AND F UTURE W ORK

Original: In this paper we propose O PTIMISTIC
Revised: In this paper, we propose O PTIMISTIC

Original: -AMSG RAD , which combines optimistic learning and AMSG RAD to strengthen the learning process of optimization problems, in particular, deep neural networks.
Revised: -AMSG RAD , which combines optimistic learning and AMSG RAD to strengthen the learning process of optimization problems, in particular, deep neural networks.

Original: The idea of adding optimistic step can be easily extended to other optimization algorithms, e g A DAM and A DA G RAD .
Revised: The idea of adding optimistic step can be easily extended to other optimization algorithms, e.g A DAM and A DA G RAD .

Original: We provide
Revised: We provide

Original: O PTIMISTIC -A DA G RAD algorithm and theoretical results in Appendix F. A potential direction based on this work is to improve the method for predicting next gradient.
Revised: O PTIMISTIC -A DA G RAD algorithm and theoretical results in Appendix F. A potential direction based on this work is to improve the method for predicting next gradient.

Original: We expect that optimistic acceleration strategy could be widely used in various optimization problems.
Revised: We expect that optimistic acceleration strategy could be widely used in various optimization problems.

Original: A M ORE E XPERIMENTAL R ESULTS A. CHOOSING PARAMETER r
Revised: A M ORE E XPERIMENTAL R ESULTS A. CHOOSING PARAMETER r

Original: In O PTIMISTIC -AMSG RAD , parameter r, the number of previous gradients used, may affect the performance dramatically.
Revised: In O PTIMISTIC -AMSG RAD , parameter r the number of previous gradients used, may affect the performance dramatically.

Original: If we choose r too small ( e.g r < 10 ), the optimistic updates will start at very early stage, but the information we collect from the past is limited.
Revised: If we choose r too small ( e g r < 10 ), the optimistic updates will start at very early stage, but the information we collect from the past is limited.

Original: This may make our training process off-track at ﬁrst several iterations.
Revised: This may make our training process off-track at ﬁrst several iterations.

Original: On the other hand, if r is chosen to be too large ( e.g r > = 30 ), although we may get a better prediction of next gradient, the optimistic step will start late so O PTIMISTIC
Revised: On the other hand, if r is chosen to be too large ( e.g r > = 30 ), although we may get a better prediction of next gradient, the optimistic step will start late so O PTIMISTIC

Original: -AMSG RAD may miss great chances to improve the learning performance at early stages.
Revised: -AMSG RAD may miss great chances to improve the learning performance at early stages.

Original: Additionally, we need more room to store past gradients, hence the operational time will increase.
Revised: Additionally, we need more room to store past gradients, hence the operational time will increase.

Revised: [Figure]

Original: The empirical impact of different r value is reported in ﬁgure 4.
Revised: The empirical impact of different r value is reported in ﬁgure 4.

Original: We suggest 10 ≤ r ≤ 20 as an ideal range, and r =tend to perform well in most training tasks.
Revised: We suggest 10 ≤ r ≤ 20 as an ideal range, and r =tend to perform well in most training tasks.

Original: Actually, we may make the algorithm more ﬂexible by ”early start”.
Revised: Actually, we may make the algorithm more ﬂexible by ”early start”.

Original: For example, when we set r = 20 , we can instead start adding optimistic step at iteration 10, and gradually increase the number of past gradients we use to predict next move from 10 to 20 in next 10 iterations.
Revised: For example, when we set r = 20 , we can instead start adding optimistic step at iteration 10, and gradually increase the number of past gradients we use to predict next move from 10 to 20 in next 10 iterations.

Original: After iteration 21, we ﬁx the moving window size for optimistic prediction as 20.
Revised: After iteration 21, we ﬁx the moving window size for optimistic prediction as 20.

Original: This may bring enhancement to O PTIMISTIC
Revised: This may bring enhancement to O PTIMISTIC

Original: -AMSG RAD because it can seek opportunities to accelerate learning in ﬁrst several iterations, which is critical for indicating a better direction towards the minimum loss.[Figure]
Revised: -AMSG RAD because it can seek opportunities to accelerate learning in ﬁrst several iterations, which is critical for indicating a better direction towards the minimum loss.

Original: A. O PTIMISTIC ADAM
Revised: A. O PTIMISTIC ADAM

Original: We also conduct experiments on O PTIMISTIC -A DAM and A DAM -DISZ.
Revised: We also conduct experiments on O PTIMISTIC -A DAM and A DAM -DISZ.

Original: The experiment setting is similar to
Revised: The experiment setting is similar to

Original: O PTIMISTIC -AMSG RAD [Figure] where we ﬁx β 1 , β 2 and compare the performance using the best learning rate with respect to A DAM 
Revised: O PTIMISTIC -AMSG RAD , where we ﬁx β 1 , β 2 and compare the performance using the best learning rate with respect to A DAM .

Original: We provide a brief summary of the results in ﬁgure [Figure] with r = 15 .
Revised: We provide a brief summary of the results in ﬁgure 5, with r = 15 .

Original: The improvement brought by
Revised: The improvement brought by

Original: O PTIMISTIC -A DAM is obvious, indicating that adding optimistic step could also enhance the performance of A DAM optimizer
Revised: O PTIMISTIC -A DAM is obvious, indicating that adding optimistic step could also enhance the performance of A DAM optimizer

Original: [Figure]
Revised: [Figure]

Original: B P ROOF OF LEMMA
Revised: B P ROOF OF LEMMA

Original: Lemma 1 Denote D ∞ =
Revised: Lemma [Equation] [Figure]

Original: max t (cid:107) w t − 12 − w ∗ (cid:107) ∞ .
Revised: [Equation] [Figure] [Equation] [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation] 
Revised: [Equation] [Figure] [Equation] [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof of this lemma basically follows that of (Reddi et al.
Revised: The proof of this lemma basically follows that of (Reddi et al.

Original: We have that
Revised: We have that

Original: [Equation]
Revised: [Equation]

Original: By rearranging the terms above and summing it from t = 1 , . . .
Revised: By rearranging the terms above and summing it from t = 1 , . . .

Original: , T ,
Revised: , T ,

Revised: T (cid:88) t

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality is due to Young’s inequality and the second one is due to the constraint 0 < β 1 ≤ 1 so that η t2(1 − β
Revised: where the ﬁrst inequality is due to Young’s inequality and the second one is due to the constraint 0 < β 1 ≤ 1 so that η t

Original: [Equation]
Revised: [Equation]

Original: +

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: Now we can bound (cid:80) Tt =

Original: [Equation]
Revised: [Equation]

Original: ≤ η t (1 − β
Revised: w

Original: [Equation]
Revised: [Equation]

Original: Now we can bound (cid:80) Tt =1 (cid:104) w t − 12 − w ∗ , g t (cid:105) in (6) as
Revised: in (6) as

Original: T (cid:88)t =
Revised: T (cid:88)

Original: [Equation]
Revised: [Equation]

Original: [Equation] where the ﬁrst inequality is due to (12).
Revised: where the ﬁrst inequality is due to (12) [Equation]

Original: Continue the analysis, we have
Revised: Continue the analysis, we have

Original: [Equation]
Revised: [Equation]

Original: where the last equality is by telescoping sum.
Revised: where the last equality is by telescoping sum

Revised: [Equation]

Original: C P ROOF OF LEMMA
Revised: C P ROOF OF LEMMA

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From the update,
Revised: From the update,

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have that
Revised: Therefore, we have that

Original: [Equation]
Revised: [Equation]

Original: where (a) is by H ¨ older’s inequality and (b) is by Young’s inequality.
Revised: where (a) is by H ¨ older’s inequality and (b) is by Young’s inequality.

Original: D P ROOF OF T HEOREM
Revised: D P ROOF OF T HEOREM

Original: Theorem 1 Denote γ := β 1 / √ β 2 < 1 and D ∞ =
Revised: Theorem 1 Denote γ := β 1 / √ β 2 < 1 and D ∞ =

Original: max t (cid:107) w t − 12 − w ∗ (cid:107) ∞ .
Revised: max t (cid:107) w t − 12 − w ∗ (cid:107) ∞ .

Original: Then,
Revised: Then,

Original: Regret T ≤ 2 η T (1 − β 1 ) D
Revised: Regret T ≤ 2 η T (1 − β 1 ) D

Original: 2 ∞ d (cid:88) i
Revised: 2 ∞ d (cid:88) i

Original: =
Revised: =

Original: ˆ v T [ i ] 2 + D 2 ∞
Revised: ˆ v T [ i ] 2 + D 2 ∞

Original: T (cid:88) t = d (cid:88) i =
Revised: T (cid:88) t = d (cid:88) i =

Original: β 1 ˆ v t
Revised: β 1 ˆ v t

Original: [ i ] 1 /2 η t (1 − β 1 ) +
Revised: [ i ] 1 /2 η t (1 − β 1 ) +

Original: T (cid:88) t = 2 η t (1 − β 1 ) (cid:107) g t − m t (cid:107) 2 ψ ∗ t +
Revised: T (cid:88) t = 2 η t (1 − β 1 ) (cid:107) g t − m t (cid:107) 2 ψ ∗ t +

Original: T (cid:88) t =
Revised: T (cid:88) t =

Original: η t
Revised: η t

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Recall that from (7), we show that
Revised: Recall that from (7), we show that

Original: [Equation]
Revised: [Equation]

Original: To proceed
Revised: To proceed

Original: [Equation]
Revised: [Equation]

Original: let us analyze (cid:80)
Revised: let us analyze (cid:80)

Original: Tt =
Revised: Tt =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: t − 1 above.
Revised: t − 1 above.

Original: Notice that
Revised: Notice that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where we use the update rule
Revised: where we use the update rule

Original: [Equation]
Revised: [Equation]

Original: w t = w t − 12 − η t 4 1 − β
Revised: w t = w t − 12 − η t 4 1 − β

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we have that
Revised: we have that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 21) where the second inequality is due to that the sequence { ˆ v t [ i ] } is non-decreasing and the last equality is because θ t − h t = β 1 θ t − 

Revised: 21) where the second inequality is due to that the sequence { ˆ v t [ i ] } is non-decreasing and the last equality is because θ t − h t = β 1 θ t −

Original: + (1 − β 1 )( g t ) − (cid:0)
Revised: + (1 − β 1 )( g t ) − (cid:0)

Original: [Equation]
Revised: [Equation]

Original: + (1 − β 1 )( m t ) (cid:1) =
Revised: + (1 − β 1 )( m t ) (cid:1) =

Original: (1 − β 1 )( g t − m t ) .
Revised: (1 − β 1 )( g t − m t ) .

Original: To summarize,
Revised: To summarize,

Original: Regret
Revised: Regret

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: E P ROOF OF T HEOREM
Revised: E P ROOF OF T HEOREM

Original: Theorem 2 There exists a convex online learning problem such that ADAM-DISZ has nonzero average regret (i.e.
Revised: Theorem 2 There exists a convex online learning problem such that ADAM-DISZ has nonzero average regret (i.e.

Original: RegretT (cid:54) =
Revised: RegretT (cid:54) =

Original: o ( T ) )
Revised: o ( T ) )

Original: We basically follow the same setting as Theorem 1 of Reddi et al.
Revised: We basically follow the same setting as Theorem 1 of Reddi et al.

Original: In each round, the loss function (cid:96) t ( w ) is linear and the learner’s decision space is K
Revised: In each round, the loss function (cid:96) t ( w ) is linear and the learner’s decision space is K

Original: = [ − 1 , 1] f t ( w ) = (cid:26) Cw , for t mod 3 =− w , otherwise , where C ≥ 4 .
Revised: = [ − 1 , 1] f t ( w ) = (cid:26) Cw , for t mod 3 =− w , otherwise , where C ≥ 4 .

Original: For this loss function sequences, the point w = − 1 achieves the minimum regret, i.e. − 1 = arg min w ∈K (cid:80) Tt
Revised: For this loss function sequences, the point w = − 1 achieves the minimum regret, i.e. − 1 = arg min w ∈K (cid:80) Tt

Original: =1 f t ( w ) when T → ∞ .
Revised: =1 f t ( w ) when T → ∞ .

Original: Consider the execution of ADAM-DISZ with β 1 = 0 , β 2 = 1 1+2 C 2 , η t = η √ t with η < 13 √ 1 − β 2 .
Revised: Consider the execution of ADAM-DISZ with β 1 = 0 , β 2 = 1 1+2 C 2 , η t = η √ t with η < 13 √ 1 − β 2 .

Original: Notice that β 21 / √ β 2 < 1 in this case, which satisﬁes the conditions of Kingma & Ba (2015).
Revised: Notice that β 21 / √ β 2 < 1 in this case, which satisﬁes the conditions of Kingma & Ba (2015).

Original: The goal is to show that for all t , w 3 t +1 = 1 , and w t > 0 .
Revised: The goal is to show that for all t , w 3 t +1 = 1 , and w t > 0 .

Original: Let us denote ˆ w t
Revised: Let us denote ˆ w t

Original: +1 :
Revised: +1 :

Original: = w t − 2 η t θ t √ v t + η t θ t − 1 √ v t − 1 and w t +1 :
Revised: = w t − 2 η t θ t √ v t + η t θ t − 1 √ v t − 1 and w t +1 :

Original: =
Revised: =

Original: Π k
Revised: Π k

Original: [ ˆ w t +1 ] .
Revised: [ ˆ w t +1 ] .

Original: Assume the initial point is 1 .
Revised: Assume the initial point is 1 .

Original: As pointed it out by Reddi et al.
Revised: As pointed it out by Reddi et al.

Original: , the assumption of the initial point is without loss of generality.
Revised: , the assumption of the initial point is without loss of generality.

Original: If the initial point is not 1 , then one can translate the coordinate system to a new coordinate system so that the initial point is 1 and then choose the loss function sequences for the new coordinate system.
Revised: If the initial point is not 1 , then one can translate the coordinate system to a new coordinate system so that the initial point is 1 and then choose the loss function sequences for the new coordinate system.

Original: Therefore, the base case x 1 = 1 is true.
Revised: Therefore, the base case x 1 = 1 is true.

Original: Now assume that for some t > 0 , x 3 t +1 = 1 and x s > 0 for all s ≤ 3 t + 1 .
Revised: Now assume that for some t > 0 , x 3 t +1 = 1 and x s > 0 for all s ≤ 3 t + 1 .

Original: Observe that.
Revised: Observe that.

Original: ∇ f
Revised: ∇ f

Original: [Equation]
Revised: [Equation]

Original: (cid:26)
Revised: (cid:26)

Original: [Equation]
Revised: [Equation]

Original: , for t mod 3 =− 1 , otherwise ,
Revised: , for t mod 3 =− 1 , otherwise ,

Original: According to the update of ADAM-DISZ
Revised: According to the update of ADAM-DISZ

Original: [Equation]
Revised: [Equation]

Original: So [Equation] ˆ w 3 t +2 =
Revised: So, ˆ w 3 t +2 =

Original: w 3 t +
Revised: w 3 t +

Original: [Equation]
Revised: [Equation]

Original: For ˆ w 3 t +4 , let us ﬁrst consider the case that ˆ w 3 t +3 <
Revised: For ˆ w 3 t +4 , let us ﬁrst consider the case that ˆ w 3 t +3 <

Original: [Equation]
Revised: [Equation]

Original: η
Revised: η

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where (a) uses (cid:113) β 2 C 
Revised: where (a) uses (cid:113) β 2 C 2 +

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: /= √ 1 − β 2 for the choice of β 2 = 1 / (1 + 2 C 2 ) , (b) is because 1 − √ 3 t +1 √ 3 t +2 is a decreasing function of t and has its largest value at t = 1 for any t > 0 , which is 1 − ( √ 4 √ 5 ) ≤ 0 .
Revised: /= √ 1 − β 2 for the choice of β 2 = 1 / (1 + 2 C 2 ) , (b) is because 1 − √ 3 t +1 √ 3 t +2 is a decreasing function of t and has its largest value at t = 1 for any t > 0 , which is 1 − ( √ 4 √ 5 ) ≤ 0 .

Original: To summarize, we have that w 3 t +4 = 1 .
Revised: To summarize, we have that w 3 t +4 = 1 .

Original: Now if it is the case that ˆ w 3 t +3 ≥
Revised: Now if it is the case that ˆ w 3 t +3 ≥

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is because
Revised: where the last inequality is because

Original: [Equation]
Revised: [Equation]

Original: To see this,
Revised: To see this,

Original: [Equation]
Revised: [Equation]

Original: So, the above inequality is equivalent to
Revised: So, the above inequality is equivalent to

Original: [Equation]
Revised: [Equation]

Original: which is true.
Revised: which is true.

Original: This means that w 3 t +4 = 1 .
Revised: This means that w 3 t +4 = 1 .

Original: Therefore, we have completed the induction.
Revised: Therefore, we have completed the induction.

Original: To summarize, we have (cid:96) 3 t +1 ( w 3 t +1 )
Revised: To summarize, we have (cid:96) 3 t +1 ( w 3 t +1 )

Original: + (cid:96) 3 t +2 ( w 3 t +2 )
Revised: + (cid:96) 3 t +2 ( w 3 t +2 )

Original: + (cid:96) 3 t +3 ( w 3 t +3 ) − (cid:96) 3 t +1 ( − 1) − (cid:96) 3 t +2 ( w − 1 ) − (cid:96) 3 t +3 ( − 1) ≥ 2 C − 4 .
Revised: + (cid:96) 3 t +3 ( w 3 t +3 ) − (cid:96) 3 t +1 ( − 1) − (cid:96) 3 t +2 ( w − 1 ) − (cid:96) 3 t +3 ( − 1) ≥ 2 C − 4 .

Original: That is, for every 3 steps, the algorithms suffers regret at least 2 C − 4 , this means that the regret over T rounds would be (2 C − 4) T/ 3 , which is not sublinear to T .
Revised: That is, for every 3 steps, the algorithms suffers regret at least 2 C − 4 , this means that the regret over T rounds would be (2 C − 4) T/ 3 , which is not sublinear to T .

Original: Now we have completed the proof.
Revised: Now we have completed the proof.

Original: One may follow the analysis of Theorem 2 of Reddi et al.
Revised: One may follow the analysis of Theorem 2 of Reddi et al.

Original: (2018) to generalize the result so that ADAM-DISZ does not converge for any β 1 , β 2 ∈ [0 , 1) such that β 1 < √ β 2 .
Revised: (2018) to generalize the result so that ADAM-DISZ does not converge for any β 1 , β 2 ∈ [0 , 1) such that β 1 < √ β 2 .

Original: [Equation]
Revised: [Equation]

Original: The optimistic update step can also be extended to other algorithms as well.
Revised: The optimistic update step can also be extended to other algorithms as well.

Original: For example, based on A DA G RAD , we propose O PTIMISTIC -A DA G RAD by including the optimistic update step.
Revised: For example, based on A DA G RAD , we propose O PTIMISTIC -A DA G RAD by including the optimistic update step.

Original: Algorithm
Revised: Algorithm

Original: G RAD
Revised: G RAD

Original: [Equation]
Revised: [Equation]

Original: where diag ( G t ) :=   (cid:80) ts =1 g t [1]   (cid:80) ts
Revised: where diag ( G t ) :=   (cid:80) ts =1 g t [1]   (cid:80) ts

Original: [Equation]
Revised: [Equation]

Original: g
Revised: g

Original: [Equation]
Revised: [Equation]

Original: [2]   · 
Revised: [2]   · 

Original:  6:
Revised:  6:

Original: [Equation]
Revised: [Equation]

Original: where m t +1 is the guess of g t +1 .
Revised: where m t +1 is the guess of g t +1 .

Original: 7: end for
Revised: 7: end for

Original: F. A NALYSIS :
Revised: F. A NALYSIS :

Original: We provide the regret analysis here.
Revised: We provide the regret analysis here.

Original: Let us recall the notations and assumptions ﬁrst.
Revised: Let us recall the notations and assumptions ﬁrst.

Original: We denote the Mahalanobis norm (cid:107) · (cid:107) H = (cid:112) (cid:104)· , H ·(cid:105) for some PSD matrix H .
Revised: We denote the Mahalanobis norm (cid:107) · (cid:107) H = (cid:112) (cid:104)· , H ·(cid:105) for some PSD matrix H .

Original: We let ψ t ( x ) :
Revised: We let ψ t ( x ) :

Original: = (cid:104) x, diag { ˆ v t } 1 / 2 x (cid:105) for the PSD matrix diag { ˆ v t } , where diag { ˆ v t } represents the diagonal matrix such that its i th diagonal element is ˆ v t
Revised: = (cid:104) x, diag { ˆ v t } 1 / 2 x (cid:105) for the PSD matrix diag { ˆ v t } , where diag { ˆ v t } represents the diagonal matrix such that its i th diagonal element is ˆ v t

Original: [ i ] in Algorithm 2.
Revised: [ i ] in Algorithm 2.

Original: Consequently, ψ t ( · ) is 1-strongly convex with respect to the norm (cid:107) · (cid:107) ψ t := (cid:112) (cid:104)· , diag { ˆ v t } 1 / 2 ·(cid:105) .
Revised: Consequently, ψ t ( · ) is 1-strongly convex with respect to the norm (cid:107) · (cid:107) ψ t := (cid:112) (cid:104)· , diag { ˆ v t } 1 / 2 ·(cid:105) .

Original: Namely, ψ t ( · ) satisﬁes ψ t ( u ) ≥ ψ t ( v ) + (cid:104) ψ t ( v ) , u − v (cid:105) + 1 2 (cid:107) u − v (cid:107) 2 ψ t for any point u, v .
Revised: Namely, ψ t ( · ) satisﬁes ψ t ( u ) ≥ ψ t ( v ) + (cid:104) ψ t ( v ) , u − v (cid:105) + 1 2 (cid:107) u − v (cid:107) 2 ψ t for any point u, v .

Original: A consequence of 1-strongly convexity of ψ t ( · ) is that B ψ t ( u, v ) ≥ ψ t , where the Bregman divergence B ψ t ( u, v ) is deﬁned as B ψ t ( u, v ) :
Revised: A consequence of 1-strongly convexity of ψ t ( · ) is that B ψ t ( u, v ) ≥ ψ t , where the Bregman divergence B ψ t ( u, v ) is deﬁned as B ψ t ( u, v ) :

Original: = ψ t ( u ) − ψ t ( v ) −(cid:104)
Revised: = ψ t ( u ) − ψ t ( v ) −(cid:104)

Original: ψ t ( v ) , u − v (cid:105) and ψ t is called the distance generating function.
Revised: ψ t ( v ) , u − v (cid:105) and ψ t is called the distance generating function.

Original: We can also deﬁne the associate dual norm as ψ ∗ t ( x ) := (cid:104) x, diag { ˆ v t } − 1 / 2 x (cid:105) .
Revised: We can also deﬁne the associate dual norm as ψ ∗ t ( x ) := (cid:104) x, diag { ˆ v t } − 1 / 2 x (cid:105) .

Original: We assume that the model parameter w is in d -dimensional space.
Revised: We assume that the model parameter w is in d -dimensional space.

Original: That is, w ∈ R d .
Revised: That is, w ∈ R d .

Original: It sufﬁces to analyze Algorithm 6, which holds for any convex set K .
Revised: It sufﬁces to analyze Algorithm 6, which holds for any convex set K .

Original: The algorithm reduces to Algorithm 5 when K = R d .
Revised: The algorithm reduces to Algorithm 5 when K = R d .

Original: 1: Required: parameter η t .
Revised: 1: Required: parameter η t .

Original: 2: Init: w 1 .
Revised: 2: Init: w 1 .

Original: 3: for t = 1 to T do 4:
Revised: 3: for t = 1 to T do 4:

Original: Get current gradient g t at w t .
Revised: Get current gradient g t at w t .

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where m t +1 is the guess of g t +1 .
Revised: where m t +1 is the guess of g t +1 .

Original: 7: end for
Revised: 7: end for

Original: By regret decomposition, we have that
Revised: By regret decomposition, we have that

Original: Regret T :=
Revised: Regret T :=

Original: T (cid:88) t = (cid:96) t ( w t ) − min w ∈K
Revised: T (cid:88) t = (cid:96) t ( w t ) − min w ∈K

Original: T (cid:88) t = (cid:96) t ( w ) ≤
Revised: T (cid:88) t = (cid:96) t ( w ) ≤

Original: T (cid:88) t
Revised: T (cid:88) t

Original: = (cid:104) w t − w ∗ , ∇ (cid:96) t ( w t ) (cid:105) =
Revised: = (cid:104) w t − w ∗ , ∇ (cid:96) t ( w t ) (cid:105) =

Original: T (cid:88) t
Revised: T (cid:88) t

Original: = (cid:104)
Revised: = (cid:104)

Original: [Equation]
Revised: [Equation]

Original: (cid:96)
Revised: (cid:96)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: (cid:105)
Revised: (cid:105)

Original: [Equation]
Revised: [Equation]

Original: (cid:104)
Revised: (cid:104)

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:105)
Revised: (cid:105)

Original: [Equation]
Revised: [Equation]

Original: (cid:104) w t + 12 − w ∗ , ∇ (cid:96) t ( w t ) (cid:105) .
Revised: (cid:104) w t + 12 − w ∗ , ∇ (cid:96) t ( w t ) (cid:105) .

Original: Now we are going to exploit a useful inequality: for any update of the form
Revised: Now we are going to exploit a useful inequality: for any update of the form

Original: ˆ w = arg min w ∈K (cid:104) w, θ (cid:105) + B ψ ( w, v ) , we have that (cid:104)
Revised: ˆ w = arg min w ∈K (cid:104) w, θ (cid:105) + B ψ ( w, v ) , we have that (cid:104)

Original: [Equation]
Revised: [Equation]

Original: ψ ( u
Revised: ψ ( u

Original: [Equation]
Revised: [Equation]

Original: ˆ
Revised: ˆ

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: ψ ( ˆ w
Revised: ψ ( ˆ w

Original: [Equation]
Revised: [Equation]

Original: v ) , (31) for any u ∈ K .
Revised: v ) , (31) for any u ∈ K .

Original: Using the above inequality, we have
Revised: Using the above inequality, we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: So
Revised: So

Original: [Equation]
Revised: [Equation]

Original: by (30) (32) and (33)
Revised: by (30) (32) and (33)

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: Regret T ≤
Revised: Regret T ≤

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: +η t (cid:0)
Revised: +η t (cid:0)

Original: [Equation]
Revised: [Equation]

Original: where the third inequality is because
Revised: where the third inequality is because

Original: [Equation]
Revised: [Equation]

Original: and that ψ t − 1 ( · ) is 1-strongly convex with respect to (cid:107) ·
Revised: and that ψ t − 1 ( · ) is 1-strongly convex with respect to (cid:107) ·

Original: [Equation]
Revised: [Equation]

Original: ψ t − 1 .
Revised: ψ t − 1 .

Original: To proceed, notice that, by deﬁnition of Bregman divergence, we also have
Revised: To proceed, notice that, by deﬁnition of Bregman divergence, we also have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where the vector s t ∈ R d is deﬁned as follows: its i th entry s t [ i ] is
Revised: where the vector s t ∈ R d is deﬁned as follows: its i th entry s t [ i ] is

Original: [Equation]
Revised: [Equation]

Original: Now we have
Revised: Now we have

Original: Regret
Revised: Regret

Original: [Equation]
Revised: [Equation]

Original: Therefore, we conclude the following theorem.
Revised: Therefore, we conclude the following theorem.

Original: Theorem 3.
Revised: Theorem 3.

Original: For any learning rate η t := η , O PTIMISTIC -A DA G RAD (Algorithm 6) has regret
Revised: For any learning rate η t := η , O PTIMISTIC -A DA G RAD (Algorithm 6) has regret

Original: [Equation]
Revised: [Equation]

Original: One need to compare it with the bound of A DA G RAD Duchi et al.
Revised: One need to compare it with the bound of A DA G RAD Duchi et al.

Original: (2011), which has regret
Revised: (2011), which has regret

Original: [Equation]
Revised: [Equation]

Original: T
Revised: T

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We see that when m t predicts ∇ f ( w t ) very well, the last term is dominated by the second one.
Revised: We see that when m t predicts ∇ f ( w t ) very well, the last term is dominated by the second one.

Original: One can set η of
Revised: One can set η of

Original: O PTIMISTIC -A DA G RAD to be a large step size so that O PTIMISTIC -A DA G RAD has a smaller regret than A DA G RAD .
Revised: O PTIMISTIC -A DA G RAD to be a large step size so that O PTIMISTIC -A DA G RAD has a smaller regret than A DA G RAD .