Original: O N THE C ONVERGENCE OF A DA G RAD (N ORM ) ON R d : B EYOND C ONVEXITY , N ON -A SYMPTOTIC R ATE AND A CCELERATION
Revised: O N THE C ONVERGENCE OF A DA G RAD (N ORM ) ON R d : B EYOND C ONVEXITY , N ON -A SYMPTOTIC R ATE AND A CCELERATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Existing analysis of AdaGrad and other adaptive methods for smooth convex optimization is typically for functions with bounded domain diameter.
Revised: Existing analysis of AdaGrad and other adaptive methods for smooth convex optimization is typically for functions with bounded domain diameter.

Original: In unconstrained problems, previous works guarantee an asymptotic convergence rate without an explicit constant factor that holds true for the entire function class.
Revised: In unconstrained problems, previous works guarantee an asymptotic convergence rate without an explicit constant factor that holds true for the entire function class.

Original: Furthermore, in the stochastic setting, only a modiﬁed version of AdaGrad, different from the one commonly used in practice, in which the latest gradient is not used to update the stepsize, has been analyzed.
Revised: Furthermore, in the stochastic setting, only a modiﬁed version of AdaGrad, different from the one commonly used in practice, in which the latest gradient is not used to update the stepsize, has been analyzed.

Original: Our paper aims at bridging these gaps and developing a deeper understanding of AdaGrad and its variants in the standard setting of smooth convex functions as well as the more general setting of quasar convex functions.
Revised: Our paper aims at bridging these gaps and developing a deeper understanding of AdaGrad and its variants in the standard setting of smooth convex functions as well as the more general setting of quasar convex functions.

Original: First, we demonstrate new techniques to explicitly bound the convergence rate of the vanilla AdaGrad for unconstrained problems in both deterministic and stochastic settings.
Revised: First, we demonstrate new techniques to explicitly bound the convergence rate of the vanilla AdaGrad for unconstrained problems in both deterministic and stochastic settings.

Original: Second, we propose a variant of AdaGrad for which we can show the convergence of the last iterate, instead of the average iterate.
Revised: Second, we propose a variant of AdaGrad for which we can show the convergence of the last iterate, instead of the average iterate.

Original: Finally, we give new accelerated adaptive algorithms and their convergence guarantee in the deterministic setting with explicit dependency on the problem parameters, improving upon the asymptotic rate shown in previous works.
Revised: Finally, we give new accelerated adaptive algorithms and their convergence guarantee in the deterministic setting with explicit dependency on the problem parameters, improving upon the asymptotic rate shown in previous works.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In recent years, the prevalence of machine learning models has motivated the development of new optimization tools, among which adaptive methods such as Adam (Kingma & Ba, 2014), AmsGrad (Reddi et al., 2018), AdaGrad (Duchi et al., 2011) emerge as the most important class of algorithms.
Revised: In recent years, the prevalence of machine learning models has motivated the development of new optimization tools, among which adaptive methods such as Adam (Kingma & Ba, 2014), AmsGrad (Reddi et al., 2018), AdaGrad (Duchi et al., 2011) emerge as the most important class of algorithms.

Original: These methods do not require the knowledge of the problem parameters when setting the stepsize as traditional methods like SGD, while still showing robust performances in many ML tasks.
Revised: These methods do not require the knowledge of the problem parameters when setting the stepsize as traditional methods like SGD, while still showing robust performances in many ML tasks.

Original: However, it remains a challenge to analyze and understand the properties of these methods.
Revised: However, it remains a challenge to analyze and understand the properties of these methods.

Original: Take
Revised: Take

Original: AdaGrad and its variants for example.
Revised: AdaGrad and its variants for example.

Original: In its vanilla scalar form, also known as AdaGradNorm, the step size is set using the cumulative sum of the gradient norm of all iterates so far.
Revised: In its vanilla scalar form, also known as AdaGradNorm, the step size is set using the cumulative sum of the gradient norm of all iterates so far.

Original: The work of
Revised: The work of

Original: Ward et al.
Revised: Ward et al.

Original: (2020) has shown the convergence of this algorithm for non-convex funtions by bounding the decay of the gradient norms.
Revised: (2020) has shown the convergence of this algorithm for non-convex funtions by bounding the decay of the gradient norms.

Original: However, in convex optimization, usually we require a stronger convergence criterion—bounding the function value gap.
Revised: However, in convex optimization, usually we require a stronger convergence criterion—bounding the function value gap.

Original: This is where we lack theoretical understanding.
Revised: This is where we lack theoretical understanding.

Original: Even in the deterministic setting, most existing works (Levy, 2017; Levy et al., 2018; Ene et al., 2021) rely on the assumption that the domain of the function is bounded.
Revised: Even in the deterministic setting, most existing works (Levy, 2017; Levy et al., 2018; Ene et al., 2021) rely on the assumption that the domain of the function is bounded.

Original: The dependence on the domain diameter can become an issue if it is unknown or cannot be readily estimated.
Revised: The dependence on the domain diameter can become an issue if it is unknown or cannot be readily estimated.

Original: Other works for unconstrained problems (Antonakopoulos et al., 2020; 2022) offer a convergence rate that depends on the limit of the step size sequence.
Revised: Other works for unconstrained problems (Antonakopoulos et al., 2020; 2022) offer a convergence rate that depends on the limit of the step size sequence.

Original: This limit is shown to exist for each function, but without an explicit value, and more importantly, it is not shown to be a constant for the entire function class.
Revised: This limit is shown to exist for each function, but without an explicit value, and more importantly, it is not shown to be a constant for the entire function class.

Original: This means that these methods essentially do not tell us how fast the algorithm converges in the worst case.
Revised: This means that these methods essentially do not tell us how fast the algorithm converges in the worst case.

Original: Another work by Ene & Nguyen (2022) gives an explicit rate of convergence for the entire class but requires the strong assumption that the gradients are bounded even in the smooth setting and the convergence guarantee has additional error terms depending on this bound.
Revised: Another work by Ene & Nguyen (2022) gives an explicit rate of convergence for the entire class but requires the strong assumption that the gradients are bounded even in the smooth setting and the convergence guarantee has additional error terms depending on this bound.

Original: In the stochastic setting, one common approach is to analyze a modiﬁed version of AdaGrad with off-by-one step size, i.e. the gradient at the current time step is not taken into account when setting the new step size.
Revised: In the stochastic setting, one common approach is to analyze a modiﬁed version of AdaGrad with off-by-one step size, i.e. the gradient at the current time step is not taken into account when setting the new step size.

Original: This is where the gap between theory and practice exists.
Revised: This is where the gap between theory and practice exists.

Original: 1. O UR C ONTRIBUTION
Revised: 1. O UR C ONTRIBUTION

Original: In this paper, we make the following contributions.
Revised: In this paper, we make the following contributions.

Original: First, we demonstrate a method to show an explicit non-asymptotic convergence rate of AdaGradNorm and AdaGrad on R d in the deterministic setting.
Revised: First, we demonstrate a method to show an explicit non-asymptotic convergence rate of AdaGradNorm and AdaGrad on R d in the deterministic setting.

Original: Our method extends to a more general function class known as γ -quasar convex functions with a weaker condition for smoothness.
Revised: Our method extends to a more general function class known as γ -quasar convex functions with a weaker condition for smoothness.

Original: To the best of our knowledge, we are the ﬁrst to prove this result.
Revised: To the best of our knowledge, we are the ﬁrst to prove this result.

Original: Second, we present new techniques to analyze stochastic AdaGradNorm and offer an explicit convergence guarantee for γ -quasar convex optimization on R d with a mild assumption on the noise of the gradient estimates.
Revised: Second, we present new techniques to analyze stochastic AdaGradNorm and offer an explicit convergence guarantee for γ -quasar convex optimization on R d with a mild assumption on the noise of the gradient estimates.

Original: We propose two new variants of AdaGradNorm which demonstrate the convergence of the last iterate instead of the average iterate as shown in AdaGradNorm.
Revised: We propose two new variants of AdaGradNorm which demonstrate the convergence of the last iterate instead of the average iterate as shown in AdaGradNorm.

Original: Finally, we propose a new accelerated algorithm with two variants and show their non-asymptotic convergence rate in the deterministic setting.
Revised: Finally, we propose a new accelerated algorithm with two variants and show their non-asymptotic convergence rate in the deterministic setting.

Original: 1. R ELATED W ORK Adaptive methods
Revised: 1. R ELATED W ORK Adaptive methods

Original: There has been a long line of works on adaptive methods, including AdaGrad (Duchi et al., 2011), RMSProp (Tieleman et al., 2012) and Adam (Kingma & Ba, 2014).
Revised: There has been a long line of works on adaptive methods, including AdaGrad (Duchi et al., 2011), RMSProp (Tieleman et al., 2012) and Adam (Kingma & Ba, 2014).

Original: AdaGrad was ﬁrst designed for stochastic online optimization; subsequent works (Levy, 2017; Kavis et al., 2019; Bach & Levy, 2019;
Revised: AdaGrad was ﬁrst designed for stochastic online optimization; subsequent works (Levy, 2017; Kavis et al., 2019; Bach & Levy, 2019;

Original: Antonakopoulos et al., 2020; Ene et al., 2021) analyzed AdaGrad and various adaptive algorithms for convex optimization and generalized them for variational inequality problems.
Revised: Antonakopoulos et al., 2020; Ene et al., 2021) analyzed AdaGrad and various adaptive algorithms for convex optimization and generalized them for variational inequality problems.

Original: These works commonly assume that the optimization problem is contrained in a set with bounded diameter.
Revised: These works commonly assume that the optimization problem is contrained in a set with bounded diameter.

Original: Li & Orabona (2019) are the ﬁrst to analyze a variant of AdaGrad for unbounded domains where the latest gradient is not used to construct the step size, which differs from the standard version of AdaGrad commonly used in practice.
Revised: Li & Orabona (2019) are the ﬁrst to analyze a variant of AdaGrad for unbounded domains where the latest gradient is not used to construct the step size, which differs from the standard version of AdaGrad commonly used in practice.

Original: However, the algorithm and analysis of
Revised: However, the algorithm and analysis of

Original: Li & Orabona (2019) set the initial step size based on the smoothness parameter and thus they do not adapt to it.
Revised: Li & Orabona (2019) set the initial step size based on the smoothness parameter and thus they do not adapt to it.

Original: Other works provide convergence guarantees for adaptive methods for unbounded domains, yet without explicit dependency on the problem parameters (Antonakopoulos et al., 2020; 2022), or for a class of strongly convex functions (Xie et al., 2020).
Revised: Other works provide convergence guarantees for adaptive methods for unbounded domains, yet without explicit dependency on the problem parameters (Antonakopoulos et al., 2020; 2022), or for a class of strongly convex functions (Xie et al., 2020).

Original: Another work by Ene & Nguyen (2022) requires the strong assumption that the gradients are bounded even for smooth functions and the convergence guarantee has additional error terms depending on the gradient upperbound.
Revised: Another work by Ene & Nguyen (2022) requires the strong assumption that the gradients are bounded even for smooth functions and the convergence guarantee has additional error terms depending on the gradient upperbound.

Original: Our work analyzes the standard version of AdaGrad for unconstrained and general convex problems and shows explicit convergence rate in both the deterministic and stochastic setting.
Revised: Our work analyzes the standard version of AdaGrad for unconstrained and general convex problems and shows explicit convergence rate in both the deterministic and stochastic setting.

Original: Accelerated adaptive methods have been designed to achieve O (1 /T 2 ) and O (1 / √ T ) respectively in the deterministic and stochastic setting in the works of Levy et al.
Revised: Accelerated adaptive methods have been designed to achieve O (1 /T 2 ) and O (1 / √ T ) respectively in the deterministic and stochastic setting in the works of Levy et al.

Original: (2018); Ene & Nguyen (2022);
Revised: (2018); Ene & Nguyen (2022);

Original: Antonakopoulos et al.
Revised: Antonakopoulos et al.

Original: We show different variants and demonstrate the same but explicit accelerated convergence rate in the deterministic setting for unconstrained problems.
Revised: We show different variants and demonstrate the same but explicit accelerated convergence rate in the deterministic setting for unconstrained problems.

Original: In another line of works, different adaptive optimization methods have been introduced for convex and variational inequality problems in Malitsky & Mishchenko (2020); Malitsky (2020).

Original: These methods are also designed for unconstrained problems but only in the deterministic setting.

Original: They rely on a different scheme to set the step size, which we consider tangent to our work.

Original: Analysis beyond convexity The convergence of some variants of AdaGrad has been established for nonconvex functions in the work of Li & Orabona (2019); Ward et al.
Revised: Analysis beyond convexity The convergence of some variants of AdaGrad has been established for nonconvex functions in the work of Li & Orabona (2019); Ward et al.

Original: Faw et al.
Revised: Faw et al.

Original: under various assumptions.
Revised: under various assumptions.

Original: Other works (Li & Orabona, 2020; Kavis et al., 2022) demonstrate the convergence with high probability.
Revised: Other works (Li & Orabona, 2020; Kavis et al., 2022) demonstrate the convergence with high probability.

Original: We refer the reader to Faw et al.
Revised: We refer the reader to Faw et al.

Original: for a more detailed survey on AdaGrad-style methods for nonconvex optimization.
Revised: for a more detailed survey on AdaGrad-style methods for nonconvex optimization.

Original: In general, the criterion used to study these convergence rates is the gradient norm of the function, which is weaker than the function value gap normally used in the study of convex functions.
Revised: In general, the criterion used to study these convergence rates is the gradient norm of the function, which is weaker than the function value gap normally used in the study of convex functions.

Original: In comparison, we study the convergence of
Revised: In comparison, we study the convergence of

Original: AdaGrad via the function value gap for a broader notion of convexity, known as quasar-convexity, as well as a more generalized deﬁnition of smoothness.
Revised: AdaGrad via the function value gap for a broader notion of convexity, known as quasar-convexity, as well as a more generalized deﬁnition of smoothness.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We consider the following optimization problem: minimize x ∈ R d F ( x ) , where F is differentiable satisfying F ∗ = inf x ∈ R d F ( x ) >
Revised: We consider the following optimization problem: minimize x ∈ R d F ( x ) , where F is differentiable satisfying F ∗ = inf x ∈ R d F ( x ) >

Original: −∞ and x ∗ ∈ arg min x ∈ R d F ( x ) (cid:54) = ∅ .
Revised: −∞ and x ∗ ∈ arg min x ∈ R d F ( x ) (cid:54) = ∅ .

Original: We will use the following notations throughout the paper: a + = max { a, 0 } , a ∨ b = max { a, b } , [ n ] = { 1 , 2 , · · · , n } , and (cid:107) · (cid:107) denotes the (cid:96) 2 -norm (cid:107) · (cid:107) 2 for simplicity.
Revised: We will use the following notations throughout the paper: a + = max { a, 0 } , a ∨ b = max { a, b } , [ n ] = { 1 , 2 , · · · , n } , and (cid:107) · (cid:107) denotes the (cid:96) 2 -norm (cid:107) · (cid:107) 2 for simplicity.

Original: Initialize: x 1 , η >for t = 1 to T
Revised: Initialize: x 1 , η >for t = 1 to T

Original: [Equation]
Revised: [Equation]

Original: Initialize: x 1 , η >for t = 1 to T
Revised: Initialize: x 1 , η >for t = 1 to T

Original: [Equation]
Revised: [Equation]

Original: Additionally, we list below the assumptions that will be used in the paper.
Revised: Additionally, we list below the assumptions that will be used in the paper.

Original: γ -quasar convexity : There exists γ ∈ (0 , 1] such that F ∗ ≥ F ( x )+ 1 γ (cid:104)∇ F ( x ) , x ∗ − x (cid:105) , ∀ x ∈ R d
Revised: γ -quasar convexity : There exists γ ∈ (0 , 1] such that F ∗ ≥ F ( x )+ 1 γ (cid:104)∇ F ( x ) , x ∗ − x (cid:105) , ∀ x ∈ R d

Original: where x ∗ ∈ arg min x ∈ R d F ( x ) .
Revised: where x ∗ ∈ arg min x ∈ R d F ( x ) .

Original: When γ = 1 , F is also known as star-convex.
Revised: When γ = 1 , F is also known as star-convex.

Original: Convexity : F is convex.
Revised: Convexity : F is convex.

Original: This stronger assumption implies that Assumption 1 holds with γ = 1 . 2.
Revised: This stronger assumption implies that Assumption 1 holds with γ = 1 . 2.

Original: Weak L -smoothness : ∃ L > 0 such that F ( x ) − F ∗ ≥ (cid:107)∇ F ( x ) (cid:107) 2 / 2 L, ∀ x ∈ R d . 2’.
Revised: Weak L -smoothness : ∃ L > 0 such that F ( x ) − F ∗ ≥ (cid:107)∇ F ( x ) (cid:107) 2 / 2 L, ∀ x ∈ R d . 2’.

Original: L -smoothness : ∃ L > 0 such that F ( x ) ≤ F ( y )
Revised: L -smoothness : ∃ L > 0 such that F ( x ) ≤ F ( y )

Original: + (cid:104)∇ F ( y ) , x − y (cid:105)
Revised: + (cid:104)∇ F ( y ) , x − y (cid:105)

Original: + L 2 (cid:107) x − y (cid:107) 2 ,
Revised: + L 2 (cid:107) x − y (cid:107) 2 ,

Original: ∀ x, y ∈ R d . 2”.
Revised: ∀ x, y ∈ R d . 2”.

Original: L -smoothness : ∃ L = diag (cid:0) L i ∈
Revised: L -smoothness : ∃ L = diag (cid:0) L i ∈

Original: [ d ] (cid:1) with L i > 0 such that F ( x ) ≤ F ( y ) + (cid:104)∇ F ( y ) , x − y (cid:105) + 12 (cid:107) x − y (cid:107) 2 L , ∀ x, y ∈ R d where (cid:107)
Revised: [ d ] (cid:1) with L i > 0 such that F ( x ) ≤ F ( y ) + (cid:104)∇ F ( y ) , x − y (cid:105) + 12 (cid:107) x − y (cid:107) 2 L , ∀ x, y ∈ R d where (cid:107)

Original: a (cid:107) L = (cid:112) (cid:104) a, L a (cid:105) .
Revised: a (cid:107) L = (cid:112) (cid:104) a, L a (cid:105) .

Original: In the stochastic setting, we assume that we have access to a stochastic gradient oracle (cid:98) ∇ F ( x ) that is independent of the history of the randomness and it satisﬁes the following assumptions:
Revised: In the stochastic setting, we assume that we have access to a stochastic gradient oracle (cid:98) ∇ F ( x ) that is independent of the history of the randomness and it satisﬁes the following assumptions:

Original: Unbiased gradient estimate : E [ (cid:98) ∇ F ( x )]
Revised: Unbiased gradient estimate : E [ (cid:98) ∇ F ( x )]

Original: = ∇ F ( x ) .
Revised: = ∇ F ( x ) .

Original: Sub-Weibull noise : E (cid:104) exp (cid:16) ( (cid:107) (cid:98) ∇ F ( x ) − ∇ F ( x ) (cid:107) /σ )
Revised: Sub-Weibull noise : E (cid:104) exp (cid:16) ( (cid:107) (cid:98) ∇ F ( x ) − ∇ F ( x ) (cid:107) /σ )

Original: 1 /θ (cid:17)(cid:105) ≤ exp(1) for some θ > 0 .
Revised: 1 /θ (cid:17)(cid:105) ≤ exp(1) for some θ > 0 .

Original: Here, we give a brief discussion of our assumptions.
Revised: Here, we give a brief discussion of our assumptions.

Original: Assumption 1 is introduced by Hinder et al.
Revised: Assumption 1 is introduced by Hinder et al.

Original: and it is strictly weaker than Assumption 1’.
Revised: and it is strictly weaker than Assumption 1’.

Original: Assumption 2 is a relaxation of Assumption 2’, the latter is the standard deﬁnition of smoothness used in many existing works (see Guille-Escuret et al.
Revised: Assumption 2 is a relaxation of Assumption 2’, the latter is the standard deﬁnition of smoothness used in many existing works (see Guille-Escuret et al.

Original: (2021) for a detailed comparison between different smoothness conditions).
Revised: (2021) for a detailed comparison between different smoothness conditions).

Original: Assumption 2” is used to analyze the AdaGrad algorithm which uses per-coordinate step sizes.
Revised: Assumption 2” is used to analyze the AdaGrad algorithm which uses per-coordinate step sizes.

Original: Assumptionis a standard assumption in stochastic optimization problems.
Revised: Assumptionis a standard assumption in stochastic optimization problems.

Original: Assumption 4 is more general and encapsulates sub-Gaussian ( θ = 1 / 2 , used in Li & Orabona (2019)) and sub-exponential noise ( θ = 1 ).
Revised: Assumption 4 is more general and encapsulates sub-Gaussian ( θ = 1 / 2 , used in Li & Orabona (2019)) and sub-exponential noise ( θ = 1 ).

Original: We refer the reader to Vladimirova et al.
Revised: We refer the reader to Vladimirova et al.

Original: (2020) for more discussion on sub-Weibull noise.
Revised: (2020) for more discussion on sub-Weibull noise.

Original:  C ONVERGENCE OF A DA G RAD N ORM ON R d UNDER γ - QUASAR CONVEXITY
Revised:  C ONVERGENCE OF A DA G RAD N ORM ON R d UNDER γ - QUASAR CONVEXITY

Original: We ﬁrst turn our attention to AdaGradNorm (Algorithm 1) in the deterministic setting, which will serve as the basis for the understanding of Stochastic AdaGradNorm (Algorithm 2) and deterministic AdaGrad (Algorithm 7).
Revised: We ﬁrst turn our attention to AdaGradNorm (Algorithm 1) in the deterministic setting, which will serve as the basis for the understanding of Stochastic AdaGradNorm (Algorithm 2) and deterministic AdaGrad (Algorithm 7).

Original: To the best of our knowledge, we are the ﬁrst to present the explict convergence rate of these three algorithms on R d .
Revised: To the best of our knowledge, we are the ﬁrst to present the explict convergence rate of these three algorithms on R d .

Original: Due to the space limit, we defer the theorem of the convergence guarantee of AdaGrad and its proof to Section A.3 in the appendix.
Revised: Due to the space limit, we defer the theorem of the convergence guarantee of AdaGrad and its proof to Section A.3 in the appendix.

Original: 3. A DA G RAD N ORM
Revised: 3. A DA G RAD N ORM

Original: Previous analysis of AdaGradNorm often aims at bounding the gradient norm of smooth nonconvex functions, or is conducted for smooth convex functions in constrained problems with a bounded domain.
Revised: Previous analysis of AdaGradNorm often aims at bounding the gradient norm of smooth nonconvex functions, or is conducted for smooth convex functions in constrained problems with a bounded domain.

Original: Bounding the gradient norm is strictly weaker than bounding the function value gap due to the fact that (cid:107)∇ F ( x ) (cid:107) 2 ≤ 2 L ( F ( x ) − F ∗ ) , where L is the smoothness parameter.
Revised: Bounding the gradient norm is strictly weaker than bounding the function value gap due to the fact that (cid:107)∇ F ( x ) (cid:107) 2 ≤ 2 L ( F ( x ) − F ∗ ) , where L is the smoothness parameter.

Original: For convex functions, the common analysis will always meet the following intermediate step
Revised: For convex functions, the common analysis will always meet the following intermediate step

Original: [Equation]
Revised: [Equation]

Original: Assuming a bounded domain is a way to making the terms bt 2 η (cid:2) (cid:107) x t − x
Revised: Assuming a bounded domain is a way to making the terms b

Original: [Equation]
Revised: [Equation]

Original: − x ∗ (cid:107) 2 (cid:3) telescope after taking the sum over all iterations t .
Revised: telescope after taking the sum over all iterations t .

Original: This is critical in the analysis, but at the same time leads to the dependence on the domain diameter, which can be hard to estimate.
Revised: This is critical in the analysis, but at the same time leads to the dependence on the domain diameter, which can be hard to estimate.

Original: For unconstrained problems, a natural approach is to divide the terms by b t , so that the remaining terms 
Revised: For unconstrained problems, a natural approach is to divide the terms by b t , so that the remaining terms 12 η

Original: [Equation]
Revised: [Equation]

Original: x 
Revised: x t − x [Equation] (cid:107) [Equation]

Revised: − (cid:107) x

Original: [Equation]
Revised: [Equation]

Revised: 1 − x ∗

Original: [Equation]
Revised: [Equation]

Original:  can telescope.
Revised: 2 (cid:3) can telescope.

Original: Our key insight is that we can bound the function value gap via the step size b t , which in turn can be bounded via the function value gap.
Revised: Our key insight is that we can bound the function value gap via the step size b t , which in turn can be bounded via the function value gap.

Original: This selfbounding argument allows us to ﬁnally prove the convergence rate.
Revised: This selfbounding argument allows us to ﬁnally prove the convergence rate.

Original: This result holds under more general conditions than convexity and smoothness (Assumptions 1 and 2).
Revised: This result holds under more general conditions than convexity and smoothness (Assumptions 1 and 2).

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: With Assumptions 1 and 2, AdaGradNorm admits
Revised: With Assumptions 1 and 2, AdaGradNorm admits

Original: [Equation]
Revised: [Equation]

Original: T

Original: Proof.
Revised: Proof.

Original: Starting from the γ -quasar convexity of F , we have
Revised: Starting from the γ -quasar convexity of F , we have

Original: [Equation]
Revised: [Equation]

Original: Notice that x
Revised: Notice that x

Original: t +1 − x t = − ηb − 1 t ∇ F ( x t ) .
Revised: t +1 − x t = − ηb − 1 t ∇ F ( x t ) .

Original: Dividing both sides by b t and taking the sum over t , we obtain
Revised: Dividing both sides by b t and taking the sum over t , we obtain

Original: [Equation]
Revised: [Equation]

Original: Note that F also satisﬁes Assumption 2 , i.e., F ( x t ) − F ∗ ≥ (cid:107)∇ F ( x t ) (cid:107) 2 2 L .
Revised: Note that F also satisﬁes Assumption 2 , i.e., F ( x t ) − F ∗ ≥ (cid:107)∇ F ( x t ) (cid:107) 2 2 L .

Original: Therefore
Revised: Therefore

Original: T (cid:88) t
Revised: T (cid:88) t

Original: [Equation]
Revised: [Equation]

Original: We can bound the term A by the technique commonly used in the analysis of adaptive methods.
Revised: We can bound the term A by the technique commonly used in the analysis of adaptive methods.

Original: Let τ be the last t such that b t ≤ 2 ηLγ .
Revised: Let τ be the last t such that b t ≤ 2 ηLγ .

Original: If b 1 > 2 ηLγ , we have A < 0 ≤ 2 ηγ log + 2 ηLγb 0 .
Revised: If b 1 > 2 ηLγ , we have A < 0 ≤ 2 ηγ log + 2 ηLγb 0 .

Original: Otherwise
Revised: Otherwise

Original: A ≤ τ (cid:88) t = (cid:18)
Revised: A ≤ τ (cid:88) t = (cid:18)

Original: η γb 2 t − 2 Lb t (cid:19) (cid:107)∇ F ( x t ) (cid:107) 2 ≤
Revised: η γb 2 t − 2 Lb t (cid:19) (cid:107)∇ F ( x t ) (cid:107) 2 ≤

Original: [Equation]
Revised: [Equation]

Original: (cid:88) t =
Revised: (cid:88) t =

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  Thus we always have A ≤ 2 ηγ log + 2 ηLγb 0 , and obtain
Revised: γb 0 . Thus we always have A ≤ 2 ηγ log + 2 ηLγb 0 , and obtain

Original: T 
Revised: T (cid:88)

Original: [Equation]
Revised: [Equation]

Original: which gives
Revised: which gives

Original: T 
Revised: T (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Note that by Assumption 2 again, we have
Revised: Note that by Assumption 2 again, we have

Original: [Equation]
Revised: [Equation]

Original:  L
Revised: 2 L

Original: [Equation]
Revised: [Equation]

Original:  − F
Revised: F ( x t ) − F

Original: [Equation]
Revised: [Equation]

Original: Let ∆ T = (cid:80) T t =1 F ( x t ) − F ∗ , then 
Revised: Let ∆ T = (cid:80) T t =1 F ( x t ) − F ∗ , then [Equation] ⇒ ∆ [Equation]

Revised: ≤

Original: [Equation]
Revised: [Equation]

Revised: (cid:19) (cid:18) (cid:107) x

Revised: [Equation]

Revised: x

Revised: [Equation]

Revised: (cid:107)

Revised: [Equation]

Revised: γη +

Revised: [Equation]

Revised: η γ log

Revised: [Equation]

Original: (cid:19) .
Revised: ηL γb(cid:19) .

Original: Dividing both sides by T , we get the desired result.
Revised: Dividing both sides by T , we get the desired result.

Original: When F is convex (which implies γ = 1 ), using the above theorem and convexity, we obtain the following convergence rate for the average iterate:
Revised: When F is convex (which implies γ = 1 ), using the above theorem and convexity, we obtain the following convergence rate for the average iterate:

Original: Corollary 3.2.
Revised: Corollary 3.2.

Original: With Assumptions 1’ and 2, for ¯ x T = (cid:80) Tt =1 x t T , AdaGradNorm admits
Revised: With Assumptions 1’ and 2, for ¯ x T = (cid:80) Tt =1 x t T , AdaGradNorm admits

Original: [Equation]
Revised: [Equation]

Original: The rate in Theorem 3.1 can be improved by a factor 1 /γ by replacing Assumption 2 by 2’.
Revised: The rate in Theorem 3.1 can be improved by a factor 1 /γ by replacing Assumption 2 by 2’.

Original: The details and the proof are deferred into Section A.1 in the appendix.
Revised: The details and the proof are deferred into Section A.1 in the appendix.

Original: 3. S TOCHASTIC A DA G RAD N ORM
Revised: 3. S TOCHASTIC A DA G RAD N ORM

Original: In this section, we consider the stochastic setting where we only have access to an unbiased gradient estimate (cid:98) ∇ F ( x t ) of ∇ F ( x t ) (Assumption 3).
Revised: In this section, we consider the stochastic setting where we only have access to an unbiased gradient estimate (cid:98) ∇ F ( x t ) of ∇ F ( x t ) (Assumption 3).

Original: As expected for a stochastic method, the accumulation of noise is the reason that we can only expect an O (1 / √ T ) convergence rate, instead of O (1 /T ) .
Revised: As expected for a stochastic method, the accumulation of noise is the reason that we can only expect an O (1 / √ T ) convergence rate, instead of O (1 /T ) .

Original: This convergence rate is already shown by prior works (Levy et al., 2018) under the bounded domain assumption.
Revised: This convergence rate is already shown by prior works (Levy et al., 2018) under the bounded domain assumption.

Original: However, in an unbounded domain, when extending our previous analysis to the stochastic setting, that is, dividing both sides by b t , we will face several challenges.
Revised: However, in an unbounded domain, when extending our previous analysis to the stochastic setting, that is, dividing both sides by b t , we will face several challenges.

Original: One of such is the term b − 1 t (cid:104)∇ F ( x t ) − (cid:98) ∇ F ( x t ) , x t − x ∗ (cid:105) .
Revised: One of such is the term b − 1 t (cid:104)∇ F ( x t ) − (cid:98) ∇ F ( x t ) , x t − x ∗ (cid:105) .

Original: To handle this term, often we see that existing works, such as Li & Orabona (2019), analyze a modiﬁed version of Stochastic AdaGradNorm with off-byone stepsize, i.e., b t = (cid:113) b 20 + (cid:80) t − 1 i =1 (cid:107) (cid:98) ∇ F ( x i ) (cid:107) 2 in which the latest gradient (cid:98) ∇ F ( x t ) is not used to calculate b t .
Revised: To handle this term, often we see that existing works, such as Li & Orabona (2019), analyze a modiﬁed version of Stochastic AdaGradNorm with off-byone stepsize, i.e., b t = (cid:113) b 20 + (cid:80) t − 1 i =1 (cid:107) (cid:98) ∇ F ( x i ) (cid:107) 2 in which the latest gradient (cid:98) ∇ F ( x t ) is not used to calculate b t .

Original: This allows to decouple the dependency of b t on the randomness at time t , thus in expectation E [ b − 1 t (cid:104)∇ F ( x t ) − (cid:98) ∇ F ( x t ) , x t − x ∗ (cid:105) ]
Revised: This allows to decouple the dependency of b t on the randomness at time t , thus in expectation E [ b − 1 t (cid:104)∇ F ( x t ) − (cid:98) ∇ F ( x t ) , x t − x ∗ (cid:105) ]

Original: = 0 .
Revised: = 0 .

Original: Yet, this analysis does not apply to the standard algorithm which is more commonly used in practice.
Revised: Yet, this analysis does not apply to the standard algorithm which is more commonly used in practice.

Original: To the best of our knowledge, we are the ﬁrst to propose a new technique that can show the convergence of Algorithm 2 on R d without going through the off-by-one stepsize.
Revised: To the best of our knowledge, we are the ﬁrst to propose a new technique that can show the convergence of Algorithm 2 on R d without going through the off-by-one stepsize.

Original: Here, we brieﬂy compared the assumptions in our analysis with the assumptions in Li & Orabona (2019).
Revised: Here, we brieﬂy compared the assumptions in our analysis with the assumptions in Li & Orabona (2019).

Original: Assumptions 2’ and 3 used in both works are standard.
Revised: Assumptions 2’ and 3 used in both works are standard.

Original: Meanwhile, Assumptions 1 ( γ -quasar convexity) and 4 (sub-Weibull noise) in our analysis are much weaker than the convexity and sub-Gaussian noise assumptions in Li & Orabona (2019).
Revised: Meanwhile, Assumptions 1 ( γ -quasar convexity) and 4 (sub-Weibull noise) in our analysis are much weaker than the convexity and sub-Gaussian noise assumptions in Li & Orabona (2019).

Original: Besides, we note that, while the guarantee in Li & Orabona (2019) is a bound on E (cid:2)(cid:113) ( (cid:80) Tt =
Revised: Besides, we note that, while the guarantee in Li & Orabona (2019) is a bound on E (cid:2)(cid:113) ( (cid:80) Tt =

Original: 1 F ( x t ) − F ( x ∗ )) /T (cid:3) , we will present a bound for
Revised: 1 F ( x t ) − F ( x ∗ )) /T (cid:3) , we will present a bound for

Original: E (cid:2) ( (cid:80) Tt =
Revised: E (cid:2) ( (cid:80) Tt =

Original: 1 F ( x t ) − F ( x ∗ )) /T (cid:3) , which is a stronger criterion that is often used in convex analysis.
Revised: 1 F ( x t ) − F ( x ∗ )) /T (cid:3) , which is a stronger criterion that is often used in convex analysis.

Original: We also remark that the algorithm and analysis of Li & Orabona (2019) still require the smoothness parameter to set the initial stepsize, thus their method is not fully adaptive.
Revised: We also remark that the algorithm and analysis of Li & Orabona (2019) still require the smoothness parameter to set the initial stepsize, thus their method is not fully adaptive.

Original: The ﬁrst observation is that, if we let ξ t := (cid:98) ∇ F ( x t ) − ∇ F ( x t ) be the stochastic error and M T := max t ∈
Revised: The ﬁrst observation is that, if we let ξ t := (cid:98) ∇ F ( x t ) − ∇ F ( x t ) be the stochastic error and M T := max t ∈

Original: [ T ] (cid:107) ξ t (cid:107) 2 , M T is bounded by σ 2 log 2 θ eTδ with probability at least 1 − δ (c.f.
Revised: [ T ] (cid:107) ξ t (cid:107) 2 , M T is bounded by σ 2 log 2 θ eTδ with probability at least 1 − δ (c.f.

Original: Lemma A.4 in
Revised: Lemma A.4 in

Original: Appendix A), which can give a high probability bound on b T .
Revised: Appendix A), which can give a high probability bound on b T .

Original: Lemma 3.3.
Revised: Lemma 3.3.

Original: Suppose F satisﬁes Assumptions 2’ and 4, if M T ≤ σ 2 log 2 θ eTδ , then
Revised: Suppose F satisﬁes Assumptions 2’ and 4, if M T ≤ σ 2 log 2 θ eTδ , then

Original: [Equation]
Revised: [Equation]

Original: Lemma 3.3 gives us an insight: b t = (cid:101)
Revised: Lemma 3.3 gives us an insight: b t = (cid:101)

Original: O (1 + σ (cid:113) t log 2 θ t ) .
Revised: O (1 + σ (cid:113) t log 2 θ t ) .

Original: Note that this can be expected since we know the classic choice of the step size for SGD is of the order of O (1 + σ √ t ) .
Revised: Note that this can be expected since we know the classic choice of the step size for SGD is of the order of O (1 + σ √ t ) .

Original: Hence, if we are willing to accept extra log terms in the convergence guarantee, the appearance of log b t is accommodatable.
Revised: Hence, if we are willing to accept extra log terms in the convergence guarantee, the appearance of log b t is accommodatable.

Original: Next we will introduce our novel technique, which, to the best of our knowledge, is the ﬁrst method that allows us to analyze the standard Stochastic AdaGradNorm on R d .
Revised: Next we will introduce our novel technique, which, to the best of our knowledge, is the ﬁrst method that allows us to analyze the standard Stochastic AdaGradNorm on R d .

Original: Lemma 3.4.
Revised: Lemma 3.4.

Original: Suppose F satisﬁes Assumptions 1 and 3 then
Revised: Suppose F satisﬁes Assumptions 1 and 3 then

Original: [Equation]
Revised: [Equation]

Original: Proof sketch.
Revised: Proof sketch.

Original: Starting from the γ -quasar convexity, with simple transformations, we obtain
Revised: Starting from the γ -quasar convexity, with simple transformations, we obtain

Original: [Equation]
Revised: [Equation]

Original: Here we introduce our novel technique: instead of dividing by b t , we divide both sides by 2 b t − b 0 .
Revised: Here we introduce our novel technique: instead of dividing by b t , we divide both sides by 2 b t − b 0 .

Original: This divisor causes a slight non-uniformity between the coefﬁcients of the distance terms (cid:107) x
Revised: This divisor causes a slight non-uniformity between the coefﬁcients of the distance terms (cid:107) x

Original: t − x ∗ (cid:107)making the sum of them not telescoping.
Revised: t − x ∗ (cid:107)making the sum of them not telescoping.

Original: However, this is exactly what we want to handle the difﬁcult term (cid:104)− ξ t ,x t − x ∗ (cid:105) γ (2 b t − b 0 ) which does not disappear after taking the expectation.
Revised: However, this is exactly what we want to handle the difﬁcult term (cid:104)− ξ t ,x t − x [Equation] (cid:105) γ [Equation]

Revised: 2 b t −

Original: [Equation]
Revised: [Equation]

Revised: 0 ) which does not disappear after taking the expectation.

Revised: [Equation]

Original: The key step is to use Cauchy-Schwarz inequality for the term
Revised: The key step is to use Cauchy-Schwarz inequality for the term

Original: [Equation]
Revised: [Equation]

Original: with the appropriate coefﬁcient λ so that the term 
Revised: with the appropriate coefﬁcient λ so that the term (cid:107) x

Original: [Equation]
Revised: [Equation]

Original: − x 
Revised: x ∗

Original: [Equation]
Revised: [Equation]

Original: 2 can be absorbed to make a telescoping sum
Revised: 2 can be absorbed to make a telescoping sum

Original: [Equation]
Revised: [Equation]

Original: The remaining terms are free of x ∗ ; hence can be more easily bounded.
Revised: The remaining terms are free of x ∗ ; hence can be more easily bounded.

Original: We can obtain
Revised: We can obtain

Original: [Equation]
Revised: [Equation]

Original: where Z t = ηγb 0 (cid:16)
Revised: where Z t = ηγb 0 (cid:16)

Original: [Equation]
Revised: [Equation]

Original: Now we have a telescoping sum
Revised: Now we have a telescoping sum

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: − b 0 ) .
Revised: b 0 ) .

Original: Taking the sum over t , we have
Revised: Taking the sum over t , we have

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:98) [Equation] (cid:13)(cid:13)(cid:13) [Equation]
Revised: 

Original:  .
Revised:  .

Original: Proceeding to bound each term, we will obtain Lemma 3.4.
Revised: Proceeding to bound each term, we will obtain Lemma 3.4.

Original: We emphasize the following crucial aspect of Lemma 3.4: the inequality gives us a relationship between the function gap and the stepsize b T , which we know how to bound with high probability under Assumptions 2’ and 4.
Revised: We emphasize the following crucial aspect of Lemma 3.4: the inequality gives us a relationship between the function gap and the stepsize b T , which we know how to bound with high probability under Assumptions 2’ and 4.

Original: On the other hand, this relationship is not ideal due to the fact that on the L.H.S. of (1), we have not obtained a decoupling between the function gap and b T .
Revised: On the other hand, this relationship is not ideal due to the fact that on the L.H.S. of (1), we have not obtained a decoupling between the function gap and b T .

Original: To this end, we introduce the second novel technique.
Revised: To this end, we introduce the second novel technique.

Original: Let ∆ T :=
Revised: Let ∆ T :=

Original: (cid:80) Tt =
Revised: (cid:80) Tt =

Original: 1 F ( x t ) − F ( x ∗ ) , we write
Revised: 1 F ( x t ) − F ( x ∗ ) , we write

Original: [Equation]
Revised: [Equation]

Original: where we deﬁne the event E ( δ )
Revised: where we deﬁne the event E ( δ )

Original: = (cid:110) M T ≤ σ 2 log 2 θ eTδ (cid:111) .
Revised: = (cid:110) M T ≤ σ 2 log 2 θ eTδ (cid:111) .

Original: For the ﬁrst term, when E ( δ ) happens, we also know from Lemma 3.3 that the stepsize is bounded.
Revised: For the ﬁrst term, when E ( δ ) happens, we also know from Lemma 3.3 that the stepsize is bounded.

Original: Thus we can bound
Revised: Thus we can bound

Original: [Equation]
Revised: [Equation]

Original: which leads us back to Lemma 3.4.
Revised: which leads us back to Lemma 3.4.

Original: We can bound the second term using a tail bound for the event
Revised: We can bound the second term using a tail bound for the event

Original: E c ( δ ) , knowing from the ﬁrst observation that Pr [ E c ( δ )] ≤ δ .
Revised: E c ( δ ) , knowing from the ﬁrst observation that Pr [ E c ( δ )] ≤ δ .

Original: From this insight, and using the self-bounding argument as in the proof of Theorem 3.1, we ﬁnally obtain the following result.
Revised: From this insight, and using the self-bounding argument as in the proof of Theorem 3.1, we ﬁnally obtain the following result.

Original: Theorem 3.5.
Revised: Theorem 3.5.

Original: Suppose F satisﬁes Assumptions 1, 2’, 3 and 4, Stochastic AdaGradNorm admits
Revised: Suppose F satisﬁes Assumptions 1, 2’, 3 and 4, Stochastic AdaGradNorm admits

Original: [Equation]
Revised: [Equation]

Original: Initialize: x 1 , η > 0 , ∆ > 0 , p t >for t = 1 to T
Revised: Initialize: x 1 , η > 0 , ∆ > 0 , p t >for t = 1 to T

Original: [Equation]
Revised: [Equation]

Original: Algorithm Last Initialize: x 1 , η > 0 , δ ∈ [2 / 3 ,1) , p t >for t = 1 to T b t = (cid:16) b 20 + (cid:80) ti =1 (cid:107)∇ F ( x i ) (cid:107) 2 p i (cid:17)x t +1 = x
Revised: Initialize: x 1 , η > 0 , δ ∈ [2 / 3 ,

Original:  t − η b δt b 1 − δ t − 1 ∇ F ( x t )
Revised: [Equation] , p t >for t = 1 to T [Equation]

Original: Remark 3.6 .
Revised: Remark 3.6 .

Original: InTheorem 3.5, when the deterministic setting is considered, i.e, by setting σ = 0 , we obtain the standard convergence rate E (cid:2) ( (cid:80) Tt =
Revised: In the big- O notation, we only show the dependency on σ, T and θ for simplicity. The dependency on the other parameters will be made explicit in the proof of the theorem. By setting σ = 0 , we obtain the standard convergence rate E (cid:2) ( (cid:80) Tt =

Original: 1 F ( x t ) − F ( x ∗ )) /T
Revised: 1 F ( x t ) − F ( x ∗ )) /T

Original: (cid:3)
Revised: (cid:3)

Original: =
Revised: =

Original: O (1 /T ) as shown in Section 3.1.
Revised: O (1 /T ) as shown in Section 3.1 for the deterministic setting.

Original: This means our analysis adapts to the noise parameter σ .
Revised: This means our analysis adapts to the noise parameter σ .

Original: Finally, it is worth pointing out that even when we relax Assumption 2’ to Assumption 2, we can still provide a convergence guarantee for Stochastic AdaGradNorm.
Revised: Finally, it is worth pointing out that even when we relax Assumption 2’ to Assumption 2, we can still provide a convergence guarantee for Stochastic AdaGradNorm.

Original: We present the result in Theorem
Revised: We present the result in Theorem

Original: A.9 in the appendix.
Revised: A.9 in the appendix.

Original:  L AST ITERATE CONVERGENCE OF VARIANTS OF A DA G RAD N ORM FOR γ - QUASAR CONVEX AND SMOOTH MINIMIZATION ON R d
Revised:  L AST ITERATE CONVERGENCE OF VARIANTS OF A DA G RAD N ORM FOR γ - QUASAR CONVEX AND SMOOTH MINIMIZATION ON R d

Original: In Section 3, under Assumptions 1 and 2, we proved
Revised: In Section 3, under Assumptions 1 and 2, we proved

Original: that the average iterate produced by AdaGradNorm converges at the 1 /T rate, i.e., (cid:16)(cid:80) Tt =1 F ( x t ) − F ∗ (cid:17) /T
Revised: that the average iterate produced by AdaGradNorm converges at the 1 /T rate, i.e., (cid:16)(cid:80) Tt =1 F ( x t ) − F ∗ (cid:17) /T

Original: =
Revised: =

Original: O (1 /T ) .
Revised: O (1 /T ) .

Original: A natural question is whether there exists an adaptive algorithm that can guarantee the convergence of the last iterate.
Revised: A natural question is whether there exists an adaptive algorithm that can guarantee the convergence of the last iterate.

Original: In this section, we give an afﬁrmative answer by presenting two simple variants of AdaGradNorm and show convergence of the last iterate under Assumptions 1 and 2’.
Revised: In this section, we give an afﬁrmative answer by presenting two simple variants of AdaGradNorm and show convergence of the last iterate under Assumptions 1 and 2’.

Original: In Algorithm 3, by setting p i = i − 1 , (cid:107)∇ F ( x i ) (cid:107) 2 has a bigger coefﬁcient than in the standard
Revised: In Algorithm 3, by setting p i = i − 1 , (cid:107)∇ F ( x i ) (cid:107) 2 has a bigger coefﬁcient than in the standard

Original: AdaGradNorm.
Revised: AdaGradNorm.

Original: Should we use the 12 -power ( ∆ = 0 ) instead of 1 2+∆ with ∆ > 0 , b t will grow faster compared with the same term in AdaGradNorm.
Revised: Should we use the 12 -power ( ∆ = 0 ) instead of 1 2+∆ with ∆ > 0 , b t will grow faster compared with the same term in AdaGradNorm.

Original: We will see later that ∆ = 0 still leads to the convergence of the last iterate.
Revised: We will see later that ∆ = 0 still leads to the convergence of the last iterate.

Original: However, we ﬁrst focus on the easier case with ∆ > 0 and state convergence rate of Algorithm 3 in Theorem 4.1.
Revised: However, we ﬁrst focus on the easier case with ∆ > 0 and state convergence rate of Algorithm 3 in Theorem 4.1.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: With Assumptions 1 and 2’, by taking p t = 1 t in Algorithm 3, we have
Revised: With Assumptions 1 and 2’, by taking p t = 1 t in Algorithm 3, we have

Original: [Equation]
Revised: [Equation]

Original: T whereh (∆ [Equation] =
Revised: where

Original: [Equation]
Revised: [Equation]

Original: log
Revised: log

Original: [Equation]
Revised: [Equation]

Original: ∆ ∈ (0 , 1) and
Revised: ∆ ∈ (0 , 1) and

Original: [Equation]
Revised: [Equation]

Original: An issue with variant 3 is that, when using 1 2+∆ -power, the stepsize ceases to be scale-invariant.
Revised: An issue with variant 3 is that, when using 1 2+∆ -power, the stepsize ceases to be scale-invariant.

Original: Algorithm 4 shows a different approach, using the scale-invariant power 12 , but a different stepsize b δt b 1 − δ t − 1 , for a constant δ ∈ [2 / 3 , 1) .
Revised: Algorithm 4 shows a different approach, using the scale-invariant power 12 , but a different stepsize b δt b 1 − δ t − 1 , for a constant δ ∈ [2 / 3 , 1) .

Original: The tradeoff is that the provable convergence rate of the second variant depends exponentially on the smoothness parameter.
Revised: The tradeoff is that the provable convergence rate of the second variant depends exponentially on the smoothness parameter.

Original: We also note that, when δ
Revised: We also note that, when δ

Original: = 1 , we obtain the same algorithm as when setting ∆ = 0 in the previous variant.
Revised: = 1 , we obtain the same algorithm as when setting ∆ = 0 in the previous variant.

Original: Remark 4.2 .
Revised: Remark 4.2 .

Original: b 0 in every algorithm is only for stabilization and is set to a constant that is very close to 0 in practice.
Revised: b 0 in every algorithm is only for stabilization and is set to a constant that is very close to 0 in practice.

Original: However, the ﬁrst stepsize in Algorithm 4, i.e., b δ 1 b 1 − δwill explode.
Revised: However, the ﬁrst stepsize in Algorithm 4, i.e., b δ 1 b 1 − δwill explode.

Original: To avoid this issue, we can simply set the ﬁrst stepsize as b 1 instead of b δ 1 b 1 − δ.
Revised: To avoid this issue, we can simply set the ﬁrst stepsize as b 1 instead of b δ 1 b 1 − δ.

Original: We note that, under this change,
Revised: We note that, under this change,

Original: Algorithm 4 still admits a provable convergence rate.
Revised: Algorithm 4 still admits a provable convergence rate.

Original: However, for simplicity, we keep b δ 1 b 1 − δin both the description of the algorithm and its analysis.
Revised: However, for simplicity, we keep b δ 1 b 1 − δin both the description of the algorithm and its analysis.

Original: Theorem 4.3.
Revised: Theorem 4.3.

Original: With Assumptions 1 and 2’, by taking p t = 1 t in Algorithm 4, we have
Revised: With Assumptions 1 and 2’, by taking p t = 1 t in Algorithm 4, we have

Original: [Equation]
Revised: [Equation]

Original: where k ( δ ) =
Revised: where k ( δ ) =

Original: [Equation]

Original: x

Original: [Equation]
Revised: [Equation]

Original: To ﬁnish this section, we brieﬂy discuss the case when ∆ = 0 in Algorithm 3 or equivalently δ
Revised: To ﬁnish this section, we brieﬂy discuss the case when ∆ = 0 in Algorithm 3 or equivalently δ

Original: = 1 in Algorithm 4.
Revised: = 1 in Algorithm 4.

Original: First, by seeing ∆ tends to 0 , we can expect a convergence rate depending exponentially on the problem parameters.
Revised: First, by seeing ∆ tends to 0 , we can expect a convergence rate depending exponentially on the problem parameters.

Original: When ∆ = 0 , while we can still expect a bound of the function gap via the ﬁnal stepsize b T , bounding b T becomes problematic.
Revised: When ∆ = 0 , while we can still expect a bound of the function gap via the ﬁnal stepsize b T , bounding b T becomes problematic.

Original: In the proof of Theorem 4.1, to bound b T , we use the sum (cid:80)
Revised: In the proof of Theorem 4.1, to bound b T , we use the sum (cid:80)

Original: Tt
Revised: Tt

Original: [Equation]
Revised: [Equation]

Original: This sum only admits a lower bound in terms of b T when ∆ > 0 , thus the argument does not work when ∆ = 0 .
Revised: This sum only admits a lower bound in terms of b T when ∆ > 0 , thus the argument does not work when ∆ = 0 .

Original: However, it is still possible to give an asymptotic rate under the γ -quasar convexity assumption.
Revised: However, it is still possible to give an asymptotic rate under the γ -quasar convexity assumption.

Original: If we further assume that F is convex, we can give a non-asymptotic rate.
Revised: If we further assume that F is convex, we can give a non-asymptotic rate.

Original: The main idea on how to bound b T is as follows.
Revised: The main idea on how to bound b T is as follows.

Original: Let τ be the last time such that b t ≤ ηL/ 2 .
Revised: Let τ be the last time such that b t ≤ ηL/ 2 .

Original: The increment from b τ +1 to b T can be bounded by observing that the increase in each step (cid:107)∇ F ( x t ) (cid:107) 2 ≤ 23 p t b 2 t .
Revised: The increment from b τ +1 to b T can be bounded by observing that the increase in each step (cid:107)∇ F ( x t ) (cid:107) 2 ≤ 23 p t b 2 t .

Original: Moreover, the critical step is the increase from b τ to b τ +1 , which again can be analyzed via the function gap and smoothness.
Revised: Moreover, the critical step is the increase from b τ to b τ +1 , which again can be analyzed via the function gap and smoothness.

Original: We present the asymptotic and a non-symptotic convergence rate and their analysis in Sections B.4 and
Revised: We present the asymptotic and a non-symptotic convergence rate and their analysis in Sections B.4 and

Original: B.5 in the appendix.
Revised: B.5 in the appendix.

Original:  A CCELERATED VARIANTS OF A DA G RAD N ORM FOR CONVEX AND SMOOTH MINIMIZATION ON R d
Revised:  A CCELERATED VARIANTS OF A DA G RAD N ORM FOR CONVEX AND SMOOTH MINIMIZATION ON R d

Original: In this section, by using the stronger Assumption 1’, we give two new algorithms that achieve the accelerated rate O (1 /T 2 ) , matching the optimal rate in T for convex and smooth optimization for unconstrained deterministic problems.
Revised: In this section, by using the stronger Assumption 1’, we give two new algorithms that achieve the accelerated rate O (1 /T 2 ) , matching the optimal rate in T for convex and smooth optimization for unconstrained deterministic problems.

Original: Our new algorithms are adapted from the acceleration scheme introduced in Auslender & Teboulle (2006) (see also Lan (2020)).
Revised: Our new algorithms are adapted from the acceleration scheme introduced in Auslender & Teboulle (2006) (see also Lan (2020)).

Original: They are also similar to existing adaptive accelerated methods designed for bounded domains, including Levy et al.
Revised: They are also similar to existing adaptive accelerated methods designed for bounded domains, including Levy et al.

Original: ; Ene et al.
Revised: ; Ene et al.

Original: However, previous analysis does not apply in unconstrained problems; we therefore have to make necessary modiﬁcations.
Revised: However, previous analysis does not apply in unconstrained problems; we therefore have to make necessary modiﬁcations.

Original: To the best of our knowledge, in unconstrained problems under the deterministic setting, the only existing analysis for an accelerated method was introduced in Antonakopoulos et al.
Revised: To the best of our knowledge, in unconstrained problems under the deterministic setting, the only existing analysis for an accelerated method was introduced in Antonakopoulos et al.

Original: Here we discuss some limitations of this work.
Revised: Here we discuss some limitations of this work.

Original: The convergence rate for the weighted average iterate x T +1 / 2 is given by
Revised: The convergence rate for the weighted average iterate x T +1 / 2 is given by

Original: [Equation]
Revised: [Equation]

Original: (cid:17)(cid:19) where h is a K h -strongly convex mirror map function, R h = max h
Revised: (cid:17)(cid:19) where h is a K h -strongly convex mirror map function, R h = max h

Original: x ) − min h ( x ) is the range of h .
Revised: x ) − min h ( x ) is the range of h .

Original: This result is only applicable when the domain is unbounded but the range of the mirror map is bounded.
Revised: This result is only applicable when the domain is unbounded but the range of the mirror map is bounded.

Original: Even in the standard (cid:96) 2 setup with h ( x ) = 12 (cid:107) x (cid:107) 2 , this assumption does not hold.
Revised: Even in the standard (cid:96) 2 setup with h ( x ) = 12 (cid:107) x (cid:107) 2 , this assumption does not hold.

Original: Moreover, due to the term lim t →∞ b t , the above guarantee is dependent on the particular function.
Revised: Moreover, due to the term lim t →∞ b t , the above guarantee is dependent on the particular function.

Original: Thus, while a standard convergence guarantee is applicable to say, all SVM models with Huber loss, the above guarantee varies for each SVM model and there is no universal bound for all of them.
Revised: Thus, while a standard convergence guarantee is applicable to say, all SVM models with Huber loss, the above guarantee varies for each SVM model and there is no universal bound for all of them.

Original: We further highlight some key differences between this work and ours.
Revised: We further highlight some key differences between this work and ours.

Original: While the convergence rate above depends on the convergence of the stepsize, for both our variants, we will show an explicit convergence rate that holds universally for the entire function class.
Revised: While the convergence rate above depends on the convergence of the stepsize, for both our variants, we will show an explicit convergence rate that holds universally for the entire function class.

Original: Second, the algorithm in Antonakopoulos et al.
Revised: Second, the algorithm in Antonakopoulos et al.

Original: (2022) is based on an extra gradient method which requires to calculate gradients twice in one iteration.
Revised: (2022) is based on an extra gradient method which requires to calculate gradients twice in one iteration.

Original: Instead, our algorithms only need one gradient computation per iteration.
Revised: Instead, our algorithms only need one gradient computation per iteration.

Original: Finally, our algorithms guarantee the convergence of the last iterate as opposed to that for the weighted average iterate as shown above.
Revised: Finally, our algorithms guarantee the convergence of the last iterate as opposed to that for the weighted average iterate as shown above.

Original: Algorithm 5 shows the ﬁrst variant.
Revised: Algorithm 5 shows the ﬁrst variant.

Original: For an accelerated method, the step size typically has the form b t = (cid:16)
Revised: For an accelerated method, the step size typically has the form b t = (cid:16)

Original: b 20 + (cid:80) ti =
Revised: b 20 + (cid:80) ti =

Original: 1 s i (cid:107)∇ F i (cid:107) 2 (cid:17) 12 where ∇ F i is the gradient evaluated at time i , and s i
Revised: 1 s i (cid:107)∇ F i (cid:107) 2 (cid:17) 12 where ∇ F i is the gradient evaluated at time i , and s i

Original: = O ( i 2 ) .
Revised: = O ( i 2 ) .

Original: However in order to be able to give an explicit convergence rate, Algorithm 5 uses a smaller b t with power 1 2+∆ , with ∆ > 0 .
Revised: However in order to be able to give an explicit convergence rate, Algorithm 5 uses a smaller b t with power 1 2+∆ , with ∆ > 0 .

Original: When ∆ = 0 , we can only show an asymptotic convergence rate, similarly to Antonakopoulos et al.
Revised: When ∆ = 0 , we can only show an asymptotic convergence rate, similarly to Antonakopoulos et al.

Original: We ﬁrst focus on the case when ∆ > 0 .
Revised: We ﬁrst focus on the case when ∆ > 0 .

Original: In the appendix we will discuss the convergence of the algorithm when ∆ = 0 .
Revised: In the appendix we will discuss the convergence of the algorithm when ∆ = 0 .

Original: We have the following theorem.
Revised: We have the following theorem.

Original: Initialize: x 1 = w 1 , η > 0 , ∆ > 0 , a t > 0 , q t >for t = 1 to T
Revised: Initialize: x 1 = w 1 , η > 0 , ∆ > 0 , a t > 0 , q t >for t = 1 to T

Original: [Equation]
Revised: [Equation]

Original: Initialize: x 1 = w 1 , η > 0 , δ ∈ [2 / 3 , 1) , a t > 0 , q t >for t = 1 to T v t =
Revised: Initialize: x 1 = w 1 , η > 0 , δ ∈ [2 / 3 , 1) , a t > 0 , q t >for t = 1 to T v t =

Original: (1 − a t ) w t + a t x t
Revised: (1 − a t ) w t + a t x t

Original: [Equation]
Revised: [Equation]

Original: vi
Revised: v

Original: [Equation]
Revised: [Equation]

Original: 2 q 2 i
Revised: q 2 i

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Theorem 5.1.
Revised: Theorem 5.1.

Original: Suppose F satisﬁes Assumptions 1’ and 2’, let a t = 2 t
Revised: Suppose F satisﬁes Assumptions 1’ and 2’, let a t = 2 t

Original: +1 , q t = 2 t in Alg.
Revised: +1 , q t = 2 t in Alg.

Original: 5, then
Revised: 5, then

Revised: F

Original: [Equation]
Revised: [Equation]

Original:  +
Revised: T +

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Similarly to the second variant in the previous section, we also have a scale-invariant accelerated algorithm, shown in Algorithm 6 using power 12 but a smaller stepsize b δt b 1 − δ t − 1 .
Revised: Similarly to the second variant in the previous section, we also have a scale-invariant accelerated algorithm, shown in Algorithm 6 using power 12 but a smaller stepsize b δt b 1 − δ t − 1 .

Original: This algorithm also has an exponential dependency on the problem parameters, which is given in the following theorem.
Revised: This algorithm also has an exponential dependency on the problem parameters, which is given in the following theorem.

Original: Remark 5.2 .
Revised: Remark 5.2 .

Original: Similar to Remark 4.2, the ﬁrst stepsize in Algorithm 6, i.e., b δ 1 b 1 − δcan be replaced by b 1 .
Revised: Similar to Remark 4.2, the ﬁrst stepsize in Algorithm 6, i.e., b δ 1 b 1 − δcan be replaced by b 1 .

Original: However, for simplicity, we keep b δ 1 b 1 − δin both the description of the algorithm and its analysis.
Revised: However, for simplicity, we keep b δ 1 b 1 − δin both the description of the algorithm and its analysis.

Original: Theorem 5.3.
Revised: Theorem 5.3.

Original: Suppose F satisﬁes Assumptions 1’ and 2’, let a t = 2 t
Revised: Suppose F satisﬁes Assumptions 1’ and 2’, let a t = 2 t

Original: +1 , q t = 2 t in Alg. 6, then
Revised: +1 , q t = 2 t in Alg. 6, then

Original: [Equation]
Revised: [Equation]

Original: where s ( δ ) = (cid:107) x ∗ − x 1 (cid:107) 2 2 η
Revised: where s ( δ ) = (cid:107) x ∗ − x 1 (cid:107) 2 2 η

Original: [Equation]
Revised: [Equation]

Original: Similarly to the previous section, we give a more detailed discussion of the convergence of the
Revised: Similarly to the previous section, we give a more detailed discussion of the convergence of the

Original: Algorithm 5 when ∆ = 0 or equivalently Algorithm 6 when δ = 1 in Section C.4 in the appendix.
Revised: Algorithm 5 when ∆ = 0 or equivalently Algorithm 6 when δ = 1 in Section C.4 in the appendix.

Original: While we can still show an accelerated O (1 /T 2 ) asymptotic convergence rate, we only present an
Revised: While we can still show an accelerated O (1 /T 2 ) asymptotic convergence rate, we only present an

Original: O (1 /T 2 + 1 /T ) non-asymptotic rate.
Revised: O (1 /T 2 + 1 /T ) non-asymptotic rate.

Original: The difference between these algorithms and the ones in the previous section is that the stepsize b t increases much faster.
Revised: The difference between these algorithms and the ones in the previous section is that the stepsize b t increases much faster.

Original: More precisely, the increment in each step is now O ( t 2 (cid:107)∇ F ( v t ) (cid:107) 2 ) instead of O ( t (cid:107)∇ F ( x t ) (cid:107) 2 ) .
Revised: More precisely, the increment in each step is now O ( t 2 (cid:107)∇ F ( v t ) (cid:107) 2 ) instead of O ( t (cid:107)∇ F ( x t ) (cid:107) 2 ) .

Original: Thus we can only show an upperbound for b t that grows linearly with time, which leads to the O (1 /T 2 + 1 /T ) convergence rate.
Revised: Thus we can only show an upperbound for b t that grows linearly with time, which leads to the O (1 /T 2 + 1 /T ) convergence rate.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: In this paper, we go back to the most basic AdaGrad algorithm and study its convergence rate in generalized smooth convex optimization.
Revised: In this paper, we go back to the most basic AdaGrad algorithm and study its convergence rate in generalized smooth convex optimization.

Original: We prove explicit convergence guarantees for unconstrained problems in both the deterministic and stochastic setting.
Revised: We prove explicit convergence guarantees for unconstrained problems in both the deterministic and stochastic setting.

Original: Building on these insights, we propose new algorithms that exhibit last iterate convergence, with and without acceleration.
Revised: Building on these insights, we propose new algorithms that exhibit last iterate convergence, with and without acceleration.

Original: We see our work as primarily theoretical since the ﬁrst and foremost goal is to understand properties of existing algorithms that work well in practice.
Revised: We see our work as primarily theoretical since the ﬁrst and foremost goal is to understand properties of existing algorithms that work well in practice.

Original: We refer the reader to the long line of previous works (Duchi et al., 2011; Levy, 2017; Kavis et al., 2019; Bach & Levy, 2019;
Revised: We refer the reader to the long line of previous works (Duchi et al., 2011; Levy, 2017; Kavis et al., 2019; Bach & Levy, 2019;

Original: Antonakopoulos et al., 2020; Ene et al., 2021; Ene & Nguyen, 2022; Antonakopoulos et al., 2022) that have already demonstrated the behavior of AdaGrad and accelerated adaptive algorithms empirically. 
Revised: Antonakopoulos et al., 2020; Ene et al., 2021; Ene & Nguyen, 2022; Antonakopoulos et al., 2022) that have already demonstrated the behavior of AdaGrad and accelerated adaptive algorithms empirically. A CKNOWLEDGMENTS

Revised: TN and AE were supported in part by NSF CAREER grant CCF-1750333, NSF grant III-1908510, and an Alfred P. Sloan Research Fellowship.

Revised: HN was supported in part by NSF CAREER grant

Revised: CCF-1750716 and NSF grant CCF-1909314.

Original: Reproducibility Statement.
Revised: Reproducibility Statement.

Original: We include the full proofs of all theorems in the Appendix.
Revised: We include the full proofs of all theorems in the Appendix

Original: A M ISSING PROOFS FROM S ECTIONA. A DA G RAD N ORM
Revised: A M ISSING PROOFS FROM S ECTIONA. A DA G RAD N ORM

Original: As we pointed out before, it is possible to obtain an improvement by a factor 1 /γ compared with
Revised: As we pointed out before, it is possible to obtain an improvement by a factor 1 /γ compared with

Original: Theorem 3.1 by assuming L -smoothness instead of weak L -smoothness
Revised: Theorem 3.1 by assuming L -smoothness instead of weak L -smoothness

Original: [Equation]
Revised: [Equation]

Original: Theorem A.1.
Revised: Theorem A.1.

Original: With Assumptions 1 and 2’
Revised: With Assumptions 1 and 2’

Original: [Equation]
Revised: [Equation]

Original: AdaGradNorm admits
Revised: AdaGradNorm admits

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Note that Assumption 2’ can imply Assumption [Equation] so following the same proof of Theorem 3.
Revised: Note that Assumption 2’ can imply Assumption 2, so following the same proof of Theorem 3.

Original: [Equation]
Revised: [Equation]

Original: we still have
Revised: we still have

Original: [Equation]
Revised: [Equation]

Original: However, from here
Revised: However, from here

Original: [Equation]
Revised: [Equation]

Original: we will bound b T directly
Revised: we will bound b T directly

Original: [Equation]
Revised: [Equation]

Original: rathe than use the self bounded argument in the previous proof.
Revised: rathe than use the self bounded argument in the previous proof.

Original: By the L -smoothness
Revised: By the L -smoothness

Original: [Equation]
Revised: [Equation]

Original: we know
Revised: we know

Original: [Equation]
Revised: [Equation]

Original: Sum up from 1 to T , we know
Revised: Sum up from 1 to T , we know

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: Use the the same proof technique as before, we can bound
Revised: Use the the same proof technique as before, we can bound

Original: T (cid:88) t = 
Revised: T (cid:88) t = (cid:18)

Original: [Equation]
Revised: [Equation]

Original: and

Original: T (cid:88) t = (cid:107)∇ F ( x t ) (cid:107)
Revised: (cid:107)∇ [Equation] x [Equation]

Original: [Equation]
Revised: [Equation]

Original: b
Revised: +

Original: [Equation]
Revised: [Equation]

Original:  =
Revised: and T (cid:88) t = (cid:107)∇ F ( x t ) (cid:107)b t =

Original: T (cid:88) t =
Revised: T (cid:88) t =

Original: b 2 t − b 2 t −
Revised: b 2 t − b 2 t −b t ≥

Revised: T (cid:88) t = b

Original: [Equation]
Revised: [Equation]

Original:  b [Equation] ≥
Revised: = b T − b 0 .

Original: [Equation]

Original: Hence, we know
Revised: Hence, we know

Original: [Equation]
Revised: [Equation]

Original: By using this bound on b T , we can get the ﬁnal result with an improvement by a factor 1 /γ .
Revised: By using this bound on b T , we can get the ﬁnal result with an improvement by a factor 1 /γ .

Original: A. S TOCHASTIC A DA G RAD N ORM
Revised: A. S TOCHASTIC A DA G RAD N ORM

Original: We will employ the following notations for convenience
Revised: We will employ the following notations for convenience

Original: [Equation]
Revised: [Equation]

Original: Before diving into the details of our proof
Revised: Before diving into the details of our proof

Original: [Equation]
Revised: [Equation]

Original: we ﬁrst present some technical results we will use in the proof of Theorem 3.
Revised: we ﬁrst present some technical results we will use in the proof of Theorem 3.

Original: [Equation]
Revised: [Equation]

Original: A.2. T ECHNICAL LEMMAS
Revised: A.2. T ECHNICAL LEMMAS

Original: To start with, under Assumptions 1 and 3 only [Equation] we can obtain a bound for a term close to our ﬁnal goal ∆ T .
Revised: To start with, under Assumptions 1 and 3 only, we can obtain a bound for a term close to our ﬁnal goal ∆ T .

Original: Lemma A.2.
Revised: Lemma A.2.

Original: (Lemma 3.4) Suppose F satisﬁes Assumptions 1 and 3, we have
Revised: (Lemma 3.4) Suppose F satisﬁes Assumptions 1 and 3, we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We start by using the γ -quasar convexity of the function F
Revised: We start by using the γ -quasar convexity of the function F

Original: [Equation]
Revised: [Equation]

Original: Dividing both sides by 2 b t − b 0 and taking expactations, we have
Revised: Dividing both sides by 2 b t − b 0 and taking expactations, we have

Original: [Equation]
Revised: [Equation]

Original: Now we no longer have a telescoping sum in the R.H.S..
Revised: Now we no longer have a telescoping sum in the R.H.S..

Original: However, this is exactly what we want to handle the difﬁcult term (cid:104)− ξ t ,x t − x ∗ (cid:105) γ (2 b t − b 0 ) which does not disappear after taking the expectation.
Revised: However, this is exactly what we want to handle the difﬁcult term (cid:104)− ξ t ,x t − x ∗ [Equation] γ (2 b t − b 0 ) which does not disappear after taking the expectation.

Original: The key step is to use Cauchy-Schwarz inequality for the term
Revised: The key step is to use Cauchy-Schwarz inequality for the term

Original: [Equation]
Revised: [Equation]

Original: with the appropriate coefﬁcient λ so that the term (cid:107) x t − x ∗ (cid:107) 2 can be absorbed to make a telescoping sum
Revised: with the appropriate coefﬁcient λ so that the term (cid:107) x t − x ∗ (cid:107) 2 can be absorbed to make a telescoping sum

Original: [Equation]
Revised: [Equation]

Original: The remaining terms are free of x ∗ ; hence can be more easily
Revised: The remaining terms are free of x ∗ ; hence can be more easily

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Thus we have
Revised: Thus we have

Original: [Equation]
Revised: [Equation]

Original: (cid:21) ≤ E (cid:20) η γb(cid:18) 2 b t − 1 − b 0 − 2 b t − b(cid:19) (cid:107) ξ t (cid:107) 2 (cid:21) + E (cid:34) b t − 1 (cid:107) x
Revised: (cid:21) ≤ E (cid:20) η γb(cid:18) 2 b t − 1 − b 0 − 2 b t − b(cid:19) (cid:107) ξ t (cid:107) 2 (cid:21) + E (cid:34) b t − 1 (cid:107) x

Original: t − x ∗ (cid:107)2 γη (2 b t − 1 − b 0 ) − b t (cid:107) x
Revised: t − x ∗ (cid:107)2 γη (2 b t − 1 − b 0 ) − b t (cid:107) x

Original: t +1 − x ∗ (cid:107)2 γη (2 b t − b 0 ) (cid:35) + E   η (cid:13)(cid:13)(cid:13)(cid:98) ∇ F ( x t ) (cid:13)(cid:13)(cid:13)2 γb t (2 b t − b 0 ) 
Revised: t +1 − x ∗ (cid:107)2 γη (2 b t − b 0 ) (cid:35) + E   η (cid:13)(cid:13)(cid:13)(cid:98) ∇ F ( x t ) (cid:13)(cid:13)(cid:13)2 γb t (2 b t − b 0 ) 

Original:  ≤ E (cid:20) η γb(cid:18) 2 b t − 1 − b 0 − 2 b t − b(cid:19) (cid:107) ξ t (cid:107) 2 (cid:21) + E (cid:34) b t − 1 (cid:107) x
Revised:  ≤ E (cid:20) η γb(cid:18) 2 b t − 1 − b 0 − 2 b t − b(cid:19) (cid:107) ξ t (cid:107) 2 (cid:21) + E (cid:34) b t − 1 (cid:107) x

Original: t − x ∗ (cid:107)2 γη (2 b t − 1 − b 0 ) − b t (cid:107) x
Revised: t − x ∗ (cid:107)2 γη (2 b t − 1 − b 0 ) − b t (cid:107) x

Original: t +1 − x ∗ (cid:107)2 γη (2 b t − b 0 ) (cid:35) + E   η (cid:13)(cid:13)(cid:13)(cid:98) ∇ F ( x t ) (cid:13)(cid:13)(cid:13)2 γb 2 t 
Revised: t +1 − x ∗ (cid:107)2 γη (2 b t − b 0 ) (cid:35) + E   η (cid:13)(cid:13)(cid:13)(cid:98) ∇ F ( x t ) (cid:13)(cid:13)(cid:13)2 γb 2 t 

Original: 
Revised: 

Original: Now we have a telescoping sum b t − 1 (cid:107) x
Revised: Now we have a telescoping sum b t − 1 (cid:107) x

Original: t − x ∗ (cid:107)2 γη (2 b t − 1 − b 0 ) − b t (cid:107) x
Revised: t − x ∗ (cid:107)2 γη (2 b t − 1 − b 0 ) − b t (cid:107) x

Original: t +1 − x ∗ (cid:107)2 γη (2 b t − b 0 ) and the remaining terms are free of x t − x ∗ .
Revised: t +1 − x ∗ (cid:107)2 γη (2 b t − b 0 ) and the remaining terms are free of x t − x ∗ .

Original: Taking the sum over t , we have
Revised: Taking the sum over t , we have

Original: [Equation]
Revised: [Equation]

Original: First for the easy term E
Revised: First for the easy term E

Original: [Equation]
Revised: [Equation]

Original: , we have
Revised: , we have

Original: [Equation]
Revised: [Equation]

Original: Next, we bound
Revised: Next, we bound

Original: E
Revised: E

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Plugging the bounds (3) and (4) into (2), we have
Revised: Plugging the bounds (3) and (4) into (2), we have

Original: E
Revised: E

Revised: [Equation]

Revised: T

Original: [Equation]
Revised: [Equation]

Original: The last step is using (cid:80) Tt 
Revised: The last step is using (cid:80) Tt =

Original: =1 F [Equation] x t ) 
Revised: 1 F ( x t ) − F ∗ 2 b t ≥

Original: [Equation]
Revised: [Equation]

Original: F ∗
Revised: Tt

Original: [Equation]
Revised: [Equation]

Original: b t ≥
Revised: 1 F

Original: [Equation]
Revised: [Equation]

Original: F
Revised: x

Original: [Equation]
Revised: [Equation]

Original: F ∗
Revised: F ∗

Original: [Equation]
Revised: [Equation]

Original: T = ∆ T 2 b T to ﬁnish the proof.
Revised: T = ∆ T 2 b T to ﬁnish the proof.

Original: Due to the appearance of M T in Lemma A.2, it is natural to consider what we can obtain under the additional Assumption 4, i.e., sub-Weibull noise with parameter θ .
Revised: Due to the appearance of M T in Lemma A.2, it is natural to consider what we can obtain under the additional Assumption 4, i.e., sub-Weibull noise with parameter θ .

Original: We ﬁrst provide the following simple bound on E (cid:2) (cid:107) ξ t (cid:107) 2 (cid:3) .
Revised: We ﬁrst provide the following simple bound on E (cid:2) (cid:107) ξ t (cid:107) 2 (cid:3) .

Original: The result is not new and the proof is only included for completeness.
Revised: The result is not new and the proof is only included for completeness.

Original: Lemma A.3.
Revised: Lemma A.3.

Original: Under Assumption 4, ∀ t ∈ [ T ] , we have
Revised: Under Assumption 4, ∀ t ∈ [ T ] , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We ﬁrst note that from the deﬁnition of sub-Weibull noise, the tail of (cid:107) ξ t (cid:107) can be bounded as follows
Revised: We ﬁrst note that from the deﬁnition of sub-Weibull noise, the tail of (cid:107) ξ t (cid:107) can be bounded as follows

Original: [Equation]
Revised: [Equation]

Original: Then we can obtain
Revised: Then we can obtain

Original: [Equation]
Revised: [Equation]

Original: where u is substituted by σv θ in the second equation.
Revised: where u is substituted by σv θ in the second equation.

Original: Next, we prove a high probability bound on M T , the proof of which is inspired by Lemma 5 in Li & Orabona (2020).
Revised: Next, we prove a high probability bound on M T , the proof of which is inspired by Lemma 5 in Li & Orabona (2020).

Original: Lemma A.4.
Revised: Lemma A.4.

Original: Under Assumption 4, given 0 < δ < 1 , deﬁne the event
Revised: Under Assumption 4, given 0 < δ < 1 , deﬁne the event

Original: [Equation]
Revised: [Equation]

Original: we have Pr [ E ( δ )] ≥ 1 − δ .
Revised: we have Pr [ E ( δ )] ≥ 1 − δ .

Original: Proof.
Revised: Proof.

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Choose u
Revised: Choose u

Original: = σ 2 log 2 θ eTδ to obtain
Revised: = σ 2 log 2 θ eTδ to obtain

Original: [Equation]
Revised: [Equation]

Original: Lastly, we will ﬁnd an upper bound on the p -th moment of M T .
Revised: Lastly, we will ﬁnd an upper bound on the p -th moment of M T .

Original: Lemma A.5.
Revised: Lemma A.5.

Original: Under Assumption 4, given p > 0 , there is
Revised: Under Assumption 4, given p > 0 , there is

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Note that in Lemma A.4, we proved
Revised: Note that in Lemma A.4, we proved

Original: [Equation]
Revised: [Equation]

Original: Let E ( δ ) be the same as it in Lemma A.4.
Revised: Let E ( δ ) be the same as it in Lemma A.4.

Original: Then, by Holder’s inequality we have
Revised: Then, by Holder’s inequality we have

Original: [Equation]
Revised: [Equation]

Original: Choose δ = 1 Γ(4 θp +1) eT < 1 , we have
Revised: Choose δ = 1 Γ(4 θp +1) eT < 1 , we have

Original: [Equation]
Revised: [Equation]

Original: Note that all the above results only depend on Assumptions 1, 3 and 4 without requiring the smoothness of F .
Revised: Note that all the above results only depend on Assumptions 1, 3 and 4 without requiring the smoothness of F .

Original: A.2. P ROOF OF T HEOREM 3. 
Revised: A.2. P ROOF OF T HEOREM 3.

Original: Theorem 3.5 requires Assumption 2’ additionally.
Revised: Theorem 3.5 requires Assumption 2’ additionally.

Original: Thus we ﬁrst show that under Assumptions 2’ and 4.
Revised: Thus we ﬁrst show that under Assumptions 2’ and 4.

Original: b T enjoys a (cid:101) O
Revised: b T enjoys a (cid:101) O

Original: (1 + σ (cid:113) T log 2 θ T ) upper bound with high probability.
Revised: (1 + σ (cid:113) T log 2 θ T ) upper bound with high probability.

Original: Lemma A.6.
Revised: Lemma A.6.

Original: (Lemma 3.3) Suppose F satisﬁes Assumptions 2’and 4.
Revised: (Lemma 3.3) Suppose F satisﬁes Assumptions 2’and 4.

Original: Under the event E ( δ ) =
Revised: Under the event E ( δ ) =

Original: (cid:110)
Revised: (cid:110)

Original: [Equation] eTδ
Revised: M

Original: [Equation]
Revised: [Equation]

Original:  , we have
Revised: 2 log 2 θ eTδ (cid:111) , we have

Original: [Equation]
Revised: [Equation]

Original: Additionally, by Lemma A.4, there is
Revised: Additionally, by Lemma A.4, there is

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We start by using the smoothness of F
Revised: We start by using the smoothness of F

Original: [Equation]
Revised: [Equation]

Original: + η 2 L 2 b 2 t (cid:107) (cid:98) ∇ F ( x t ) (cid:107)⇒ (cid:107) (cid:98) ∇ F ( x t ) (cid:107)b t ≤η ( F ( x t ) − F ( x t +1 ))
Revised: η 2 L 2 b 2 t (cid:107) (cid:98) ∇ F ( x t ) (cid:107)⇒ (cid:107) (cid:98) ∇ F ( x t ) (cid:107)b t ≤η ( F ( x t ) − F ( x t +1 ))

Original: + 2 (cid:104) ξ t , (cid:98) ∇ F ( x t )
Revised: + 2 (cid:104) ξ t , (cid:98) ∇ F ( x t )

Original: (cid:105) b t + (cid:18) ηL b 2 t − b t (cid:19) (cid:107) (cid:98) ∇ F ( x t ) (cid:107) 2 .
Revised: (cid:105) b t + (cid:18) ηL b 2 t − b t (cid:19) (cid:107) (cid:98) ∇ F ( x t ) (cid:107) 2 .

Original: Taking the sum over t we have
Revised: Taking the sum over t we have

Original: T (cid:88) t
Revised: T (cid:88) t

Original: = (cid:107) (cid:98) ∇ F ( x t ) (cid:107)b
Revised: = (cid:107) (cid:98) ∇ F ( [Equation] (cid:107)b

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ( F [Equation] x

Original: [Equation]
Revised: [Equation]

Original: ( F ( x 1 ) − F ∗ ) η +
Revised: ) − F [Equation] ) η +

Original: T (cid:88) t
Revised: T (cid:88) t

Original: = (cid:104) ξ t , (cid:98) ∇ F ( x t ) (cid:105) b t +
Revised: = (cid:104) ξ t , (cid:98) ∇ F ( x t ) (cid:105) b t +

Original: T (cid:88) t = (cid:18)
Revised: T (cid:88) t = (cid:18)

Original: ηL b 2 t − b t (cid:19) (cid:107) (cid:98) ∇ F ( x t ) (cid:107) 2 .
Revised: ηL b 2 t − b t (cid:19) (cid:107) (cid:98) ∇ F ( x t ) (cid:107) 2 .

Original: Using the common technique, we know that (cid:80) Tt =1 (cid:16) ηLb 2 t − 1 b t (cid:17) (cid:107) (cid:98) ∇ F ( x t ) (cid:107) 2 ≤ 2 ηL log + ηLb 0 .
Revised: Using the common technique, we know that (cid:80) Tt =1 (cid:16) ηLb 2 t − 1 b t (cid:17) (cid:107) (cid:98) ∇ F ( x t ) (cid:107) 2 ≤ 2 ηL log + ηLb 0 .

Original: Moreover, for the L.H.S.
Revised: Moreover, for the L.H.S.

Original: T (cid:88) t
Revised: T (cid:88) t

Original: = (cid:107) (cid:98) ∇ F ( x t ) (cid:107)b t =
Revised: = (cid:107) (cid:98) ∇ F ( x t ) (cid:107)b t =

Original: T (cid:88) t =
Revised: T (cid:88) t =

Original: b 2 t − b 2 t −b t ≥
Revised: b 2 t − b 2 t −b t ≥

Original: T (cid:88) t =
Revised: T (cid:88) t =

Original: b t − b t − 1 =
Revised: b t − b t − 1 =

Original: b T − b 0 .
Revised: b T − b 0 .

Original: Thus we have 
Revised: Thus we have b

Original: [Equation]
Revised: [Equation]

Revised: ≤

Revised: [Equation]

Revised: ) −

Original:  For the last term in this equation, we notice that (cid:104) ξ t , (cid:98) ∇ F ( x t ) (cid:105) ≤ (cid:107)ξ t (cid:107)(cid:107) (cid:98) ∇ F ( x t ) (cid:107) ≤ √
Revised: [Equation] ∗ [Equation] For the last term in this equation, we notice that (cid:104) ξ t , (cid:98) ∇ F ( x t ) (cid:105) ≤ (cid:107)

Original:  M 
Revised: [Equation] t (cid:107)(cid:107) (cid:98) ∇ F ( x t ) (cid:107) ≤ √ M T (cid:107)

Original: [Equation]
Revised: [Equation]

Original:  F ( x t ) (cid:107) , hence
Revised: ∇ F ( x t ) (cid:107) , hence

Original: [Equation]
Revised: [Equation]

Original: where ( a ) is due to Jensen’s inequality.
Revised: where ( a ) is due to Jensen’s inequality.

Original: We can write
Revised: We can write

Original: [Equation]
Revised: [Equation]

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: which gives us
Revised: which gives us

Original: [Equation]
Revised: [Equation]

Original: Recall the deﬁnition of the event E ( δ ) is M T ≤ σ 2 log 2 θ eTδ , thus we know
Revised: Recall the deﬁnition of the event E ( δ ) is M T ≤ σ 2 log 2 θ eTδ , thus we know

Original: [Equation]
Revised: [Equation]

Original: By using Lemma A.
Revised: By using Lemma A.

Original: [Equation]
Revised: [Equation]

Original: we can consider the following decomposition
Revised: we can consider the following decomposition

Original: [Equation]
Revised: [Equation]

Original: ∆ T 1 E c ( δ ) (cid:3) ≤ g T ( δ ) E (cid:20) ∆ T b T 1 E ( δ ) (cid:21) + E (cid:2) ∆ T 1 E c ( δ ) (cid:3) ≤ g T ( δ ) E (cid:20) ∆ T b T (cid:21) + E (cid:2)
Revised: ∆ T 1 E c ( δ ) (cid:3) ≤ g T ( δ ) E (cid:20) ∆ T b T 1 E ( δ ) (cid:21) + E (cid:2) ∆ T 1 E c ( δ ) (cid:3) ≤ g T ( δ ) E (cid:20) ∆ T b T (cid:21) + E (cid:2)

Original: ∆ T 1 E c ( δ ) (cid:3) .
Revised: ∆ T 1 E c ( δ ) (cid:3) .

Original: Note that Lemma A.2 tells us
Revised: Note that Lemma A.2 tells us

Original: [Equation]
Revised: [Equation]

Original: Hence our remaining task is to ﬁnd a proper bound on E (cid:2)
Revised: Hence our remaining task is to ﬁnd a proper bound on E (cid:2)

Original: ∆ T 1 E c ( δ ) (cid:3) , which is stated in the following lemma.
Revised: ∆ T 1 E c ( δ ) (cid:3) , which is stated in the following lemma.

Original: Lemma A.7.
Revised: Lemma A.7.

Original: Under Assumptions 2’ and 4 we have
Revised: Under Assumptions 2’ and 4 we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We restart from the smoothness of F :
Revised: We restart from the smoothness of F :

Original: [Equation]
Revised: [Equation]

Original: Taking the sum over s , we have for t ≥
Revised: Taking the sum over s , we have for t ≥

Original: [Equation]
Revised: [Equation]

Original: Following the same proof of Lemma A
Revised: Following the same proof of Lemma A

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Now we bound ∆ T as follows
Revised: Now we bound ∆ T as follows

Original: [Equation]
Revised: [Equation]

Original: Thus we obtain
Revised: Thus we obtain

Original: [Equation]
Revised: [Equation]

Original: Here we invoke Holder’s inequality for three variables: for p, q, r > 0 , 1 /p
Revised: Here we invoke Holder’s inequality for three variables: for p, q, r > 0 , 1 /p

Original: + 1 /q + 1 /r
Revised: + 1 /q + 1 /r

Original: = 1 then
Revised: = 1 then

Original: E [ XY Z ] ≤ E 1 /p
Revised: E [ XY Z ] ≤ E 1 /p

Original: [ X p ] E 1 /q
Revised: [ X p ] E 1 /q

Original: [ Y q ] E 1 /r [ Z r ] .
Revised: [ Y q ] E 1 /r [ Z r ] .

Original: By substituting X = √ M T , Y = (cid:113) log b 2 T b 20 , Z = 1 E c ( δ ) , and p = 4 , q = 2 , r = 4 , we have
Revised: By substituting X = √ M T , Y = (cid:113) log b 2 T b 20 , Z = 1 E c ( δ ) , and p = 4 , q = 2 , r = 4 , we have

Original: [Equation]
Revised: [Equation]

Original: So ﬁnally we get
Revised: So ﬁnally we get

Original: [Equation]
Revised: [Equation]

Original: Lemma A.8.
Revised: Lemma A.8.

Original: Suppose F satisﬁes Assumptions 1, 2’, 3 and 4 then
Revised: Suppose F satisﬁes Assumptions 1, 2’, 3 and 4 then

Original: [Equation]
Revised: [Equation]

Original: √ T where C 1 = 2 (2 θ − 1)
Revised: √ T where C 1 = 2 (2 θ − 1)

Original: + log 2 θ (cid:0) Γ(4 θ + 1)
Revised: + log 2 θ (cid:0) Γ(4 θ + 1)

Original: e 2 (cid:1) + 1 and C 2 = 2 ( θ − 1) + log θ (cid:0) Γ(8 θ + 1) e 2 (cid:1) + 1 are two constants and
Revised: e 2 (cid:1) + 1 and C 2 = 2 ( θ − 1) + log θ (cid:0) Γ(8 θ + 1) e 2 (cid:1) + 1 are two constants and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: As stated above
Revised: As stated above

Original: [Equation]
Revised: [Equation]

Original: we know
Revised: we know

Original: [Equation]
Revised: [Equation]

Original: where ( a ) is due to
Revised: where ( a ) is due to

Original: Lemma A
Revised: Lemma A

Original: [Equation]
Revised: [Equation]

Original: ( b ) is by Lemma A.2.
Revised: ( b ) is by Lemma A.2.

Original: Lemma A.7 gives us
Revised: Lemma A.7 gives us

Original: [Equation]
Revised: [Equation]

Original: Pluggin in this bound
Revised: Pluggin in this bound

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Now we take δ = T − 4 and let g T :=
Revised: Now we take δ = T − 4 and let g T :=

Original: g T ( T − 4 ) to obtain
Revised: g T ( T − 4 ) to obtain

Original: E [∆ T ]
Revised: E [∆ T ]

Original: [Equation]
Revised: [Equation]

Original: From Lemma A.5, we know
Revised: From Lemma A.5, we know

Original: E [
Revised: E [

Original: [Equation]

Original: [Equation]

Original: log

Original: [Equation]
Revised: [Equation]

Original: T )

Original: [Equation]

Original: C 1 ) and
Revised: C 1 ) and

Original: E(cid:2) M [Equation] log
Revised: E

Original: [Equation]
Revised: [Equation]

Original: + 1) ⇒E [Equation] [Equation] (cid:2)
Revised: ) ⇒

Original: M

Original: [Equation]
Revised: [Equation]

Original: = σ (cid:16) log

Original: [Equation] [Equation] ≤ [Equation]
Revised: =

Original: log

Original: [Equation]
Revised: [Equation]

Original: C

Original: [Equation]

Original: Hence we have
Revised: Hence we have

Original: E [∆ T ] ≤g T 
Revised: E [∆ T ] ≤

Original: [Equation]
Revised: [Equation]

Original: +
Revised: 

Original: [Equation]
Revised: [Equation]

Original: With these results, we can ﬁnally show the theorem 3.5.
Revised: With these results, we can ﬁnally show the theorem 3.5.

Original: Proof of Theorem 3.5 .
Revised: Proof of Theorem 3.5 .

Original: The key technique we use is the self-bounding argument.
Revised: The key technique we use is the self-bounding argument.

Original: That is, we have expressed a bound for E [∆ T ] via E [ b 2 T /b 20 ] , now we will show how to bound this term via ∆ T .
Revised: That is, we have expressed a bound for E [∆ T ] via E [ b 2 T /b 20 ] , now we will show how to bound this term via ∆ T .

Original: To do this, we rely on the smoothness assumption and
Revised: To do this, we rely on the smoothness assumption and

Original: Lemma A.
Revised: Lemma A.

Original: E

Original: [Equation]

Original: E

Original: [Equation]

Original: [Equation]

Original: T (cid:88) t

Original: [Equation]

Original: (cid:35) ≤

Original: [Equation]

Original: E (cid:34) T (

Original: cid:88) t

Original: [Equation]

Original: (cid:107)

Original: [Equation]

Original: t (cid:107)

Original: [Equation]

Original: (cid:35) + E (cid:34) T (cid:88) t =

Original: [Equation]
Revised: [Equation]

Original: (cid:107)∇ F ( x t ) (cid:107)(cid:35) ≤ b 20 + 2Γ(2 θ + 1) eσ 2 T + E (cid:34) 4 L

Original: T (cid:88) t =

Original: F ( x t )

Original: [Equation]

Original: F ( x ∗

Original: [Equation]

Original: (cid:35) ≤ b 20 +

Original: [Equation]

Original: (2 θ + 1) eσ 2 T +

Original: [Equation]

Original: L E [∆ T ] .

Original: Thus from Lemma A.8 we can write
Revised: Thus from Lemma A.8 we can write

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Now we solve (5).
Revised: Now we solve (5).

Original: Consider two cases:
Revised: Consider two cases:

Original: If 4 L E [∆ T ] ≤ 2Γ(2 θ + 1) eσ 2 T then 
Revised: If 4 L E [∆ T ] ≤ 2Γ(2 θ + 1) eσ 2 T then E [∆ [Equation]

Original: E
Revised: G

Original: [Equation]
Revised: [Equation]

Revised: G 1 log

Original: [Equation]
Revised: [Equation]

Revised: + 4Γ(2 θ + 1) eσ 2 T b(cid:19) .

Original: If 4 L E [∆ T ] ≥ 2Γ(2 θ + 1) eσ 2 T then
Revised: If 4 L E [∆ T ] ≥ 2Γ(2 θ + 1) eσ 2 T then

Original: E [∆ T ] ≤ G 0 + G 1 log 
Revised: E [∆ T ] ≤ G 0 + G 1 log (cid:18)

Original: [Equation]
Revised: [Equation]

Original: [∆
Revised: [∆

Original: [Equation]
Revised: [Equation]

Original: In both cases
Revised: In both cases

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Dividing both sides by T concludes the proof.
Revised: Dividing both sides by T concludes the proof.

Original: A.2. C ONVERGENCE OF S TOCHASTIC A DA G RAD N ORM UNDER WEAKER ASSUMPTIONS
Revised: A.2. C ONVERGENCE OF S TOCHASTIC A DA G RAD N ORM UNDER WEAKER ASSUMPTIONS

Original: Note that Theorem 3.5 depends on the stronger Assumption 2’ instead of Assumption 2.
Revised: Note that Theorem 3.5 depends on the stronger Assumption 2’ instead of Assumption 2.

Original: Besides, in Section 3.1, we proved that Assumptions 1 and 2 are enough to ensure that AdaGradNorm can converge in the deterministic setting.
Revised: Besides, in Section 3.1, we proved that Assumptions 1 and 2 are enough to ensure that AdaGradNorm can converge in the deterministic setting.

Original: Hence it is reasonable to conjecture Stochastic AdaGradNorm can also converge if replacing Assumption 2’ by Assumption 2.
Revised: Hence it is reasonable to conjecture Stochastic AdaGradNorm can also converge if replacing Assumption 2’ by Assumption 2.

Original: In this section, we show that, indeed, this conjecture is true.
Revised: In this section, we show that, indeed, this conjecture is true.

Original: Theorem A.9.
Revised: Theorem A.9.

Original: Suppose F satisﬁes Assumptions 1, 2, 3 and 4.
Revised: Suppose F satisﬁes Assumptions 1, 2, 3 and 4.

Original: Stochastic AdaGradNorm admits
Revised: Stochastic AdaGradNorm admits

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First we invoke Lemma A
Revised: First we invoke Lemma A

Original: [Equation]
Revised: [Equation]

Original: 2 to get
Revised: 2 to get

Original: [Equation]
Revised: [Equation]

Original: Using Holder’s inequality we have
Revised: Using Holder’s inequality we have

Original: [Equation]
Revised: [Equation]

Original: Applying Lemma A.5 with p = 1 to get
Revised: Applying Lemma A.5 with p = 1 to get

Original: [Equation]
Revised: [Equation]

Original: log [Equation] where C = 2 2 θ log 2 θ (cid:0) Γ(4 θ + 1) e 2 (cid:1)
Revised: where C = 2 2 θ log 2 θ (cid:0) Γ(4 θ + 1) e 2 (cid:1)

Original: + 1 .
Revised: + 1 .

Original: Besides, note that
Revised: Besides, note that

Original: [Equation]
Revised: [Equation]

Original: Thus we know
Revised: Thus we know

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to Lemma A.3.
Revised: where the last inequality is due to Lemma A.3.

Original: Hence, by letting
Revised: Hence, by letting

Original: [Equation]
Revised: [Equation]

Original: we can solve the following inequality
Revised: we can solve the following inequality

Original: [Equation]
Revised: [Equation]

Original: to get the ﬁnal result
Revised: to get the ﬁnal result.

Original: [Equation]
Revised: A.

Original: [Figure]
Revised: A DA G RAD

Original: Initialize: x 1 , η >for t = 1 to T for j = 1 to d
Revised: Initialize: x 1 , η >for t = 1 to T for j = 1 to d

Original: [Equation]
Revised: [Equation]

Original: In this section, we will extend the result of AdaGradNorm to AdaGrad (Algorithm 7) in the deterministic setting.
Revised: In this section, we will extend the result of AdaGradNorm to AdaGrad (Algorithm 7) in the deterministic setting.

Original: To our knowledge, we are the ﬁrst to give the explicit bound of the counvergence rate of AdaGrad on R d .
Revised: To our knowledge, we are the ﬁrst to give the explicit bound of the counvergence rate of AdaGrad on R d .

Original: First, we examine the growth of the stepsize.
Revised: First, we examine the growth of the stepsize.

Original: Lemma A.10.
Revised: Lemma A.10.

Original: Suppose F satisﬁes Assumptions 1 and 2”, we have
Revised: Suppose F satisﬁes Assumptions 1 and 2”, we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By smoothness we have
Revised: By smoothness we have

Original: [Equation]
Revised: [Equation]

Original: ⇒
Revised: ⇒

Original: T (cid:88) t = d (cid:88)
Revised: T (cid:88) t = d (cid:88)

Original: j = η
Revised: j = η

Original: [Equation]
Revised: [Equation]

Original: +

Original: T (cid:88) t = d (cid:88)
Revised: T (cid:88) t = d (cid:88)

Original: j = (cid:32)
Revised: j = (cid:32)

Original: L j η2 b 2 t,j − η 2 b t,j (cid:33) ∇ j F ( x t ) 2 .
Revised: L j η2 b 2 t,j − η 2 b t,j (cid:33) ∇ j F ( x t ) 2 .

Original: Note that, for the L.H.S.,
Revised: Note that, for the L.H.S.,

Original: T (cid:88) t = d (cid:88)
Revised: T (cid:88) t = d (cid:88)

Original: j = η
Revised: j = η

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: j =
Revised: j =

Original: T (cid:88) t =
Revised: T (cid:88) t =

Original: [Equation]
Revised: [Equation]

Original: −
Revised: 2 t,

Original: [Equation]
Revised: [Equation]

Original:  b [Equation] j ≥ η d (cid:88) j =
Revised: 2 t − 1 ,j b t,j ≥ η d (cid:88) j =

Original: T (cid:88) t = b t [Equation] j
Revised: T (cid:88) t = b t,j

Original: [Equation]
Revised: [Equation]

Original: b t [Equation] 1 ,j = η d (cid:88)
Revised: b t − 1 ,j = η d (cid:88)

Original: j = ( b T,j − b 0 ,j ) .
Revised: j = ( b T,j − b 0 ,j ) .

Original: Besides,
Revised: Besides,

Original: T (cid:88) t = d (cid:88)
Revised: T (cid:88) t = d (cid:88)

Original: j = (cid:32)
Revised: j = (cid:32)

Original: L j η

Original: [Equation]
Revised: [Equation]

Original: (cid:33)

Original: [Equation]
Revised: [Equation]

Original: T (cid:88) t = d (cid:88)
Revised: T (cid:88) t = d (cid:88)

Original: j = (cid:32)
Revised: j = (cid:32)

Original: L j η b 2 t,j − b t,j (cid:33) ∇ j F ( x t )≤ η d (cid:88)
Revised: L j η b 2 t,j − b t,j (cid:33) ∇ j F ( x t )≤ η d (cid:88)

Original: j = τ j (cid:88) t =
Revised: j = τ j (cid:88) t =

Original: L j η b 2 t,j ∇ i F ( x t )( τ j is the last t such that b t,j ≤ ηL j ) =
Revised: L j η b 2 t,j ∇ i F ( x t )( τ j is the last t such that b t,j ≤ ηL j ) =

Original: η d (cid:88) j
Revised: η d (cid:88) j

Original: =
Revised: =

Original: L j τ i (cid:88) t = b 2 t,j − b 2 t − 1 ,j b 2 t,j 
Revised: L j τ i (cid:88) t = b 2 t,j − b 2 t − 1 ,j b 2 t,j ≤

Original: [Equation]
Revised: [Equation]

Original: Hence we have
Revised: Hence we have

Original: [Equation]
Revised: [Equation]

Original: Theorem A.11 states the convergence guarantee for Algorithm 7.
Revised: Theorem A.11 states the convergence guarantee for Algorithm 7.

Original: Theorem A.11.
Revised: Theorem A.11.

Original: Suppose F satisﬁes Assumptions 1 and 2”, we have
Revised: Suppose F satisﬁes Assumptions 1 and 2”, we have

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Before going into the proof
Revised: Before going into the proof

Original: [Equation]
Revised: [Equation]

Original: it is worth discussing the result above as well as the main challenges and differences compared with AdaGradNorm.
Revised: it is worth discussing the result above as well as the main challenges and differences compared with AdaGradNorm.

Original: For simplicity, let b t =
Revised: For simplicity, let b t =

Original: diag ( b t,i ) .
Revised: diag ( b t,i ) .

Original: We can expect that, by a similar argument that we used before to bound the function value gap via the stepsize, we will have
Revised: We can expect that, by a similar argument that we used before to bound the function value gap via the stepsize, we will have

Original: [Equation]
Revised: [Equation]

Original: T where g ( b T ) is a function of the last stepsize and the factor (cid:107) x 
Revised: T where g ( b T ) is a function of the last stepsize and the factor (cid:107) x 1 − x

Original: [Equation]
Revised: [Equation]

Original: is obtained in a similar manner as before, but in d -dimensions.
Revised: is obtained in a similar manner as before, but in d -dimensions.

Original: The challenge is that since the stepsize is a vector, it is not possible to use “division” by the stepsize as in AdaGradNorm.
Revised: The challenge is that since the stepsize is a vector, it is not possible to use “division” by the stepsize as in AdaGradNorm.

Original: On the one hand, we can overcome this by rewriting the argument; on the other hand, this problem will incur an exponential rate for g ( b T ) dependent on the smoothness parameters.
Revised: On the one hand, we can overcome this by rewriting the argument; on the other hand, this problem will incur an exponential rate for g ( b T ) dependent on the smoothness parameters.

Original: Proof.
Revised: Proof.

Original: We can write
Revised: We can write

Original: [Equation]
Revised: [Equation]

Original: Starting from γ -quasar convexity
Revised: Starting from γ -quasar convexity

Original: [Equation]
Revised: [Equation]

Original: Note that F also satisﬁes Assumption 2 (cid:48)(cid:48)
Revised: Note that F also satisﬁes Assumption 2 (cid:48)(cid:48)

Original: [Equation]
Revised: [Equation]

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: Taking the sum over t
Revised: Taking the sum over t

Original: T (cid:88)
Revised: T (cid:88)

Revised: [Equation]

Revised: −

Revised: [Equation]

Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: Due to the excess term (cid:80) Tt =2 (cid:107) x
Revised: b T γη + [Equation] = (cid:107) [Equation]

Revised: [Equation]

Revised: +

Revised: [Equation]

Revised: (cid:88)

Revised: [Equation]

Revised: (cid:18)

Revised: [Equation]

Revised: −

Revised: [Equation]

Revised: (cid:19) ∇

Revised: [Equation]

Original:  t − x ∗ (cid:107) 2 b t − b t − 1 γη in the RHS, we need to proceed and bound (cid:107) x
Revised: F ( x [Equation] Due to the excess term (cid:80) Tt =2 (cid:107) x t − x ∗ (cid:107) 2 b t − b t − 1 γη in the RHS, we need to proceed and bound (cid:107) x

Original: t − x ∗ (cid:107) 2 b t − b t − 1 .
Revised: t − x ∗ (cid:107) 2 b t − b t − 1 .

Original: First, observe that since the L.H.S. is non-negative, 
Revised: First, observe that since the L.H.S. is non-negative, (cid:107) x

Original: [Equation]
Revised: [Equation]

Revised: +1 − x

Revised: [Equation]

Revised: b T

Revised: [Equation]

Revised: [Equation]

Original: To upperbound (cid:107) x
Revised: To upperbound (cid:107) x

Original: t − x ∗ (cid:107) 2 b t − b t − 1 , a key observation is that 
Revised: t − x ∗ (cid:107) 2 b t − b t − 1 , a key observation is that (cid:107) [Equation] x [Equation]

Revised: 1 −

Original: [Equation]
Revised: [Equation]

Revised: ×

Revised: [Equation]

Revised: b T ≥

Revised: [Equation]

Revised: x

Revised: [Equation]

Revised: − x

Revised: [Equation]

Original:  k b T,k b T +1 ,k − b T,k
Revised: b [Equation] [Equation] b T min k b T,k b T +1 ,k − b T,k

Original: Hence for T ≥(cid:107)
Revised: Hence for T ≥(cid:107)

Original: x T +1 − x ∗ (cid:107) 2 b
Revised: x T +1 − x ∗ (cid:107) 2 b

Original: T +1 − b T γη ≤ max k (cid:18) b T +1 ,k b T,k −(cid:19)   (cid:107) x
Revised: T +1 − b T γη ≤ max k (cid:18) b T +1 ,k b T,k −(cid:19)   (cid:107) x

Original: 1 − x ∗ (cid:107) 2 bγη +
Revised: 1 − x ∗ (cid:107) 2 bγη +

Original: T (cid:88) t
Revised: T (cid:88) t

Original: = (cid:107) x
Revised: = (cid:107) x

Original: t − x ∗ (cid:107) 2 b t − b t −γη +
Revised: t − x ∗ (cid:107) 2 b t − b t −γη +

Original: T (cid:88) t = d (cid:88)
Revised: T (cid:88) t = d (cid:88)

Original: j = (cid:18) η γb t,j − 2 L j (cid:19) ∇ j F ( x t )
Revised: j = (cid:18) η γb t,j − 2 L j (cid:19) ∇ j F ( x t )

Original: 
Revised: 

Original: By using this bound for the last term (cid:107) x
Revised: By using this bound for the last term (cid:107) x

Original: T − x ∗ (cid:107) 2 b T − b T − 1 γη we obtain
Revised: T − x ∗ (cid:107) 2 b T − b T − 1 γη we obtain

Original: T (cid:88) t =
Revised: T (cid:88) t =

Original: F ( x t ) − F ∗ ≤(cid:107)
Revised: F ( x t ) − F ∗ ≤(cid:107)

Original: x 1 − x ∗ (cid:107) 

Revised: x 1 − x ∗ (cid:107)

Original: b 1 − (cid:107) x T
Revised: b 1 − (cid:107) x T

Original: +1 − x ∗ (cid:107) 2 b T γη +
Revised: +1 − x ∗ (cid:107) 2 b T γη +

Original: T (cid:88) t
Revised: T (cid:88) t

Original: = (cid:107) x
Revised: = (cid:107) x

Original: t − x ∗ (cid:107) 2 b t − b t −γη +
Revised: t − x ∗ (cid:107) 2 b t − b t −γη +

Original: T (cid:88) t = d (cid:88)
Revised: T (cid:88) t = d (cid:88)

Original: j = (cid:18) η γb t,j − 2 L j (cid:19) ∇ j F ( x t )≤(cid:107)
Revised: j = (cid:18) η γb t,j − 2 L j (cid:19) ∇ j F ( x t )≤(cid:107)

Original: x 1 − x ∗ (cid:107) 2 bγη +
Revised: x 1 − x ∗ (cid:107) 2 bγη +

Original: T − 1 (cid:88) t
Revised: T − 1 (cid:88) t

Original: = (cid:107) x
Revised: = (cid:107) x

Original: t − x ∗ (cid:107) 2 b t − b t −γη +
Revised: t − x ∗ (cid:107) 2 b t − b t −γη +

Original: T (cid:88) t = d (cid:88)
Revised: T (cid:88) t = d (cid:88)

Original: j = (cid:18) η γb t,j − 2 L j (cid:19) ∇ j F ( x t )
Revised: j = (cid:18) η γb t,j − 2 L j (cid:19) ∇ j F ( x t )

Original: + max k (cid:18) b T,k b T − 1 ,k −(cid:19)
Revised: + max k (cid:18) b T,k b T − 1 ,k −(cid:19)

Original:   (cid:107) x 1 − x ∗ (cid:107) 2 bγη
Revised:   (cid:107) x 1 − x ∗ (cid:107) 2 bγη

Original: +
Revised: +

Original: T − 1 (cid:88) t
Revised: T − 1 (cid:88) t

Original: = (cid:107) x
Revised: = (cid:107) x

Original: t − x ∗ (cid:107) 2 b t − b t −γη +
Revised: t − x ∗ (cid:107) 2 b t − b t −γη +

Original: T − 1 (cid:88) t = d (cid:88)
Revised: T − 1 (cid:88) t = d (cid:88)

Original: j = (cid:18) η γb t,j − 2 L j (cid:19) ∇ j F ( x t )
Revised: j = (cid:18) η γb t,j − 2 L j (cid:19) ∇ j F ( x t )

Original:  = max k b T,k b T − 1 ,k   (cid:107) x
Revised:  = max k b T,k b T − 1 ,k   (cid:107) x

Original: 1 − x ∗ (cid:107) 2 bγη +
Revised: 1 − x ∗ (cid:107) 2 bγη +

Original: T − 1 (cid:88) t
Revised: T − 1 (cid:88) t

Original: = (cid:107) x
Revised: = (cid:107) x

Original: t − x ∗ (cid:107) 2 b t − b t −γη +
Revised: t − x ∗ (cid:107) 2 b t − b t −γη +

Original: T − 1 (cid:88) t = d (cid:88)
Revised: T − 1 (cid:88) t = d (cid:88)

Original: j = (cid:18) η γb t,j − 2 L j (cid:19) ∇ j F ( x t )
Revised: j = (cid:18) η γb t,j − 2 L j (cid:19) ∇ j F ( x t )

Original:  + d (cid:88)
Revised:  + d (cid:88)

Original: j = (cid:18) η γb T,j − 2 L j (cid:19) ∇ j F ( x T )
Revised: j = (cid:18) η γb T,j − 2 L j (cid:19) ∇ j F ( x T )

Original: Continue to unroll this relation and for convenience let (cid:81)
Revised: Continue to unroll this relation and for convenience let (cid:81)

Original: Tt =
Revised: Tt =

Original: T +1 max k b t,k b t − 1 ,k
Revised: T +1 max k b t,k b t − 1 ,k

Original: = 1 , we have
Revised: = 1 , we have

Original: T (cid:88) t =
Revised: T (cid:88) t =

Original: F ( x t ) − F ∗ ≤ (cid:34) T (cid:89) t = max k b t,k b t − 1 ,k (cid:35) (cid:107) x
Revised: F ( x t ) − F ∗ ≤ (cid:34) T (cid:89) t = max k b t,k b t − 1 ,k (cid:35) (cid:107) x

Original: 1 − x ∗ (cid:107) 2 bγη +
Revised: 1 − x ∗ (cid:107) 2 bγη +

Original: T (cid:88) t
Revised: T (cid:88) t

Original: = (cid:32) T (cid:89) (cid:96) =
Revised: = (cid:32) T (cid:89) (cid:96) =

Original: t + max k b (cid:96),k b (cid:96) − 1 ,k (cid:33)   d (cid:88)
Revised: t + max k b (cid:96),k b (cid:96) − 1 ,k (cid:33)   d (cid:88)

Original: j = (cid:18) η γb t,j − 2 L j (cid:19) ∇ j F ( x t )
Revised: j = (cid:18) η γb t,j − 2 L j (cid:19) ∇ j F ( x t )

Original:  = (cid:34) T (cid:89) t = max k b t,k b t − 1 ,k (cid:35) (cid:107) x 1 − x ∗ (cid:107) 2 bγη + d (cid:88)
Revised:  = (cid:34) T (cid:89) t = max k b t,k b t − 1 ,k (cid:35) (cid:107) x 1 − x ∗ (cid:107) 2 bγη + d (cid:88)

Original: j =
Revised: j =

Original: T (cid:88) t
Revised: T (cid:88) t

Original: = (cid:32) T (cid:89)
Revised: = (cid:32) T (cid:89)

Original: [Equation]
Revised: [Equation]

Original: max
Revised: max

Original: [Equation]

Original: [Equation]

Original: ∇

Original: [Equation]
Revised: [Equation]

Original: F

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: Given j , if b 1 ,j > 2 ηL j γ , we know
Revised: Given j , if b 1 ,j > 2 ηL j γ , we know

Original: T (cid:88)
Revised: T (cid:88)

Original: [Equation]
Revised: [Equation]

Original: max
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Otherwise, let τ j be the last t such that b t,j ≤ 2 ηL j γ , we also have
Revised: Otherwise, let τ j be the last t such that b t,j ≤ 2 ηL j γ , we also have

Original: T 
Revised: T (cid:88) t = (cid:32) T (cid:89) (cid:96) = t + max k b (cid:96),k b (cid:96) − 1 ,k (cid:33) (cid:18) η γb t,j − 2 L j (cid:19) ∇ j F ( x t )≤ τ j (cid:88) t =

Original: [Table]
Revised: (cid:32) T (cid:89) (cid:96) = t + max k b (cid:96),k b (cid:96) − 1 ,k (cid:33) (cid:18) η γb t,j − 2 L j (cid:19) ∇ j F ( x t )≤ (cid:34) T ( cid:89) t = max k b t,k b t − 1 ,k (cid:35) (cid:32) τ j (cid:88) t = (cid:18) η γb t,j − 2 L j (cid:19) ∇ j F ( x t )(cid:33) ≤ (cid:34) T (cid:89) t = max k b t,k b t − 1 ,k (cid:35) (cid:32)

Original:  b t,k b t − 1 ,k (cid:35) (cid:32) 2 η γ τ j (cid:88) t
Revised: τ j (cid:88) t = η γb t,j ∇ j F ( x t )(cid:33) = (cid:34) T (cid:89) t = max k b t,k b t − 1 ,k (cid:35) (cid:32) η γ τ j (cid:88) t = b 2 t,j − b 2 t − 1 ,j b t,j (cid:33) ≤ (cid:34) T (cid:89) t = max k b t,k b t − 1 ,k (cid:35) (cid:32) 2 η γ τ j (cid:88) t

Original: =
Revised: =

Original: b t,j − b t − 1 ,j (cid:33) ≤ 2 η γ (cid:34) T (cid:89) t = max k b t,k b t − 1 ,k (cid:35) (cid:18) 2 ηL j γ − b 0 ,j (cid:19) + .
Revised: b t,j − b t − 1 ,j (cid:33) ≤ 2 η γ (cid:34) T (cid:89) t = max k b t,k b t − 1 ,k (cid:35) (cid:18) 2 ηL j γ − b 0 ,j (cid:19) + .

Original: Hence we know d (cid:88)
Revised: Hence we know d (cid:88)

Original: j =
Revised: j =

Original: T (cid:88) t
Revised: T (cid:88) t

Original: = (cid:32) T (cid:89) (cid:96) =
Revised: = (cid:32) T (cid:89) (cid:96) =

Original: t + max k b (cid:96),k b (cid:96) − 1 ,k (cid:33) (cid:18) η γb t,j − 2 L j (cid:19) ∇ j F ( x t )≤ 2 η γ (cid:34) T (cid:89) t = max k b t,k b t − 1 ,k (cid:35)
Revised: t + max k b (cid:96),k b (cid:96) − 1 ,k (cid:33) (cid:18) η γb t,j − 2 L j (cid:19) ∇ j F ( x t )≤ 2 η γ (cid:34) T (cid:89) t = max k b t,k b t − 1 ,k (cid:35)

Original:   d (cid:88)
Revised:   d (cid:88)

Original: j = (cid:18) 2 ηL j γ − b 0 ,j (cid:19) +   .
Revised: j = (cid:18) 2 ηL j γ − b 0 ,j (cid:19) +   .

Original: Thus we have
Revised: Thus we have

Original: T (cid:88) t =
Revised: T (cid:88) t =

Original: F ( x t ) − F ∗ ≤ (cid:34) T (cid:89) t = max k b t,k b t − 1 ,k (cid:35) (cid:107) x
Revised: F ( x t ) − F ∗ ≤ (cid:34) T (cid:89) t = max k b t,k b t − 1 ,k (cid:35) (cid:107) x

Original: 1 − x ∗ (cid:107) 2 bγη + d (cid:88)
Revised: 1 − x ∗ (cid:107) 2 bγη + d (cid:88)

Original: j =
Revised: j =

Original: T (cid:88) t
Revised: T (cid:88) t

Original: = (cid:32) T (cid:89) (cid:96) =
Revised: = (cid:32) T (cid:89) (cid:96) =

Original: t + max k b (cid:96),k b (cid:96) − 1 ,k (cid:33) (cid:18) η γb t,j − 2 L j (cid:19) ∇ j F ( x t )≤ (cid:34) T (
Revised: t + max k b (cid:96),k b (cid:96) − 1 ,k (cid:33) (cid:18) η γb t,j − 2 L j (cid:19) ∇ j F ( x t )≤ (cid:34) T (

Original: cid:89) t = max k b t,k b t − 1 ,k (cid:35)   (cid:107) x 1 − x ∗ (cid:107) 2 bγη + 2 η γ d (cid:88)
Revised: cid:89) t = max k b t,k b t − 1 ,k (cid:35)   (cid:107) x 1 − x ∗ (cid:107) 2 bγη + 2 η γ d (cid:88)

Original: j = (cid:18) 2 ηL j γ − b 0 ,j (cid:19) + 
Revised: j = (cid:18) 2 ηL j γ − b 0 ,j (cid:19) + 

Original: 
Revised: 

Original: From Lemma A. d (cid:88)
Revised: From Lemma A. d (cid:88)

Original: j = b T,j ≤ d (cid:88)
Revised: j = b T,j ≤ d (cid:88)

Original: j = b 0 ,j + 2( F ( x 1 ) − F ∗ )
Revised: j = b 0 ,j + 2( F ( x 1 ) − F ∗ )

Original: η
Revised: η

Original: + 2 η d (cid:88) j
Revised: + 2 η d (cid:88) j

Original: =
Revised: =

Original: L j log + ηL j b 0 ,j
Revised: L j log + ηL j b 0 ,j

Original: Using AM-GM we have d (cid:89) j = b T,j ≤ (cid:32)(cid:80) dj
Revised: Using AM-GM we have d (cid:89) j = b T,j ≤ (cid:32)(cid:80) dj

Original: =1 b T,j d (cid:33) d ≤d d   d (cid:88)
Revised: =1 b T,j d (cid:33) d ≤d d   d (cid:88)

Original: j = b 0 ,j + 2( F ( x 1 ) − F ∗ )
Revised: j = b 0 ,j + 2( F ( x 1 ) − F ∗ )

Original: η
Revised: η

Original: + 2 η d (cid:88) j
Revised: + 2 η d (cid:88) j

Original: =
Revised: =

Original: L j log + ηL j b 0 ,j  
Revised: L j log + ηL j b 0 ,j  

Original: d
Revised: d

Original: Note that
Revised: Note that

Original: T (cid:89) t = max j b t,j b t − 1 ,j ≤

Original: T (cid:89) t = d (cid:89)
Revised: T (cid:89) [Equation] = [Equation]

Original: j = b t,j b t − 1 ,j ≤ d (cid:89) j = b T,j b 0 ,j ≤ d d (cid:81) dj
Revised: max [Equation] (cid:89) [Equation]

Original: =1 b 0 ,j   d (cid:88)
Revised: = [Equation] (cid:89)

Original: j = b 0 ,j + 2( F ( x 1 ) − F ∗ )
Revised: j = [Equation] (cid:89) [Equation]

Original: η
Revised: = [Equation] [Equation]  

Original: + 2 η d (cid:88)j
Revised: [Equation] (cid:88)

Original:  = L j log + ηL j b 0 ,j  
Revised: [Equation] = [Equation]

Original: d
Revised: [Equation] ) η + [Equation] (cid:88)

Original: Hence
Revised: j = L

Original: T (cid:88) t =
Revised: [Equation]

Original: F ( x t ) − F ∗ ≤ (cid:34) T (cid:89) t = max j [Equation] (cid:35)
Revised: log + ηL

Original: [Equation]
Revised: [Equation]

Original: γ d (cid:88)

Original: [Equation] (cid:18) [Equation] ηL j γ
Revised: 0 ,j  

Original: [Equation] 0 ,j (cid:19) [Equation]  ≤
Revised: d

Original: [Equation] [Equation] d (cid:18) (cid:107) x [Equation]
Revised: Hence

Original: x [Equation] (cid:107)
Revised: T (cid:88) t =

Original: [Equation]
Revised: [Equation]

Original: γη

Original: [Equation]
Revised: [Equation]

Original: 2 ηγ (cid:80) [Equation] =
Revised: ≤ (cid:34) T (cid:89) t =

Original: [Equation]
Revised: [Equation]

Original: (cid:16) 2 ηL j γ − [Equation] 0 ,j (cid:17)
Revised: max

Original: [Equation]
Revised: [Equation]

Original: (cid:19) d [Equation] =
Revised: γ

Original: [Equation]
Revised: [Equation]

Original: 0 ,j which ﬁnishes the proof.
Revised: which ﬁnishes the proof.

Original: B M ISSING PROOFS FROM S ECTIONB. I MPORTANT LEMMA
Revised: B M ISSING PROOFS FROM S ECTIONB. I MPORTANT LEMMA

Original: First, we state a general lemma that can be used for a more general setting.
Revised: First, we state a general lemma that can be used for a more general setting.

Original: The proof of the lemma is standard.
Revised: The proof of the lemma is standard.

Original: Lemma B.1.
Revised: Lemma B.1.

Original: Suppose F satisﬁes Assumptions 1 and 2’ and the following conditions hold:
Revised: Suppose F satisﬁes Assumptions 1 and 2’ and the following conditions hold:

Original: • x t is generated by x t +1 = x
Revised: • x t is generated by x t +1 = x

Original: t − ηc t ∇ F ( x t ) , with η > 0 and c t > 0 is non-decreasing; • p t ∈ (0 , 1] satisﬁes 1 p t ≥ 1 − p t +1 p t
Revised: t − ηc t ∇ F ( x t ) , with η > 0 and c t > 0 is non-decreasing; • p t ∈ (0 , 1] satisﬁes 1 p t ≥ 1 − p t +1 p t

Original: +1 , p 1 = 1 ;
Revised: +1 , p 1 = 1 ;

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Starting from L -smoothness
Revised: Starting from L -smoothness

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: Note that Assumption 2 can be implied by Assumption 2’, hence we have
Revised: Note that Assumption 2 can be implied by Assumption 2’, hence we have

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original:  p 
Revised: γη − p [Equation] +

Original: [Equation]
Revised: [Equation]

Original: We obtain
Revised: We obtain

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: x 
Revised: x t

Original: [Equation]
Revised: [Equation]

Original: Note that we require 1 p t ≥ 1 − p t +1 p t +1 and c t is increasing hence
Revised: Note that we require 1 p t ≥ 1 − p t +1 p t +1 and c t is increasing hence

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: which leads to
Revised: which leads to

Original: [Equation]
Revised: [Equation]

Original: c

Original: [Equation]

Original: By setting p 1 = 1 we get the desired result.
Revised: By setting p 1 = 1 we get the desired result.

Original: B. F IRST VARIANT
Revised: B. F IRST VARIANT

Original: Note that if we assume p t satisﬁes the condition in Lemma B.1, by replacing c t by b t , we have
Revised: Note that if we assume p t satisﬁes the condition in Lemma B.1, by replacing c t by b t , we have

Original: [Equation]
Revised: [Equation]

Original: immediately.
Revised: immediately.

Original: Now our two left tasks are to bound the residual term
Revised: Now our two left tasks are to bound the residual term

Original: [Equation]
Revised: [Equation]

Original: and ﬁnd an upper bound on b T .
Revised: and ﬁnd an upper bound on b T .

Original: Lemmas B. and B.3 demonstrate how we achieve these two goals.
Revised: Lemmas B. and B.3 demonstrate how we achieve these two goals.

Original: Lemma B.2.
Revised: Lemma B.2.

Original: If p t ≤ 1 for every t , we have
Revised: If p t ≤ 1 for every t , we have

Original: [Equation]
Revised: [Equation]

Original: where h (∆) :
Revised: where h (∆) :

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: ∆ ∈ (0 , 1) and g (∆) :
Revised: ∈ (0 , 1) and g (∆) :

Original: = (2 + ∆)
Revised: = (2 + ∆)

Original: η γ (cid:18) 2 ηL γ (cid:19) ∆ log + 2 ηL γb 0 .
Revised: η γ (cid:18) 2 ηL γ (cid:19) ∆ log + 2 ηL γb 0 .

Original: Proof.
Revised: Proof.

Original: We ﬁrst bound
Revised: We ﬁrst bound

Original: T (cid:88) t = (cid:18)
Revised: T (cid:88) t = (cid:18)

Original: L 2 b t − 2 η (cid:19)
Revised: L 2 b t − 2 η (cid:19)

Original: η 2 (cid:107)∇ F ( x t ) (cid:107)b 2 t p t .
Revised: η 2 (cid:107)∇ F ( x t ) (cid:107)b 2 t p t .

Original: If b 1 > ηL , we know
Revised: If b 1 > ηL , we know

Original: T (cid:88) t = (cid:18)
Revised: T (cid:88) t = (cid:18)

Original: L 2 b t − 2 η (cid:19)
Revised: L 2 b t − 2 η (cid:19)

Original: η 2 (cid:107)∇ F ( x t ) (cid:107)b 2 t p t < 0 ≤ h (∆) .
Revised: η 2 (cid:107)∇ F ( x t ) (cid:107)b 2 t p t < 0 ≤ h (∆) .

Original: Otherwise, we deﬁne the time τ = max { t ∈ [ T ] , b t ≤ ηL } .
Revised: Otherwise, we deﬁne the time τ = max { t ∈ [ T ] , b t ≤ ηL } .

Original: Hence, we have
Revised: Hence, we have

Original: T (cid:88) t = (cid:18)
Revised: T (cid:88) t = (cid:18)

Original: L 2 b t − 2 η (cid:19)
Revised: L 2 b t − 2 η (cid:19)

Original: η 2 (cid:107)∇ F ( x t ) (cid:107)b 2 t p t = τ (cid:88) t =
Revised: η 2 (cid:107)∇ F ( x t ) (cid:107)b 2 t p t = τ (cid:88) t =

Original: (cid:18)
Revised: (cid:18)

Original: L 2 b t − 2 η (cid:19)
Revised: L 2 b t − 2 η (cid:19)

Original: η 2 (cid:107)∇ F ( x t ) (cid:107)b 2 t p t +
Revised: η 2 (cid:107)∇ F ( x t ) (cid:107)b 2 t p t +

Original: T (cid:88) t = τ (cid:18)
Revised: T (cid:88) t = τ (cid:18)

Original: L 2 b t − 2 η (cid:19)
Revised: L 2 b t − 2 η (cid:19)

Original: η 2 (cid:107)∇ F ( x t ) (cid:107)b 2 t p t ≤ τ (cid:88) t = (cid:18)
Revised: η 2 (cid:107)∇ F ( x t ) (cid:107)b 2 t p t ≤ τ (cid:88) t = (cid:18)

Original: L 2 b t − 2 η (cid:19)
Revised: L 2 b t − 2 η (cid:19)

Original: η 2 (cid:107)∇ F ( x t ) (cid:107)b 2 t p t ≤ τ (cid:88) t =
Revised: η 2 (cid:107)∇ F ( x t ) (cid:107)b 2 t p t ≤ τ (cid:88) t =

Original: L 2 b t × η 2 (cid:107)∇ F ( x t ) (cid:107)b 2 t p t =
Revised: L 2 b t × η 2 (cid:107)∇ F ( x t ) (cid:107)b 2 t p t =

Original: η 2 L τ (cid:88) t =
Revised: η 2 L τ (cid:88) t =

Original: b 2+∆ t − b 2+∆ t −b 3 t =
Revised: b 2+∆ t − b 2+∆ t −b 3 t =

Original: η
Revised: η

Original: 2 L τ (cid:88) t =
Revised: 2 L τ (cid:88) t =

Original: b 2+∆ t − b 2+∆ t −b 2+∆ t × b ∆ − 1 t ≤  
Revised: b 2+∆ t − b 2+∆ t −b 2+∆ t × b ∆ − 1 t ≤  

Original: η 2 L 2 (cid:80) τt =1 b 2+∆ t − b 2+∆ t −b 2+∆ t × ( ηL ) ∆ −∆ ≥η 2 L 2 (cid:80) τt
Revised: η 2 L 2 (cid:80) τt =1 b 2+∆ t − b 2+∆ t −b 2+∆ t × ( ηL ) ∆ −∆ ≥η 2 L 2 (cid:80) τt

Original: =1 b 2+∆ t − b 2+∆ t −b 2+∆ t × 1 b 1 − ∆∆ <=
Revised: =1 b 2+∆ t − b 2+∆ t −b 2+∆ t × 1 b 1 − ∆∆ <=

Original:   η ( ηL ) ∆(cid:80)
Revised:   η ( ηL ) ∆(cid:80)

Original: τt =1 b 2+∆ t − b 2+∆ t −b 2+∆ t ∆ ≥η 2 L 2 b 1 − ∆(cid:80)
Revised: τt =1 b 2+∆ t − b 2+∆ t −b 2+∆ t ∆ ≥η 2 L 2 b 1 − ∆(cid:80)

Original: τt =1 b 2+∆ t − b 2+∆ t −b 2+∆ t ∆ <≤   (2+∆) η ( ηL ) ∆log ηLb∆ ≥(2+∆) η 2 L 2 b 1 − ∆log ηLb∆ <≤ h (∆) .
Revised: τt =1 b 2+∆ t − b 2+∆ t −b 2+∆ t ∆ <≤   (2+∆) η ( ηL ) ∆log ηLb∆ ≥(2+∆) η 2 L 2 b 1 − ∆log ηLb∆ <≤ h (∆) .

Original: By applying a similar argument, we can prove
Revised: By applying a similar argument, we can prove

Original: [Equation]
Revised: [Equation]

Original: Lemma B.3.
Revised: Lemma B.3.

Original: Suppose all the conditions in Lemma B.1 are satisﬁed by replacing c t by b t , additionally, assume p t ≤ 1 , we will have
Revised: Suppose all the conditions in Lemma B.1 are satisﬁed by replacing c t by b t , additionally, assume p t ≤ 1 , we will have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Using Lemma B.1 by replacing c t by b t , we know
Revised: Using Lemma B.1 by replacing c t by b t , we know

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is by Lemma B.2.
Revised: where the last inequality is by Lemma B.2.

Original: Noticing F ( x T +1 )
Revised: Noticing F ( x T +1 )

Original: − F ∗ ≥ 0 , we know
Revised: − F ∗ ≥ 0 , we know

Original: [Equation]
Revised: [Equation]

Original: Now we use the update rule of b t to get 
Revised: Now we use the update rule of b t to get T

Original: [Equation]
Revised: [Equation]

Original: Hence we know b
Revised: Hence we know b

Original: [Equation]
Revised: [Equation]

Original: Equipped with Lemmas B.2 and B.3, we can give a proof of Theorem 4.1.
Revised: Equipped with Lemmas B.2 and B.3, we can give a proof of Theorem 4.1.

Original: Proof.
Revised: Proof.

Original: Note that if p t = 1 t , all the conditions in Lemma B.1 are satisﬁed by replacing c t by b t .
Revised: Note that if p t = 1 t , all the conditions in Lemma B.1 are satisﬁed by replacing c t by b t .

Original: Hence we have
Revised: Hence we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is by Lemma B.2.
Revised: where the last inequality is by Lemma B.2.

Original: Multiplying both sides by p T b T , we get
Revised: Multiplying both sides by p T b T , we get

Original: [Equation]
Revised: [Equation]

Original: By using the upper bound of b T in Lemma B.3, we ﬁnish the proof.
Revised: By using the upper bound of b T in Lemma B.3, we ﬁnish the proof.

Original: B. S ECOND VARIANT
Revised: B. S ECOND VARIANT

Original: Similar to the previous section, what we need to do is to bound the residual term
Revised: Similar to the previous section, what we need to do is to bound the residual term

Original: [Equation]
Revised: [Equation]

Original: and ﬁnd an upper bound on c T where c t =
Revised: and ﬁnd an upper bound on c T where c t =

Original: b δt b 1 − δ t −here.
Revised: b δt b 1 − δ t −here.

Original: We ﬁrst bound the residual term by the following lemma.
Revised: We ﬁrst bound the residual term by the following lemma.

Original: Lemma B.4.
Revised: Lemma B.4.

Original: If p t ≤ 1 for every t , we have
Revised: If p t ≤ 1 for every t , we have

Original: [Equation]
Revised: [Equation]

Original: where c t =
Revised: where c t =

Original: b δt b 1 − δ t − 1 .
Revised: b δt b 1 − δ t − 1 .

Original: Proof.
Revised: Proof.

Original: Note that c t c t − 1 =
Revised: Note that c t c t − 1 =

Original: b δt b
Revised: b δt b

Original: [Equation]
Revised: [Equation]

Original: δ [Equation] − 1 b 1 − δ t − 2 ≥ 1 , this means c t is monotone increasing.
Revised: b 1 − δ t − 2 ≥ 1 , this means c t is monotone increasing.

Original: We ﬁrst bound
Revised: We ﬁrst bound

Original: [Equation]
Revised: [Equation]

Original: If c 1 > ηL , we know 
Revised: If c 1 > ηL , we know T

Original: [Equation]
Revised: [Equation]

Revised: t =

Revised: [Equation]

Original: (cid:18)
Revised: (cid:18)

Original: L 2 c t − 2 η
Revised: L 2 c t − 2 η

Original: (cid:19)
Revised: (cid:19)

Original: η 2 (cid:107)∇ F ( x t ) (cid:107)c 2 t p t < 0 ≤ η 2 L b(cid:32) 1 − (cid:18) bηL (cid:19) 1 δ (cid:33) + .
Revised: η 2 (cid:107)∇ F ( x t ) (cid:107)c 2 t p t < 0 ≤ η 2 L b(cid:32) 1 − (cid:18) bηL (cid:19) 1 δ (cid:33) + .

Original: Otherwise, let τ = max { t ∈
Revised: Otherwise, let τ = max { t ∈

Original: [ T ] , c t ≤ ηL } .
Revised: [ T ] , c t ≤ ηL } .

Original: We have
Revised: We have

Original: T (cid:88) t = (cid:18)
Revised: T (cid:88) t = (cid:18)

Original: L 2 c t − 2 η (cid:19)
Revised: L 2 c t − 2 η (cid:19)

Original: η 2 (cid:107)∇ F ( x t ) (cid:107)c 2 t p t ≤ τ (cid:88) t = η 2 L (cid:107)∇ F ( x t ) (cid:107)2 c 3 t p t =
Revised: η 2 (cid:107)∇ F ( x t ) (cid:107)c 2 t p t ≤ τ (cid:88) t = η 2 L (cid:107)∇ F ( x t ) (cid:107)2 c 3 t p t =

Original: η 2 L τ (cid:88) t =
Revised: η 2 L τ (cid:88) t =

Original: b 2 t − b 2 t −b 3 δt b 3 − 3 δ t −≤ η 2 L τ (cid:88) t = b t − b t −b 3 δ − 1 t b 3 − 3 δ t −≤ η 2 L τ (cid:88) t =
Revised: b 2 t − b 2 t −b 3 δt b 3 − 3 δ t −≤ η 2 L τ (cid:88) t = b t − b t −b 3 δ − 1 t b 3 − 3 δ t −≤ η 2 L τ (cid:88) t =

Original: b t − b t −b t b t −≤ η 2 L (cid:18)b 0 − b τ (cid:19) .
Revised: b t − b t −b t b t −≤ η 2 L (cid:18)b 0 − b τ (cid:19) .

Original: Note that c τ = b δτ b 1 − δ τ − 1 ≤ ηL ⇒ b τ ≤ (cid:18) ηL
Revised: Note that c τ = b δτ b 1 − δ τ − 1 ≤ ηL ⇒ b τ ≤ (cid:18) ηL

Original: b 1 − δ τ −(cid:19) 1 /δ .
Revised: b 1 − δ τ −(cid:19) 1 /δ .

Original: Hence b 0 − b τ ≤ b 0 − ( b τ − 1 ) 1 δ −( ηL ) 1 /δ ≤b(cid:32) 1 − (cid:18) bηL (cid:19) 1 δ (cid:33)
Revised: Hence b 0 − b τ ≤ b 0 − ( b τ − 1 ) 1 δ −( ηL ) 1 /δ ≤b(cid:32) 1 − (cid:18) bηL (cid:19) 1 δ (cid:33)

Original: + .
Revised: + .

Original: Combinging two cases, there is always
Revised: Combinging two cases, there is always

Original: T (cid:88) t = (cid:18) 
Revised: T (cid:88) t = (cid:18) L

Original: [Equation]
Revised: [Equation]

Revised: c

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: η (cid:19)

Original: c
Revised: η

Original: [Equation]
Revised: [Equation]

Revised: (cid:107)∇ F ( x t

Revised: [Equation]

Revised: (cid:107)

Revised: [Equation]

Revised: c

Revised: [Equation]

Revised: t p

Revised: [Equation]

Revised: η

Revised: [Equation]

Revised: b(cid:32) 1 − (cid:18)

Revised: [Equation]

Original:  (cid:19) 1 δ (cid:33) + .
Revised: ηL (cid:19) 1 δ (cid:33) + .

Original: Now we turn to the second bound.
Revised: Now we turn to the second bound.

Original: If c 1 > 2 ηL/γ , we know
Revised: If c 1 > 2 ηL/γ , we know

Original: T (cid:88) 
Revised: T (cid:88) t =

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original:  p
Revised: − p

Original: [Equation]
Revised: [Equation]

Original: Otherwise, we deﬁne the time τ = max { t ∈ [ T ] , c t ≤ 2 ηL/γ } .
Revised: Otherwise, we deﬁne the time τ = max { t ∈ [ T ] , c t ≤ 2 ηL/γ } .

Original: Then, we have
Revised: Then, we have

Original: T (cid:88)t = (cid:18) p [Equation] b
Revised: T (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Because b δt b 1 − δ t − 1 =
Revised: Because b δt b 1 − δ t − 1 =

Original: c t ≤ 2 ηL/γ for t ≤ τ , so we know b t ≤ (cid:18) 2 ηL γb 1 − δ t −(cid:19) 1 /δ .Using this bound η γ τ (cid:88) t = (cid:18) b [Equation]
Revised: c t ≤ 2 ηL/γ for t ≤ τ , so we know b t ≤ (cid:18) 2 ηL γb 1 − δ t −(cid:19) 1 /δ .

Original: b t

Original: [Equation]

Original: (cid:19)

Original: [Equation]

Original: −

Original: [Equation]

Original: −

Original: [Equation] (cid:88) [Equation] =
Revised: Using this bound η γ τ (cid:88) t =

Original: [Equation]

Original: (cid:18)

Original: [Equation]

Original: ηL

Original: [Equation]

Original: −

Original: [Equation]

Original: (cid:19)

Original: [Equation]

Original: γ

Original: [Equation]

Original: ηL

Original: [Equation]

Original: (cid:19)

Original: [Equation]

Original: δ −

Original: [Equation]

Original: (cid:88)

Original: [Equation]

Original: b

Original: [Equation]

Original: b

Original: [Equation]
Revised: [Equation]

Original: b [Equation] η γ (cid:18) 2 ηL γb(cid:19) 2 δ −logb
Revised: (cid:18) 2 ηL γb(cid:19) 2 δ −log

Original:  t b≤ 2 η γδ (cid:18) 2 ηL γb(cid:19) 2 δ −log + 2 ηL γb 0 .
Revised: b t b≤ 2 η γδ (cid:18) 2 ηL γb(cid:19) 2 δ −log + 2 ηL γb 0 .

Original: The proof is completed.
Revised: The proof is completed.

Original: As before, our last task is to bound c T .
Revised: As before, our last task is to bound c T .

Original: It is enough to bound b T since c T ≤ b T .
Revised: It is enough to bound b T since c T ≤ b T .

Original: Lemma B.5.
Revised: Lemma B.5.

Original: Suppose all the conditions in Lemma B.1 are satisﬁed by replacing c t by b t , additionally, assume p t ≤ 1 , we will have
Revised: Suppose all the conditions in Lemma B.1 are satisﬁed by replacing c t by b t , additionally, assume p t ≤ 1 , we will have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By Lemma B.1, we know
Revised: By Lemma B.1, we know

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: (cid:107)
Revised: (cid:107)∇

Original: [Equation]
Revised: [Equation]

Original: 2 c
Revised: ( x t

Original: [Equation]
Revised: [Equation]

Original: t p t T (cid:88) t = η [Equation] F
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: 2 c
Revised: c

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: x
Revised: −

Original: [Equation]
Revised: [Equation]

Original: (cid:107)γη +
Revised: (cid:107)∇

Original: [Equation]
Revised: [Equation]

Revised: ( x t ) (cid:107)2 c 2 t p t

Original:  (cid:88) t
Revised: T (cid:88) t

Original: [Equation]
Revised: = η (cid:107)∇ F ( x t ) (cid:107)2 c 2 t p t ≤ (cid:107) x 1 − x ∗ (cid:107)γη +

Original:  (cid:18)
Revised: T (cid:88) t = (cid:18)

Original: L 2 c t − 2 η + p t γη − p t c t 2 η 2 L (cid:19)
Revised: L 2 c t − 2 η + p t γη − p t c t 2 η 2 L (cid:19)

Original: η 2 (cid:107)∇ F ( x t ) (cid:107)c 2 t p t

Original: [Equation]

Original:  (cid:107) x 1 − x ∗ (cid:107)γη + η 2 L b(cid:32) 1 − (cid:18) bηL (cid:19) 1 δ (cid:33) + + 2 η γδ (cid:18) 2 ηL γb(cid:19) 2 δ −log + 2 ηL γb where the last inequality is by Lemma B.4.
Revised: η 2 (cid:107)∇ F ( x t ) (cid:107)c 2 t p t ≤ (cid:107) x 1 − x ∗ (cid:107)γη + η 2 L b(cid:32) 1 − (cid:18) bηL (cid:19) 1 δ (cid:33) + + 2 η γδ (cid:18) 2 ηL γb(cid:19) 2 δ −log + 2 ηL γb where the last inequality is by Lemma B.4.

Original: Note that for the L.H.S., we have
Revised: Note that for the L.H.S., we have

Original: T 
Revised: T (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Hence we know
Revised: Hence we know

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: Finally, the proof of Theorem 4.3 is similar to the proof of Theorem 4.1, hence, which is omitted.
Revised: Finally, the proof of Theorem 4.3 is similar to the proof of Theorem 4.1, hence, which is omitted.

Original: B. A N ASYMPTOTIC RATE WHEN ∆ = 0 AND δ =
Revised: B. A N ASYMPTOTIC RATE WHEN ∆ = 0 AND δ =

Original: As mentioned before, by setting ∆ = 0 in Algorithm 3 and δ = 1 in Algorithm 4 we obtain the same algorithm.
Revised: As mentioned before, by setting ∆ = 0 in Algorithm 3 and δ = 1 in Algorithm 4 we obtain the same algorithm.

Original: The square root update rule of b t and the step size now are both more similar to the original AdaGradNorm.
Revised: The square root update rule of b t and the step size now are both more similar to the original AdaGradNorm.

Original: Intuitively, we can also expect the convergence of the last iterate in this case; furthermore, by taking the limit when ∆ → 0 and δ → 1 , we can have a sense of the exponential dependency of the provable convergence rate on the problem parameters.
Revised: Intuitively, we can also expect the convergence of the last iterate in this case; furthermore, by taking the limit when ∆ → 0 and δ → 1 , we can have a sense of the exponential dependency of the provable convergence rate on the problem parameters.

Original: However, previous analysis strictly requires that ∆ > 0 and δ < 1 , thus does not apply here.
Revised: However, previous analysis strictly requires that ∆ > 0 and δ < 1 , thus does not apply here.

Original: In this section, we partially conﬁrm the convergence of this variant by proving an asymptotic rate, i.e., F ( x T +1 ) − F ∗ =
Revised: In this section, we partially conﬁrm the convergence of this variant by proving an asymptotic rate, i.e., F ( x T +1 ) − F ∗ =

Original: O (1 /T ) .
Revised: O (1 /T ) .

Original: Unfortunately, under Assumptions 1 and 2’, we cannot ﬁgure out the explicit dependency of the convergence rate on the problem parameters.
Revised: Unfortunately, under Assumptions 1 and 2’, we cannot ﬁgure out the explicit dependency of the convergence rate on the problem parameters.

Original: However, in the next section, we will give an explicit rate by replacing Assumption 1 with the stronger Assumption 1’.
Revised: However, in the next section, we will give an explicit rate by replacing Assumption 1 with the stronger Assumption 1’.

Original: As stated, our goal is to prove Theorem B.6 in this section.
Revised: As stated, our goal is to prove Theorem B.6 in this section.

Original: Theorem B.6.
Revised: Theorem B.6.

Original: Suppose F satisﬁes Assumptions 1 and 2’, when ∆ = 0 for Algorithm 3, or equivalently, δ = 1 for Algorithm 4, by taking p t = 1 t , we have
Revised: Suppose F satisﬁes Assumptions 1 and 2’, when ∆ = 0 for Algorithm 3, or equivalently, δ = 1 for Algorithm 4, by taking p t = 1 t , we have

Original: [Equation]
Revised: [Equation]

Original: Before starting the proof, we ﬁrst discuss why we can obtain only an asymptotic rate when ∆ =and δ = 1 .
Revised: Before starting the proof, we ﬁrst discuss why we can obtain only an asymptotic rate when ∆ =and δ = 1 .

Original: As before, one can still expect that F ( x T +1 ) − F ∗ ≤ b T C T remains true for some constant C .
Revised: As before, one can still expect that F ( x T +1 ) − F ∗ ≤ b T C T remains true for some constant C .

Original: However, a critical difference will show up when we want to ﬁnd an explicit upper bound on b T .
Revised: However, a critical difference will show up when we want to ﬁnd an explicit upper bound on b T .

Original: Using the proof of Lemma B.3 as an example (similarly for the proof of Lemma
Revised: Using the proof of Lemma B.3 as an example (similarly for the proof of Lemma

Original: B.5), one key step is to get (cid:80)
Revised: B.5), one key step is to get (cid:80)

Original: Tt =

Revised: Tt =

Original: (cid:107)∇ F ( x t ) (cid:107) 2 b 2 t p t =
Revised: (cid:107)∇ F ( x t ) (cid:107) 2 b 2 t p t =

Original: O (1) , where in the previous analysis, by replacing (cid:107)∇ F ( x
Revised: O (1) , where in the previous analysis, by replacing (cid:107)∇ F ( x

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: (cid:107) 2 p t by b 2+∆ t − b 2+∆ t − 1 with ∆ > 0 , we can lower bound (cid:80)
Revised: (cid:107) 2 p t by b 2+∆ t − b 2+∆ t − 1 with ∆ > 0 , we can lower bound (cid:80)

Original: Tt [-=
-]
Revised: Tt [-=
-]=

Original: (cid:107)∇ F ( x t ) (cid:107) 2 b 2 t p t by a function of b T and ﬁnally give an explicit bound on b T .
Revised: [Equation] F ( x t ) [Equation] 2 b 2 t p t by a function of b T and ﬁnally give an explicit bound on b T .

Original: However, this is not possible when ∆ = 0 as (cid:80)
Revised: However, this is not possible when ∆ = 0 as (cid:80)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: The only information we can get from (cid:80) Tt =1 b 2 t − b 2 t − 1 b 2 t =
Revised: The only information we can get from (cid:80) Tt =1 b 2 t − b 2 t − 1 b 2 t =

Original: O (1) is lim T →∞ b 2 T − 1 b 2 T = 1 .
Revised: O (1) is lim T →∞ b 2 T − 1 b 2 T = 1 .

Original: This is not enough to tell us whether b T is upper bounded or not.
Revised: This is not enough to tell us whether b T is upper bounded or not.

Original: In Lemma B.8, we will use a new argument to ﬁnally show that lim T →∞ b T < ∞ , which leads to an asymptotic rate as desired.
Revised: In Lemma B.8, we will use a new argument to ﬁnally show that lim T →∞ b T < ∞ , which leads to an asymptotic rate as desired.

Original: It is worth pointing out that ﬁnding an asymptotic without explicit dependency on the problem parameters is the approach used in some of the previous work, such as Antonakopoulos et al.
Revised: It is worth pointing out that ﬁnding an asymptotic without explicit dependency on the problem parameters is the approach used in some of the previous work, such as Antonakopoulos et al.

Original: This also gives us a glimpse of the method used to analyze the convergence of the accelerated methods in Section 5.
Revised: This also gives us a glimpse of the method used to analyze the convergence of the accelerated methods in Section 5.

Original: Now we start the proof.
Revised: Now we start the proof.

Original: As before, we can employ Lemma B.1.
Revised: As before, we can employ Lemma B.1.

Original: Hence we only need to bound the residual terms as following
Revised: Hence we only need to bound the residual terms as following

Original: Lemma B.7.
Revised: Lemma B.7.

Original: Suppose p t ≤ 1 , when ∆ = 0 for Algorithm 3, or equivalently, δ = 1 for Algorithm 4, we have
Revised: Suppose p t ≤ 1 , when ∆ = 0 for Algorithm 3, or equivalently, δ = 1 for Algorithm 4, we have

Original: [Equation]
Revised: [Equation]

Original: The proof is essentially similar to the proof of Lemmas B.2 and B.4, hence we omit it here.
Revised: The proof is essentially similar to the proof of Lemmas B.2 and B.4, hence we omit it here.

Original: Lemma B.8.
Revised: Lemma B.8.

Original: Suppose all the conditions in Lemma B.1 are satisﬁed by replacing c t by b t , then when ∆ = 0 for Algorithm 3, or equivalently, δ = 1 for Algorithm 4, we have
Revised: Suppose all the conditions in Lemma B [Equation] 1 are satisﬁed by replacing c t by b t , then when ∆ = 0 for Algorithm 3, or equivalently, δ = 1 for Algorithm 4, we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First note that b t is increasing, by the Monotone convergence theorem, we know lim T →∞ b T = b ∞ exists.
Revised: First note that b t is increasing, by the Monotone convergence theorem, we know lim T →∞ b T = b ∞ exists.

Original: We aim to show b ∞ < ∞ .
Revised: We aim to show b ∞ < ∞ .

Original: By Lemma B
Revised: By Lemma B

Original: [Equation]
Revised: [Equation]

Original: 1 and replacing c t by b t , we have
Revised: 1 and replacing c t by b t , we have

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: x

Original: [Equation]
Revised: [Equation]

Revised: 2 b

Revised: [Equation]

Revised: p

Revised: [Equation]

Original:  where the last inequality is by Lemma B.7.
Revised: , where the last inequality is by Lemma B.7.

Original: Noticing F ( x T +1 )
Revised: Noticing F ( x T +1 )

Original: − F ∗ ≥ 0 , we know
Revised: − F ∗ ≥ 0 , we know

Revised: T

Original: [Equation]
Revised: [Equation]

Original: which implies 
Revised: which implies ∞

Original: [Equation]
Revised: [Equation]

Original: We observe that
Revised: We observe that

Original: [Equation]
Revised: [Equation]

Original: Taking log to both sides
Revised: Taking log to both sides

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: Note that Inequality (7) tells us lim t [Equation] ( x t ) 
Revised: Note that Inequality (7) tells us lim t →∞ (cid:107)∇ F ( x t ) (cid:107)

Original: [Equation]
Revised: [Equation]

Original: = 0 , hence we can let τ be the time such that
Revised: = 0 , hence we can let τ be the time such that

Original: [Equation]
Revised: [Equation]

Original: ≤ 12 for t ≥ τ .
Revised: ≤ 12 for t ≥ τ .

Original: Then we know
Revised: Then we know

Original: [Equation]
Revised: [Equation]

Original: The above result implies log b 2 T has a uniform upper bound which means b ∞ < ∞ .
Revised: The above result implies log b 2 T has a uniform upper bound which means b ∞ < ∞ .

Original: Now we can start to prove Theorem B.6.
Revised: Now we can start to prove Theorem B.6.

Original: Proof.
Revised: Proof.

Original: Note that when ∆ = 0 for Algorithm 3, or equivalently, δ = 1 for Algorithm 4, if p t = 1 t , all the conditions in Lemma B.1 are satisﬁed by replacing c t by b t .
Revised: Note that when ∆ = 0 for Algorithm 3, or equivalently, δ = 1 for Algorithm 4, if p t = 1 t , all the conditions in Lemma B.1 are satisﬁed by replacing c t by b t .

Original: Hence we have
Revised: Hence we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is by Lemma B.7.
Revised: where the last inequality is by Lemma B.7.

Original: Multiplying both sides by p T b T , we wknow
Revised: Multiplying both sides by p T b T , we wknow

Original: [Equation]
Revised: [Equation]

Original: where the last line is by Lemma B.8.
Revised: where the last line is by Lemma B.8.

Original: B. A NON - ASYMPTOTIC RATE WHEN ∆ = 0 AND δ = 1 FOR CONVEX SMOOTH FUNCTIONS
Revised: B. A NON - ASYMPTOTIC RATE WHEN ∆ = 0 AND δ = 1 FOR CONVEX SMOOTH FUNCTIONS

Original: In the previous section, we only give an asymptotic rate when ∆ = 0 and δ = 1 .
Revised: In the previous section, we only give an asymptotic rate when ∆ = 0 and δ = 1 .

Original: In the following, we will show that, by replacing Assumption 1 by the stronger Assumption 1’, a non-asymptotic rate can be obtained as stated in Theorem B.9.
Revised: In the following, we will show that, by replacing Assumption 1 by the stronger Assumption 1’, a non-asymptotic rate can be obtained as stated in Theorem B.9.

Original: Theorem B.9.
Revised: Theorem B.9.

Original: Suppose F satisﬁes Assumptions 1’ and 2’, when ∆ = 0 for Algorithm 3, or equivalently, δ = 1 for Algorithm 4, by taking p t = 1 t , we have
Revised: Suppose F satisﬁes Assumptions 1’ and 2’, when ∆ = 0 for Algorithm 3, or equivalently, δ = 1 for Algorithm 4, by taking p t = 1 t , we have

Original: [Equation]
Revised: [Equation]

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: where b
Revised: where b

Original: [Equation]
Revised: [Equation]

Original: ηL
Revised: ηL

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst give another well-known characterization of convex and L -smooth functions without proof.
Revised: We ﬁrst give another well-known characterization of convex and L -smooth functions without proof.

Original: Lemma B.10.
Revised: Lemma B.10.

Original: Suppose F satisﬁes Assumption 1’ and 2’, then ∀ x, y ∈ R d
Revised: Suppose F satisﬁes Assumption 1’ and 2’, then ∀ x, y ∈ R d

Original: [Equation]
Revised: [Equation]

Original: Next, we state a simple variant of Lemma B.1, the proof of which is essentially the same as the proof of Lemma B.1, hence we omit it.
Revised: Next, we state a simple variant of Lemma B.1, the proof of which is essentially the same as the proof of Lemma B.1, hence we omit it.

Original: Lemma B.11.
Revised: Lemma B.11.

Original: Suppose the following conditions hold:
Revised: Suppose the following conditions hold:

Original: • F satisﬁes Assumptions 1’ and 2’; • p t ∈ (0 , 1) satisﬁes 1 p t ≥ 1 − p t +1 p t +1 , p 1 = 1 .
Revised: • F satisﬁes Assumptions 1’ and 2’; • p t ∈ (0 , 1) satisﬁes 1 p t ≥ 1 − p t +1 p t +1 , p 1 = 1 .

Original: When ∆ = 0 for Algorithm 3, or equivalently, δ = 1 for Algorithm 4, we have
Revised: When ∆ = 0 for Algorithm 3, or equivalently, δ = 1 for Algorithm 4, we have

Original: [Equation]
Revised: [Equation]

Original: The same as Lemma B.7, we give the following bound on the residual term without proof.
Revised: The same as Lemma B.7, we give the following bound on the residual term without proof.

Original: Lemma B.12.
Revised: Lemma B.12.

Original: Suppose p t ≤ 1 , when ∆ = 0 for Algorithm 3, or equivalently, δ = 1 for Algorithm 4, we have
Revised: Suppose p t ≤ 1 , when ∆ = 0 for Algorithm 3, or equivalently, δ = 1 for Algorithm 4, we have

Original: [Equation]
Revised: [Equation]

Original: Again, the above two lemmas give us
Revised: Again, the above two lemmas give us

Original: [Equation]
Revised: [Equation]

Original: W.l.o.g.
Revised: W.l.o.g.

Original: , we assume b T > ηL 2 in the following analysis.
Revised: , we assume b T > ηL 2 in the following analysis.

Original: Otherwise, we can use the bound b T ≤ ηLto get a trivial convergence rate.
Revised: Otherwise, we can use the bound b T ≤ ηLto get a trivial convergence rate.

Original: Now we deﬁne the time
Revised: Now we deﬁne the time

Original: [Equation]
Revised: [Equation]

Original: This time τ is extremly useful and will ﬁnally help us bound b T .
Revised: This time τ is extremly useful and will ﬁnally help us bound b T .

Original: Now we list the following three important lemmas related to time τ .
Revised: Now we list the following three important lemmas related to time τ .

Original: Lemma B.13.
Revised: Lemma B.13.

Original: With Assumptions 1’ and 2’, when t ≥ τ + 1 , (cid:107)∇ F ( x t ) (cid:107) is non-increasing.
Revised: With Assumptions 1’ and 2’, when t ≥ τ + 1 , (cid:107)∇ F ( x t ) (cid:107) is non-increasing.

Original: Proof.
Revised: Proof.

Original: Taking x = x t , y = x
Revised: Taking x = x t , y = x

Original: t +1 in Lemma B.10, we get
Revised: t +1 in Lemma B.10, we get

Original: [Equation]
Revised: [Equation]

Original: ⇒
Revised: ⇒

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Note that when t ≥ τ + 1 , we know b t > ηL 2 ⇒ 2 L − ηb t > 0 , hence we have (cid:18)L − [Equation] (cid:19)
Revised: Note that when t ≥ τ + 1 , we know b t > ηL 2 ⇒ 2 L − ηb t > 0 , hence we have (cid:18)

Original: [Equation]
Revised: [Equation]

Original: which implies (cid:107)∇ F ( x t +1 ) (cid:107) 2 ≤ (cid:107)∇ F ( x t ) (cid:107) 2 .
Revised: which implies (cid:107)∇ F ( x t +1 ) (cid:107) 2 ≤ (cid:107)∇ F ( x t ) (cid:107) 2 .

Original: This is just what we want.
Revised: This is just what we want.

Original: Lemma B.14.
Revised: Lemma B.14.

Original: With Assumptions 1’ and 2’, if p t = 1 t , when t ≥ τ + 2 ≥ 2 ,
Revised: With Assumptions 1’ and 2’, if p t = 1 t , when t ≥ τ + 2 ≥ 2 ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: This is because
Revised: This is because

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is because t − 1 ≥ τ + 1 , hence (cid:107)∇ F ( x t − 1 ) (cid:107) ≥ (cid:107)∇ F ( x t ) (cid:107) by Lemma
Revised: where the last inequality is because t − 1 ≥ τ + 1 , hence (cid:107)∇ F ( x t − 1 ) (cid:107) ≥ (cid:107)∇ F ( x t ) (cid:107) by Lemma

Original: B.13.
Revised: B.13.

Original: Note that t ≥ 2 , so (cid:107)∇ F ( x t ) (cid:107) 2 b 2 t p t
Revised: Note that t ≥ 2 , so (cid:107)∇ F ( x t [Equation] 2 b 2 t p t

Original: [Equation]
Revised: [Equation]

Revised: −

Original: [Equation]
Revised: [Equation]

Original: 1 ≤ 23 .
Revised: ≤ 23 .

Original: Lemma B.15.
Revised: Lemma B.15.

Original: With Assumptions 1’ and 2’, if p t = 1 t
Revised: With Assumptions 1’ and 2’, if p t = 1 t

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: If τ = 0 , we have
Revised: If τ = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Otherwise, we know τ + 1 ≥ 2 , hence
Revised: Otherwise, we know τ + 1 ≥ 2 , hence

Original: [Equation]
Revised: [Equation]

Original: where the second inequality is due to (8).
Revised: where the second inequality is due to (8).

Original: Now we combine Lemmas B.14 and B.15 to get an upper bound for b T .
Revised: Now we combine Lemmas B.14 and B.15 to get an upper bound for b T .

Original: Lemma B.16.
Revised: Lemma B.16.

Original: With Assumptions 1’ and 2’, if p t = 1 t b T 
Revised: With Assumptions 1’ and 2’, if p t = 1 t b T ≤ max

Original: [Equation]
Revised: [Equation]

Original: max
Revised: 2 ηL b 0 −(cid:19) + (cid:33) , ηL (cid:115)4 + (cid:107) x [Equation] η+ (cid:18) 2 ηL b

Original: [Equation]
Revised: [Equation]

Original: ηL
Revised: (cid:19) + exp (cid:32) [Equation] ∗ (cid:107)η [Equation]

Revised: +(cid:18)

Original: [Equation]
Revised: [Equation]

Original:  
Revised: ηL b [Equation] −(cid:19) + (cid:33) 

Original: Proof.
Revised: Proof.

Original: Note that if b T ≤ ηL 2 , we are done.
Revised: Note that if b T ≤ ηL 2 , we are done.

Original: If b T > ηL 2 , we will bound b T as follows:
Revised: If b T > ηL 2 , we will bound b T as follows:

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is by Lemma B.14.
Revised: where the last inequality is by Lemma B.14.

Original: Noticing p t = 1 t ≤ 1 , combining Lemmas B.11 and
Revised: Noticing p t = 1 t ≤ 1 , combining Lemmas B.11 and

Original: B.12,we can ﬁnd
Revised: B.12,we can ﬁnd

Original: T (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Hence we know
Revised: Hence we know

Original: [Equation]
Revised: [Equation]

Revised: τ

Revised: [Equation]

Revised: (cid:107)

Revised: [Equation]

Revised: (cid:18) 2 ηL

Revised: [Equation]

Original: (cid:19)
Revised: (cid:19)

Revised: [Equation]

Revised: b

Original: [Equation]
Revised: [Equation]

Revised: (cid:32)

Revised: [Equation]

Revised: (cid:107)

Revised: [Equation]

Revised: (cid:18)

Revised: [Equation]

Revised: −

Revised: [Equation]

Revised: (cid:19) + (cid:33) .

Original: The last step is to use the bound on b τ +1 in Lemma B.15.
Revised: The last step is to use the bound on b τ +1 in Lemma B.15.

Original: Finally, the proof of Theorem B.9 is obtained by simply using Lemma B.16 to Equation (8).
Revised: Finally, the proof of Theorem B.9 is obtained by simply using Lemma B.16 to Equation (8).

Original: C M ISSING PROOFS FROM S ECTIONC. I MPORTANT LEMMA
Revised: C M ISSING PROOFS FROM S ECTIONC. I MPORTANT LEMMA

Original: First, we state a general lemma that can be used for a more general setting.
Revised: First, we state a general lemma that can be used for a more general setting.

Original: The proof of the lemma is standard.
Revised: The proof of the lemma is standard.

Original: Lemma C.1.
Revised: Lemma C.1.

Original: Suppose F satisﬁes Assumptions 1’ and 2’ and the following conditions hold:
Revised: Suppose F satisﬁes Assumptions 1’ and 2’ and the following conditions hold:

Original: • w t is generated by
Revised: • w t is generated by

Original: [Equation]
Revised: [Equation]

Original: with η > 0 and c t > 0 is non-decreasing; • a t ∈ (0 , 1] and q t ≥ a t satisfy 1 a t q t ≥ 1 − a t 
Revised: with η > 0 and c t > 0 is non-decreasing; • a t ∈ (0 , 1] and q t ≥ a t satisfy 1 a t q t ≥ 1 − a t +1 a

Original: [Equation]
Revised: [Equation]

Original: [Equation] = [Equation]
Revised: = 1 ;

Original: Then we have
Revised: Then we have

Original: F ( w T +1 ) − F ∗ a T q T c T ≤ (cid:107) x 1 − x ∗ (cid:107)2 η +
Revised: F ( w T +1 ) − F ∗ a T q T c T ≤ (cid:107) x 1 − x ∗ (cid:107)2 η +

Original: T (cid:88) t = (cid:18) 
Revised: T (cid:88) t = (cid:18) L

Revised: [Equation]

Original: L 2 b t − 2 η (cid:19) 
Revised: b t − [Equation] η (cid:19) η

Original: η 2 (cid:107)∇ F ( v t ) (cid:107)c 2 t q 2 t
Revised: [Equation] (cid:107)∇ F ( v t ) (cid:107)c 2 t q 2 t

Original: Proof.
Revised: Proof.

Original: Starting from smoothness
Revised: Starting from smoothness

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to the convexity of F .
Revised: where the last inequality is due to the convexity of F .

Original: Using the update rule ∇ F ( v t ) =
Revised: Using the update rule ∇ F ( v t ) =

Original: q t c t η ( x t − x t +1 ) and w t +1 − v t = a t ( x t +1 − x t ) we obtain
Revised: q t c t η ( x t − x t +1 ) and w t +1 − v t = a t ( x t +1 − x t ) we obtain

Original: F 
Revised: F ( w

Original: [Equation]
Revised: [Equation]

Original: w
Revised: ≤ [Equation] −

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: Dividing both sides by a t q t c t and sumning up from 1 to T , we have
Revised: Dividing both sides by a t q t c t and sumning up from 1 to T , we have

Original: T(cid:88)
Revised: T

Original: [Equation]
Revised: [Equation]

Original: Note that a t ≤ q t ,1 a
Revised: Note that a t ≤ q t ,

Original: [Equation]
Revised: [Equation]

Original: a

Original: [Equation]
Revised: [Equation]

Original: 1 a t +1 q t +1 c t +1 and a t = 1 .
Revised: t +1 and a t = 1 .

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: C. FIRST VARIANT
Revised: C. F

Original: By using Lemma C.1, the proof idea of Theorem 5.1 is the same as the proof of Theorem 4.1.
Revised: By using Lemma C.1, the proof idea of Theorem 5.1 is the same as the proof of Theorem 4.1.

Original: Hence, we omit it for brevity.
Revised: Hence, we omit it for brevity.

Original: C. S ECOND VARIANT
Revised: C. S ECOND VARIANT

Original: By using Lemma C.1, the proof idea of Theorem 5.3 is the same as the proof of Theorem 4.3.
Revised: By using Lemma C.1, the proof idea of Theorem 5.3 is the same as the proof of Theorem 4.3.

Original: Hence, we omit it here.
Revised: Hence, we omit it here.

Original: C. A DISCUSSION ON WHEN ∆ = 0 AND δ =
Revised: C. A DISCUSSION ON WHEN ∆ = 0 AND δ =

Original: Algorithms 5 and 6 become one when ∆ = 0 and δ = 1 .
Revised: Algorithms 5 and 6 become one when ∆ = 0 and δ = 1 .

Original: As discussed in B.4, the challenge is to ﬁnd a explicit bound on b T .
Revised: As discussed in B.4, the challenge is to ﬁnd a explicit bound on b T .

Original: First, we give an asymptotic rate in Theorem C.2 of which the proof idea is the same as the proof of Theorem B.6, thus is omitted.
Revised: First, we give an asymptotic rate in Theorem C.2 of which the proof idea is the same as the proof of Theorem B.6, thus is omitted.

Original: Theorem C.2.
Revised: Theorem C.2.

Original: Suppose F satisﬁes Assumptions 1 and 2’, when ∆ = 0 for Algorithm 5, or equivalently, δ = 1 for Algorithm 6, by taking a t = 2 t +1 , p t = 2 t , we have
Revised: Suppose F satisﬁes Assumptions 1 and 2’, when ∆ = 0 for Algorithm 5, or equivalently, δ = 1 for Algorithm 6, by taking a t = 2 t +1 , p t = 2 t , we have

Original: [Equation]
Revised: [Equation]

Original: Now, we aim to prove the following non-asymptotic rate.
Revised: Now, we aim to prove the following non-asymptotic rate.

Original: Theorem C.3.
Revised: Theorem C.3.

Original: Suppose F satisﬁes Assumptions 1 and 2’, when ∆ = 0 for Algorithm 5, or equivalently, δ = 1 for Algorithm 6, by taking a t = 2 t +1 , p t = 2 t , we have
Revised: Suppose F satisﬁes Assumptions 1 and 2’, when ∆ = 0 for Algorithm 5, or equivalently, δ = 1 for Algorithm 6, by taking a t = 2 t +1 , p t = 2 t , we have

Original: [Equation]
Revised: [Equation]

Original: We shortly discuss here why we can only give a rate in the order of 1 /T but not 1 /T 2 .
Revised: We shortly discuss here why we can only give a rate in the order of 1 /T but not 1 /T 2 .

Original: Recall that in the proof of Theorem B.9, the key step is that after a certain time, (cid:107)∇ F ( x t ) (cid:107) is a non-increasing sequence, by using which we can ﬁnally give a constant upper bound on b T that ﬁnally helps us to get the ﬁnal 1 /T rate.
Revised: Recall that in the proof of Theorem B.9, the key step is that after a certain time, (cid:107)∇ F ( x t ) (cid:107) is a non-increasing sequence, by using which we can ﬁnally give a constant upper bound on b T that ﬁnally helps us to get the ﬁnal 1 /T rate.

Original: However, it is unclear under what condition on b t , (cid:107)∇ F ( v t ) (cid:107) now will be a non-increasing sequence in our accelerated algorithm.
Revised: However, it is unclear under what condition on b t , (cid:107)∇ F ( v t ) (cid:107) now will be a non-increasing sequence in our accelerated algorithm.

Original: Thus it is unclear to us whether it is possible to give a constant bound on b T .
Revised: Thus it is unclear to us whether it is possible to give a constant bound on b T .

Original: Instead, we will show b t can increase at most linearly in this accelerated scheme by a new trick, for which reason, we can ﬁnally obtain the rate in the order of 1 /T .
Revised: Instead, we will show b t can increase at most linearly in this accelerated scheme by a new trick, for which reason, we can ﬁnally obtain the rate in the order of 1 /T .

Original: This guarantees that the convergence of the last iterate is no worse than the variants in
Revised: This guarantees that the convergence of the last iterate is no worse than the variants in

Original: Section 4.
Revised: Section 4.

Original: Proof.
Revised: Proof.

Original: As before, to start with
Revised: As before, to start with

Original: [Equation]
Revised: [Equation]

Original: we use Lemma C.
Revised: we use Lemma C.

Original: [Equation]
Revised: [Equation]

Original: By using
Revised: By using

Original: [Equation]
Revised: [Equation]

Original: and the same technique in the previous proof, we know
Revised: and the same technique in the previous proof [Equation] we know

Original: [Equation]
Revised: [Equation]

Original: So we have
Revised: So we have

Original: [Equation]
Revised: [Equation]

Original: Now we turn to bound b t by observing
Revised: Now we turn to bound b t by observing

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to a t ≤ p t .
Revised: where the last inequality is due to a t ≤ p t .

Original: Then we use (cid:107) x
Revised: Then we use (cid:107) x

Revised: t +1 − x t (cid:107)

Original: [Equation]
Revised: [Equation]

Original:  +1 − x
Revised: and (cid:107)∇ F ( w t +1 ) (cid:107)

Original: [Equation]
Revised: [Equation]

Original: and (cid:107)∇ F ( w t +1 )(cid:107)
Revised: ≤ 2 L ( F ( w t +1 )

Original: [Equation]
Revised: [Equation]

Revised: F

Original: [Equation]
Revised: [Equation]

Original: to get
Revised: to get

Original: [Equation]
Revised: [Equation]

Original: Using this bound
Revised: Using this bound

Original: [Equation]
Revised: [Equation]

Original: we ﬁnally get
Revised: we ﬁnally get

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: T +.
Revised: T +.

Original: D E XPERIMENTS
Revised: D E XPERIMENTS

Original: In this section, we provide some empirical evidence to compare the performances of our algorithms in the deterministic setting.
Revised: In this section, we provide some empirical evidence to compare the performances of our algorithms in the deterministic setting.

Original: Our test function follows the quadratic function used to prove the lower bound of the ﬁrst order method constructed by Nesterov (Nesterov et al., 2018).
Revised: Our test function follows the quadratic function used to prove the lower bound of the ﬁrst order method constructed by Nesterov (Nesterov et al., 2018).

Original: That is
Revised: That is

Original: [Equation]
Revised: [Equation]

Original: where x [ i ] refers to the i -th coordinate of point x ∈ R d .
Revised: where x [ i ] refers to the i -th coordinate of point x ∈ R d .

Original: It is known that F is 4 -smooth and convex with the unique minimizer
Revised: It is known that F is 4 -smooth and convex with the unique minimizer

Original: [Equation]
Revised: [Equation]

Original: We ﬁx d = 101 and set the time horizon to T = 1000 in the test.
Revised: We ﬁx d = 101 and set the time horizon to T = 1000 in the test.

Original: The starting point x 1 is initialized randomly satisfying that every coordinate is uniformly chosen in [0 , 1) .
Revised: The starting point x 1 is initialized randomly satisfying that every coordinate is uniformly chosen in [0 , 1) .

Original: All algorithms share the same x 1 .
Revised: All algorithms share the same x 1 .

Original: For the adaptive algorithms, we choose b 0 = 10 − 2 and set η = 1 without any further tuning.
Revised: For the adaptive algorithms, we choose b 0 = 10 − 2 and set η = 1 without any further tuning.

Original: We also compare with an accelerated algorithm (Lan, 2020), which requires using the smoothness constant L = 4 .
Revised: We also compare with an accelerated algorithm (Lan, 2020), which requires using the smoothness constant L = 4 .

Original: The result is shown in Figure 1.
Revised: The result is shown in Figure 1.

Original: We can ﬁnd that our Algorithms 3 and 4 admit the last iterate convergence.
Revised: We can ﬁnd that our Algorithms 3 and 4 admit the last iterate convergence.

Original: Additionally, both our accelerated algorithms, i.e., Algorithms 5 and 6, enjoy the accelerated property without knowing the smoothness parameter and are competitive against Accelerated Gradient Descent (Lan, 2020) which requires the smooth parameter to set the step size.
Revised: Additionally, both our accelerated algorithms, i.e., Algorithms 5 and 6, enjoy the accelerated property without knowing the smoothness parameter and are competitive against Accelerated Gradient Descent (Lan, 2020) which requires the smooth parameter to set the step size.

Original: Another interesting observation is that it seems AdaGradNorm also exhibits the last iterate convergence.
Revised: Another interesting observation is that it seems AdaGradNorm also exhibits the last iterate convergence.

Original: However, whether this is indeed a property of AdaGradNorm has not been conﬁrmed by the theory.
Revised: However, whether this is indeed a property of AdaGradNorm has not been conﬁrmed by the theory.

Original: We leave this as a future direction.
Revised: We leave this as a future direction.