Original: ADA S TAR : A Method for Adapting to Interpolation
Revised: ADA S TAR : A Method for Adapting to Interpolation

Original: Abstract
Revised: Abstract

Original: Stochastic convex optimization methods are much faster at minimizing interpolation problems — problems where all sample losses share a common minimizer—than non-interpolating problems.
Revised: Stochastic convex optimization methods are much faster at minimizing interpolation problems — problems where all sample losses share a common minimizer—than non-interpolating problems.

Original: However, standard non-adaptive stochastic gradient methods require step sizes tailored for the interpolation setting, which are sub-optimal for non-interpolating problems, to attain these fast rates.
Revised: However, standard non-adaptive stochastic gradient methods require step sizes tailored for the interpolation setting, which are sub-optimal for non-interpolating problems, to attain these fast rates.

Original: This is problematic because verifying whether a problem is interpolating, without minimizing it, is difficult.
Revised: This is problematic because verifying whether a problem is interpolating, without minimizing it, is difficult.

Original: Moreover, because interpolation is not a stable property—small changes to the data distribution can transform an interpolation problem into a non-interpolating one—we would like our methods to get the fast interpolation rate when it can, while being robust to these perturbations.
Revised: Moreover, because interpolation is not a stable property—small changes to the data distribution can transform an interpolation problem into a non-interpolating one—we would like our methods to get the fast interpolation rate when it can, while being robust to these perturbations.

Original: Stochastic gradient methods with adaptive step sizes are able to achieves these two desiderata in expectation [14].
Revised: Stochastic gradient methods with adaptive step sizes are able to achieves these two desiderata in expectation [14].

Original: In this work, we build on these ideas and present ADA S TAR , an adaptive stochastic gradient method which—with high probability—attains the optimal, fast rate on smooth interpolation problems (up to log factors) and gracefully degrades with the minimal objective value for non-interpolating problems.
Revised: In this work, we build on these ideas and present ADA S TAR , an adaptive stochastic gradient method which—with high probability—attains the optimal, fast rate on smooth interpolation problems (up to log factors) and gracefully degrades with the minimal objective value for non-interpolating problems.

Original: This high probability result is crucial for our second result, where we use ADA S TAR as a building block to construct another stochastic gradient method, termed
Revised: This high probability result is crucial for our second result, where we use ADA S TAR as a building block to construct another stochastic gradient method, termed

Original: ADA S TAR -G, which adapts to interpolation and growth conditions, getting even faster rates.
Revised: ADA S TAR -G, which adapts to interpolation and growth conditions, getting even faster rates.

Original: 1. Introduction
Revised: 1. Introduction

Original: In this work, we study stochastic convex optimization, where we are given n samples s to solve minimize f ( x ) =
Revised: In this work, we study stochastic convex optimization, where we are given n samples s to solve minimize f ( x ) =

Original: E [
Revised: E [

Original: [Equation]
Revised: [Equation]

Original: subject to x ∈ X .
Revised: subject to x ∈ X .

Original: We let Ω denote the sample space and S be an associated Ω -valued random variable.
Revised: We let Ω denote the sample space and S be an associated Ω -valued random variable.

Original: For each sample s ∈ Ω , F ( · ; s ) : R d → R + is a closed, convex, non-negative function which is differentiable on the closed, convex domain X with diameter diam ( X ) ≤ D .
Revised: For each sample s ∈ Ω , F ( · ; s ) : R d → R + is a closed, convex, non-negative function which is differentiable on the closed, convex domain X with diameter diam ( X ) ≤ D .

Original: Throughout the paper, we let
Revised: Throughout the paper, we let

Original: X ⋆ := argmin x ∈X f ( x ) , use x ⋆ to denote an arbitrary element of X ⋆ , and let f ⋆ := f ( x ⋆ ) .
Revised: X ⋆ := argmin x ∈X f ( x ) , use x ⋆ to denote an arbitrary element of X ⋆ , and let f ⋆ := f ( x ⋆ ) .

Original: In particular, we will study problem (1) in the interpolation setting, which we formally define as
Revised: In particular, we will study problem (1) in the interpolation setting, which we formally define as

Original: Definition 1 (Interpolation Problem)
Revised: Definition 1 (Interpolation Problem)

Original: For a non-negative F , Problem (1) is an interpolation problem if F ( x ⋆ ; s ) = 0 for P -almost every s ∈ Ω .
Revised: For a non-negative F , Problem (1) is an interpolation problem if F ( x ⋆ ; s ) = 0 for P -almost every s ∈ Ω .

Original: i.e., f ⋆ = 0 .
Revised: i.e., f ⋆ = 0 .

Original: Interpolation problems are everywhere in modern machine learning.
Revised: Interpolation problems are everywhere in modern machine learning.

Original: Noiseless least squares, overparameterized least squares, classification problems with margin, and even the training loss of many deep learning problems are all examples of interpolation problems.
Revised: Noiseless least squares, overparameterized least squares, classification problems with margin, and even the training loss of many deep learning problems are all examples of interpolation problems.

Original: We note that our definition of interpolation differs from other definitions of interpolation defined in Asi and Duchi [2], Asi et al.
Revised: We note that our definition of interpolation differs from other definitions of interpolation defined in Asi and Duchi [2], Asi et al.

Original: [3], Chadha et al.
Revised: [3], Chadha et al.

Original: [5], Schmidt and Le Roux [16].
Revised: [5], Schmidt and Le Roux [16].

Original: They define an interpolation problem as one where there exists a common minimizer which minimizes all the sample functions simultaneously and do not require the sample functions to be non-negative.
Revised: They define an interpolation problem as one where there exists a common minimizer which minimizes all the sample functions simultaneously and do not require the sample functions to be non-negative.

Original: While our definition may initially seem more restrictive, it is really only requiring that all sample functions to be normalized such that their minimum value is 0; i.e., replace all sample functions F ( · , s ) by the function F ( · , s )
Revised: While our definition may initially seem more restrictive, it is really only requiring that all sample functions to be normalized such that their minimum value is 0; i.e., replace all sample functions F ( · , s ) by the function F ( · , s )

Original: − min x F ( x, s ) .
Revised: − min x F ( x, s ) .

Original: This transformation equalizes different sample loss functions and will ensure that the step sizes our algorithms select—which will depend on F ( · , s ) —are consistent across samples.
Revised: This transformation equalizes different sample loss functions and will ensure that the step sizes our algorithms select—which will depend on F ( · , s ) —are consistent across samples.

Original: We know that stochastic convex optimization methods are much faster at minimizing interpolation problems than non-interpolating problems [7, 12, 13, 17, 18, 20].
Revised: We know that stochastic convex optimization methods are much faster at minimizing interpolation problems than non-interpolating problems [7, 12, 13, 17, 18, 20].

Original: Empirical risk minimization, for example, has O (1 /n + (cid:112) f
Revised: Empirical risk minimization, for example, has O (1 /n + (cid:112) f

Original: ⋆ /n ) convergence rate on problem (1) assuming smoothness; i.e. it has a O (1 /n ) error on interpolation problems, and the standard O (1 / √ n ) rate on non-interpolation problems [17].
Revised: ⋆ /n ) convergence rate on problem (1) assuming smoothness; i.e. it has a O (1 /n ) error on interpolation problems, and the standard O (1 / √ n ) rate on non-interpolation problems [17].

Original: However, as datasets get larger, a natural question is whether mehtods which only require a single pass over the training data Non-adaptive stochastic gradient methods can attain the same improvements but require using step sizes tailored for the interpolation setting, which are sub-optimal for non-interpolating problems [17].
Revised: However, as datasets get larger, a natural question is whether mehtods which only require a single pass over the training data Non-adaptive stochastic gradient methods can attain the same improvements but require using step sizes tailored for the interpolation setting, which are sub-optimal for non-interpolating problems [17].

Original: This is problematic because verifying whether a problem is interpolating, without minimizing it, is difficult.
Revised: This is problematic because verifying whether a problem is interpolating, without minimizing it, is difficult.

Original: Moreover, because interpolation is not a stable property—small changes to the data distribution can transform an interpolation problem into a non-interpolating one—we would like our methods to get the fast interpolation rate when it can while being robust to these perturbations.
Revised: Moreover, because interpolation is not a stable property—small changes to the data distribution can transform an interpolation problem into a non-interpolating one—we would like our methods to get the fast interpolation rate when it can while being robust to these perturbations.

Original: Stochastic gradient methods with adaptive step sizes are able to achieves these two desiderata (e.g., Theorem 4.24 from Orabona [14]) in expectation.
Revised: Stochastic gradient methods with adaptive step sizes are able to achieves these two desiderata (e.g., Theorem 4.24 from Orabona [14]) in expectation.

Original: In this work, we build on these ideas and present ADA S TAR , an algorithm which achieves these desiderata with high probability.
Revised: In this work, we build on these ideas and present ADA S TAR , an algorithm which achieves these desiderata with high probability.

Original: This high probability result will prove to be crucial for designing algorithms which also adapt to growth conditions, which we discuss presently.
Revised: This high probability result will prove to be crucial for designing algorithms which also adapt to growth conditions, which we discuss presently.

Original: While the majority of this paper is dedicated to studying interpolation, we also would like to get even faster rates on realistic problems.
Revised: While the majority of this paper is dedicated to studying interpolation, we also would like to get even faster rates on realistic problems.

Original: To this end, we will also study how interpolation interplays with problems which have κ -growth defined as follows:
Revised: To this end, we will also study how interpolation interplays with problems which have κ -growth defined as follows:

Original: Definition 2 ( κ -growth)
Revised: Definition 2 ( κ -growth)

Original: For λ > 0 and κ ≥ 0 , f has ( λ, κ ) -growth
Revised: For λ > 0 and κ ≥ 0 , f has ( λ, κ ) -growth

Original: if and only if
Revised: if and only if

Original: [Equation]
Revised: [Equation]

Original: Problems which satisfy growth conditions are ubiquitous; linear regression for example has quadratic ( κ = 2 ) growth.
Revised: Problems which satisfy growth conditions are ubiquitous; linear regression for example has quadratic ( κ = 2 ) growth.

Original: The smaller κ is for a population function, the sharper the growth rate around the minimizer, and the faster an algorithm can minimize the population loss— O (1 /n
Revised: The smaller κ is for a population function, the sharper the growth rate around the minimizer, and the faster an algorithm can minimize the population loss— O (1 /n

Original: [Equation]
Revised: [Equation]

Original: ) ) instead of the standard O (1 / √ n ) .
Revised: ) ) instead of the standard O (1 / √ n ) .

Original: Growth and interpolation are synergistic, mutually amplifying convergence rate improvements.
Revised: Growth and interpolation are synergistic, mutually amplifying convergence rate improvements.

Original: For example, we can solve interpolating, quadratic ( κ = 2 ) growth problems exponentially fast, while we can only provide a polynomial O (1 /n ) convergence rate guarantee for problems which exclusively interpolate or exclusively have quadratic growth.
Revised: For example, we can solve interpolating, quadratic ( κ = 2 ) growth problems exponentially fast, while we can only provide a polynomial O (1 /n ) convergence rate guarantee for problems which exclusively interpolate or exclusively have quadratic growth.

Original: However, as we will see in Section 3, unlike linear regression, κ -growth is not always a straightforward property to verify.
Revised: However, as we will see in Section 3, unlike linear regression, κ -growth is not always a straightforward property to verify.

Original: For this reason, in this paper, we present ADA S TAR -G: an algorithm which adapts to growth conditions and interpolation.
Revised: For this reason, in this paper, we present ADA S TAR -G: an algorithm which adapts to growth conditions and interpolation.

Original: For problems that do not satisfy any growth conditions i.e., κ → ∞ ,
Revised: For problems that do not satisfy any growth conditions i.e., κ → ∞ ,

Original: ADA S TAR -G recovers the same rates as ADA S TAR .
Revised: ADA S TAR -G recovers the same rates as ADA S TAR .

Original: 1.1. Our contributions
Revised: 1.1. Our contributions

Original: Our technical contributions are as follows.
Revised: Our technical contributions are as follows.

Original: We present an adaptive stochastic gradient method, termed ADA S TAR , which has a convergence rate of ˜ O
Revised: We present an adaptive stochastic gradient method, termed ADA S TAR , which has a convergence rate of ˜ O

Original: (1 /n + (cid:112) f
Revised: (1 /n + (cid:112) f

Original: ⋆ /n ) in expectation and in high probability, without knowledge of f ⋆ .
Revised: ⋆ /n ) in expectation and in high probability, without knowledge of f ⋆ .

Original: In other words, it attains the minimax optimal fast rate on interpolation problems (up to log factors) and gracefully degrades with the minimal objective value for non-interpolation problems.
Revised: In other words, it attains the minimax optimal fast rate on interpolation problems (up to log factors) and gracefully degrades with the minimal objective value for non-interpolation problems.

Original: We present a second adaptive stochastic gradient method, termed ADA S TAR -G, which adapts to interpolation and κ -growth conditions for κ > 2 .
Revised: We present a second adaptive stochastic gradient method, termed ADA S TAR -G, which adapts to interpolation and κ -growth conditions for κ > 2 .

Original: In particular, with only knowledge of a lower bound on κ , our algorithm, with high probability, has a convergence rate of ˜ O
Revised: In particular, with only knowledge of a lower bound on κ , our algorithm, with high probability, has a convergence rate of ˜ O

Original: (1 /n κκ − 2 ) on interpolating κ -growth problems and a rate of ˜ O
Revised: (1 /n κκ − 2 ) on interpolating κ -growth problems and a rate of ˜ O

Original: (1 /n κ 2( κ − 1) ) , which is optimal up to log factors, on non-interpolating κ -growth problems.
Revised: (1 /n κ 2( κ − 1) ) , which is optimal up to log factors, on non-interpolating κ -growth problems.

Original: We note that information theoretic lower bounds for interpolating κ -growth problems for κ > 2 — which would prove or disprove the optimality (up to log factors) of ADA S TAR -G on said problems— are not known.
Revised: We note that information theoretic lower bounds for interpolating κ -growth problems for κ > 2 — which would prove or disprove the optimality (up to log factors) of ADA S TAR -G on said problems— are not known.

Original: We leave this open problem for future work.
Revised: We leave this open problem for future work.

Original: 1.2. Related Work
Revised: 1.2. Related Work

Original: Vaswani et al.
Revised: Vaswani et al.

Original: [19] show that constant step size AdaGrad has a similar interpolation-adaptive property as ADA S TAR for a different definition of interpolation; however, their results are only for objectives with a finite sum structure, and their convergence rate is dimension dependent.
Revised: [19] show that constant step size AdaGrad has a similar interpolation-adaptive property as ADA S TAR for a different definition of interpolation; however, their results are only for objectives with a finite sum structure, and their convergence rate is dimension dependent.

Original: As mentioned earlier, problems with growth are ubiquitous, and thus, have also been the focus of a lot of work.
Revised: As mentioned earlier, problems with growth are ubiquitous, and thus, have also been the focus of a lot of work.

Original: Hazan and Kale [10] present a stochastic gradient method which is optimal, including log factors, for quadratic growth problems.
Revised: Hazan and Kale [10] present a stochastic gradient method which is optimal, including log factors, for quadratic growth problems.

Original: Chatterjee et al.
Revised: Chatterjee et al.

Original: [6], Juditsky and Nesterov [11], Ramdas and Singh [15] develop methods which attain a faster O (1 /n κ 2( κ − 1) ) on κ -growth problems compared to the standard O (1 / √ n ) rate.
Revised: [6], Juditsky and Nesterov [11], Ramdas and Singh [15] develop methods which attain a faster O (1 /n κ 2( κ − 1) ) on κ -growth problems compared to the standard O (1 / √ n ) rate.

Original: Asi et al.
Revised: Asi et al.

Original: [4] present information theoretic lower bounds showing these faster rates are optimal.
Revised: [4] present information theoretic lower bounds showing these faster rates are optimal.

Original: Interpolation problems with growth have also been extensively studied.
Revised: Interpolation problems with growth have also been extensively studied.

Original: Vaswani et al.
Revised: Vaswani et al.

Original: [18] showed that SGD has a linear convergence rate (exponentially fast) on interpolation problems with growth.
Revised: [18] showed that SGD has a linear convergence rate (exponentially fast) on interpolation problems with growth.

Original: Asi and Duchi [2], Asi et al.
Revised: Asi and Duchi [2], Asi et al.

Original: [3] develop new methods that are more stable than SGD, which also get these fast rates on interpolation problems with growth.
Revised: [3] develop new methods that are more stable than SGD, which also get these fast rates on interpolation problems with growth.

Original: Chadha et al.
Revised: Chadha et al.

Original: [5] develop lower bounds showing the optimality of these methods.
Revised: [5] develop lower bounds showing the optimality of these methods.

Original: Notation
Revised: Notation

Original: Throughout this work, F is β -smooth; i.e., ∥ F ′ ( x ; s ) − F ′ ( y ; s ) ∥ 2 ≤ β ∥ x − y ∥ 2 .
Revised: Throughout this work, F is β -smooth; i.e., ∥ F ′ ( x ; s ) − F ′ ( y ; s ) ∥ 2 ≤ β ∥ x − y ∥ 2 .

Original: We will let h ′ ( x ) denote the gradient of a differentiable function h at x .
Revised: We will let h ′ ( x ) denote the gradient of a differentiable function h at x .

Original: We may use g k in place of
Revised: We may use g k in place of

Original: F ′ ( x k , s k ) for notational simplicity.
Revised: F ′ ( x k , s k ) for notational simplicity.

Original: We let [ n ] denote the sequence of integers 1 , . . . , n .
Revised: We let [ n ] denote the sequence of integers 1 , . . . , n .

Original: For a point u ∈ R d and a set V ⊂ R d , we let dist( u, V ) :
Revised: For a point u ∈ R d and a set V ⊂ R d , we let dist( u, V ) :

Original: = inf v ∈ V ∥
Revised: = inf v ∈ V ∥

Original: u − v ∥ 2 .
Revised: u − v ∥ 2 .

Original: 2. Adapting to Interpolation Problems
Revised: 2. Adapting to Interpolation Problems

Original: In this section, we present our algorithm ADA S TAR , which adapts to minimal function value f ⋆ , attaining the optimal fast 1 /n rate, up to log factors, on interpolation problems.
Revised: In this section, we present our algorithm ADA S TAR , which adapts to minimal function value f ⋆ , attaining the optimal fast 1 /n rate, up to log factors, on interpolation problems.

Original: ADA S TAR executes
Revised: ADA S TAR executes

Original: T = log( n ) epochs of SGD—each call of SGD is executed for n iterations—which totals to using
Revised: T = log( n ) epochs of SGD—each call of SGD is executed for n iterations—which totals to using

Original: O ( n log n ) samples.
Revised: O ( n log n ) samples.

Original: We run SGD with a special adaptive step size choice that depends on the parameter value outputted by the SGD algorithm from the previous epoch.
Revised: We run SGD with a special adaptive step size choice that depends on the parameter value outputted by the SGD algorithm from the previous epoch.

Original: In particular, the step size of the i th iteration of the t th call of SGD is chosen proportional to 1 / (cid:113)(cid:80)
Revised: In particular, the step size of the i th iteration of the t th call of SGD is chosen proportional to 1 / (cid:113)(cid:80)

Original: ik =1 F (ˆ x t ; s k ) , where
Revised: ik =1 F (ˆ x t ; s k ) , where

Original: ˆ x t is the parameter returned by t − 1 epoch’s call of SGD.
Revised: ˆ x t is the parameter returned by t − 1 epoch’s call of SGD.

Original: Intuitively, as t gets larger, the parameter returned by t th call of SGD should get closer to x ⋆ , implying that F (ˆ x t ; s k ) should be closer to f ⋆ .
Revised: Intuitively, as t gets larger, the parameter returned by t th call of SGD should get closer to x ⋆ , implying that F (ˆ x t ; s k ) should be closer to f ⋆ .

Original: This in turn means each subsequent call of SGD has a more aggressive step size choice.
Revised: This in turn means each subsequent call of SGD has a more aggressive step size choice.

Original: The fact that E [ F (ˆ x t ; s k )] ≥ f
Revised: The fact that E [ F (ˆ x t ; s k )] ≥ f

Original: ⋆ intuitively lower bounds how large this step size choice can be, preventing the algorithm from being overly aggressive and diverging.
Revised: ⋆ intuitively lower bounds how large this step size choice can be, preventing the algorithm from being overly aggressive and diverging.

Original: The fact that this lower bound depends on the optimal function value f
Revised: The fact that this lower bound depends on the optimal function value f

Original: ⋆ is precisely why this algorithm is adaptive to f ⋆ .
Revised: ⋆ is precisely why this algorithm is adaptive to f ⋆ .

Original: For problems where f ⋆ is smaller, ADA S TAR will be able to use more aggressive step sizes.
Revised: For problems where f ⋆ is smaller, ADA S TAR will be able to use more aggressive step sizes.

Original: The full description of ADA S TAR can be found in Algorithm 1; the adaptive-step-size SGD algorithm that ADA S TAR calls is described in Algorithm 2.
Revised: The full description of ADA S TAR can be found in Algorithm 1; the adaptive-step-size SGD algorithm that ADA S TAR calls is described in Algorithm 2.

Original: Algorithm 1: algorithm
Revised: Algorithm 1: algorithm

Original: : x estimate of optimal D ← diameter of X for t ← 1 to T do ˆ x t +1 ← SGD- HELPER ( F, n, X, ˆ x t , β ) end return ˆ x T
Revised: : x estimate of optimal parameter for t ← 1 to T do ˆ x t +1 ← SGD- HELPER ( F n X x t , β ) end return ˆ x T

Original: Algorithm 2: SGD- HELPER
Revised: Algorithm 2: SGD- HELPER

Original: Result: ˆ x : estimate of optimal parameter  for i ← 1 to n do α i ← min (cid:18) 14 β ,
Revised: Result: ˆ x : estimate of optimal parameter D ← diam ( X ) for i ← 1 to n do α i ← min (cid:18) 14 β ,

Original: D 2 √ 2 β (cid:80)
Revised: D 2 √ 2 β (cid:80)

Original: ik =1 F ( x 1 ; s k ) (cid:19) x
Revised: ik =1 F ( x 1 ; s k ) (cid:19) x

Original: i +1 ← Π X
Revised: i +1 ← Π X

Original: ( x i − α i F ′
Revised: ( x i − α i F ′

Original: ( x i , s i )) end return ˆ x = 1 n (cid:80) ni
Revised: ( x i , s i )) end return ˆ x = 1 n (cid:80) ni

Original: =1 x
Revised: =1 x

Original: i
Revised: i

Original: Our convergence results for ADA S TAR rely on relating the error guarantees epoch t with the error guarantee of epoch t − 1 ; this relation is presented in Lemma 3.
Revised: Our convergence results for ADA S TAR rely on relating the error guarantees epoch t with the error guarantee of epoch t − 1 ; this relation is presented in Lemma 3.

Original: The proof of Lemma 3 can be found in Section A.1.1.
Revised: The proof of Lemma 3 can be found in Section A.1.1.

Original: Lemma 3 Let F be β -smooth and non-negative.
Revised: Lemma 3 Let F be β -smooth and non-negative.

Original: Let { ˆ x t } Tt =1 be the iterates generated by running
Revised: Let { ˆ x t } Tt =1 be the iterates generated by running

Original: Algorithm 1 on Problem (1) .
Revised: Algorithm 1 on Problem (1) .

Original: We have for all t ∈
Revised: We have for all t ∈

Original: [ T ] ,
Revised: [ T ] ,

Original: [Equation]
Revised: [Equation]

Original: Once we have this relation, we can analyze the recurrence to obtain a non-recursive guarantee on the final epoch of ADA S TAR .
Revised: Once we have this relation, we can analyze the recurrence to obtain a non-recursive guarantee on the final epoch of ADA S TAR .

Original: This result is presented in Theorem 4.
Revised: This result is presented in Theorem 4.

Original: The proof of Theorem 4 can be found in Appendix A.
Revised: The proof of Theorem 4 can be found in Appendix A.

Original: Theorem 4 Let F be β -smooth and non-negative.
Revised: Theorem 4 Let F be β -smooth and non-negative.

Original: Let { ˆ x t } Tt =1 be the iterates generated by running
Revised: Let { ˆ x t } Tt =1 be the iterates generated by running

Original: Algorithm 1 on Problem (1) with T = log 4 ( n ) + 1 .
Revised: Algorithm 1 on Problem (1) with T = log 4 ( n ) + 1 .

Original: We have, with c = 18 , the error guarantee
Revised: We have, with c = 18 , the error guarantee

Original: [Equation]
Revised: [Equation]

Original: We also provide a high probability result for ADA S TAR in Corollary 5.
Revised: We also provide a high probability result for ADA S TAR in Corollary 5.

Original: The high probability result will be useful for proving guarantees about
Revised: The high probability result will be useful for proving guarantees about

Original: ADA S TAR -G.
Revised: ADA S TAR -G.

Original: The proof of Corollary 5 is very similar to the proof of Theorem 4, except that it uses a high probability version of Lemma 3 instead (i.e., Lemma 12).
Revised: The proof of Corollary 5 is very similar to the proof of Theorem 4, except that it uses a high probability version of Lemma 3 instead (i.e., Lemma 12).

Original: The full proof details can be found in Appendix B.
Revised: The full proof details can be found in Appendix B.

Original: Corollary 5 Let F be β -smooth, non-negative, and bounded above by B < ∞ .
Revised: Corollary 5 Let F be β -smooth, non-negative, and bounded above by B < ∞ .

Original: Let { ˆ x t } Tt =1 be the iterates generated by running Algorithm 1 on Problem (1) with T = log 4 ( n ) + 1 .
Revised: Let { ˆ x t } Tt =1 be the iterates generated by running Algorithm 1 on Problem (1) with T = log 4 ( n ) + 1 .

Original: With probability at least 1 − ε , we have
Revised: With probability at least 1 − ε , we have

Original: [Equation]
Revised: [Equation]

Original: ADA S TAR uses a total of O ( n log n ) samples to get an O (1 /n + (cid:112)
Revised: ADA S TAR uses a total of O ( n log n ) samples to get an O (1 /n + (cid:112)

Original: f ⋆ /n ) error guarantee.
Revised: f ⋆ /n ) error guarantee.

Original: In other words, relative to the optimal, non-adaptive first order algorithm (constant step size SGD) which leverages knowledge of f ⋆ , ADA S TAR has to use log n times more samples to adapt to f ⋆ .
Revised: In other words, relative to the optimal, non-adaptive first order algorithm (constant step size SGD) which leverages knowledge of f ⋆ , ADA S TAR has to use log n times more samples to adapt to f ⋆ .

Original: 3. Adaptating to Growth and Interpolation
Revised: 3. Adaptating to Growth and Interpolation

Original: Problems in practice often also satisfy some type of local growth condition like κ -growth.
Revised: Problems in practice often also satisfy some type of local growth condition like κ -growth.

Original: As mentioned in the introduction, these problems enjoy faster convergence guarantees: O (1 /n
Revised: As mentioned in the introduction, these problems enjoy faster convergence guarantees: O (1 /n

Original: [Equation]
Revised: [Equation]

Original: ) , for κ > 1 , compared to the standard O (1 / √ n ) rate.
Revised: ) , for κ > 1 , compared to the standard O (1 / √ n ) rate.

Original: Sometimes, growth conditions are dictated by the loss function and thus are straightforward to verify—e.g., linear regression has quadratic growth.
Revised: Sometimes, growth conditions are dictated by the loss function and thus are straightforward to verify—e.g., linear regression has quadratic growth.

Original: In other circumstances, κ depends heavily on the sampling distribution, making them difficult or impossible to verify a priori.
Revised: In other circumstances, κ depends heavily on the sampling distribution, making them difficult or impossible to verify a priori.

Original: Consider a simple one dimensional classification example.
Revised: Consider a simple one dimensional classification example.

Original: Example 1 (1D seperable classification) Suppose we are interested in classifying inputs a ∈ R into its correct binary label b ∈ {± 1 } .
Revised: Example 1 (1D seperable classification) Suppose we are interested in classifying inputs a ∈ R into its correct binary label b ∈ {± 1 } .

Original: Let the sample loss function be F ( x ; ( a, b ))
Revised: Let the sample loss function be F ( x ; ( a, b ))

Original: =
Revised: =

Original: ϕ ( xab ) , where ϕ ( t ) =
Revised: ϕ ( xab ) , where ϕ ( t ) =

Original: (1 − t ) γ + for some γ ≥ 1 .
Revised: (1 − t ) γ + for some γ ≥ 1 .

Original: Let x ⋆ > 0 be the smallest real number such that f ( x ⋆ ) =
Revised: Let x ⋆ > 0 be the smallest real number such that f ( x ⋆ ) =

Original: E [ ϕ ( x ⋆ ab )]
Revised: E [ ϕ ( x ⋆ ab )]

Original: = 0 ; i.e., it is the perfect classifier of the data with the smallest margin.
Revised: = 0 ; i.e., it is the perfect classifier of the data with the smallest margin.

Original: Suppose there exists B ∈ R such that x ⋆ ≤ B .
Revised: Suppose there exists B ∈ R such that x ⋆ ≤ B .

Original: Let x < x ⋆ (a classifier that incurs error).
Revised: Let x < x ⋆ (a classifier that incurs error).

Original: Let z :=
Revised: Let z :=

Original: ab has a density p , and suppose that x is large enough such that p has some growth locally on the boundary of the its support: p ( z ) ≥ ( z − 1 /x ⋆ ) η for z ∈ [1 /x ⋆ , 1 /x ] .
Revised: ab has a density p , and suppose that x is large enough such that p has some growth locally on the boundary of the its support: p ( z ) ≥ ( z − 1 /x ⋆ ) η for z ∈ [1 /x ⋆ , 1 /x ] .

Original: Then, the error of x can be lower bounded as
Revised: Then, the error of x can be lower bounded as

Original: [Equation]
Revised: [Equation]

Original: (cid:19)
Revised: (cid:19)

Original: [Equation]
Revised: [Equation]

Original: (cid:18) γ η
Revised: (cid:18) γ η

Original: [Equation]
Revised: [Equation]

Original: (cid:19)
Revised: (cid:19)

Original: [Equation]
Revised: [Equation]

Original: (cid:18)
Revised: (cid:18)

Original: [Equation]
Revised: [Equation]

Original: (cid:19) ( x ⋆ − x ) η
Revised: (cid:19) ( x ⋆ − x ) η

Original: + γ + ≳ | x ⋆ − x | η + γ +1 .
Revised: + γ + ≳ | x ⋆ − x | η + γ +1 .

Original: The second equality comes from repeatedly reapplying integration by parts.
Revised: The second equality comes from repeatedly reapplying integration by parts.

Original: The last equality uses the bound B .
Revised: The last equality uses the bound B .

Original: Thus, this problem has κ = η + γ + 1 growth.
Revised: Thus, this problem has κ = η + γ + 1 growth.

Original: The practitioner could get faster rates if they knew κ , but κ is difficult to obtain because η is a data dependent quantity.
Revised: The practitioner could get faster rates if they knew κ , but κ is difficult to obtain because η is a data dependent quantity.

Original: We acknowledge the example is unrealistic—the optimal strategy for the practitioner reduces to examining the sign of a .
Revised: We acknowledge the example is unrealistic—the optimal strategy for the practitioner reduces to examining the sign of a .

Original: However, this strategy would not generalize to higher dimensional settings, or settings where data that has been shifted.
Revised: However, this strategy would not generalize to higher dimensional settings, or settings where data that has been shifted.

Original: The main purpose of the one-dimensional example is to highlight how these more complex classification settings, may also have growth conditions which are hard to verify a priori.
Revised: The main purpose of the one-dimensional example is to highlight how these more complex classification settings, may also have growth conditions which are hard to verify a priori.

Original: This hope that we can leverage unverifiable growth conditions motivates developing methods which are ignorant of and can adapt to κ and interpolation.
Revised: This hope that we can leverage unverifiable growth conditions motivates developing methods which are ignorant of and can adapt to κ and interpolation.

Original: To address this issue, we present ADA S TAR -G, which adapts to κ -growth and interpolation, and only requires knowledge of a lower bound on κ , denoted by ¯ κ .
Revised: To address this issue, we present ADA S TAR -G, which adapts to κ -growth and interpolation, and only requires knowledge of a lower bound on κ , denoted by ¯ κ .

Original: ADA S TAR -G executes K
Revised: ADA S TAR -G executes K

Original: =
Revised: =

Original: O
Revised: O

Original: (log n ) epochs of ADA S TAR —recall, ADA S TAR in turn runs T
Revised: (log n ) epochs of ADA S TAR —recall, ADA S TAR in turn runs T

Original: = O (log n ) epochs of adaptive SGD for n iterations—which totals to using O ( n log 2 n ) samples.
Revised: = O (log n ) epochs of adaptive SGD for n iterations—which totals to using O ( n log 2 n ) samples.

Original: This epoch algorithm leverages the fact that growth, by definition, gives us a way of converting bounds on the function error into bounds on parameter error.
Revised: This epoch algorithm leverages the fact that growth, by definition, gives us a way of converting bounds on the function error into bounds on parameter error.

Original: Ideally we would shrink the domain in proportion to this guarantee (which depends on κ ) with every epoch.
Revised: Ideally we would shrink the domain in proportion to this guarantee (which depends on κ ) with every epoch.

Original: The idea being because the diameter of the space is now smaller, we can choose a more refined step size as well, leading to better convergence guarantees.
Revised: The idea being because the diameter of the space is now smaller, we can choose a more refined step size as well, leading to better convergence guarantees.

Original: However, since κ is unknown, ADA S TAR -G instead shrinks the diameter of the set by half with every epoch.
Revised: However, since κ is unknown, ADA S TAR -G instead shrinks the diameter of the set by half with every epoch.

Original: Up to some epoch, which we term k 0 , this halving will work perfectly and will give us the fast rates we wanted.
Revised: Up to some epoch, which we term k 0 , this halving will work perfectly and will give us the fast rates we wanted.

Original: After epoch k 0 , the shrinking will be too aggressive, but because geometric sums converge very quickly, the extra error that this geometric halving induces will not matter.
Revised: After epoch k 0 , the shrinking will be too aggressive, but because geometric sums converge very quickly, the extra error that this geometric halving induces will not matter.

Original: Thus, this strategy naturally adapts to the growth parameter κ , so long as the practitioner runs ADA S TAR -G for enough epochs, necessitating the lower bound on κ .
Revised: Thus, this strategy naturally adapts to the growth parameter κ , so long as the practitioner runs ADA S TAR -G for enough epochs, necessitating the lower bound on κ .

Original: Finally, because ADA S TAR is the subroutine used in each epoch of ADA S TAR -G, ADA S TAR -G also naturally adapts to interpolation as well.
Revised: Finally, because ADA S TAR is the subroutine used in each epoch of ADA S TAR -G, ADA S TAR -G also naturally adapts to interpolation as well.

Original: The pseudocode for ADA S TAR -G can be found in Algorithm 3, and the convergence guarantee for the algorithm can be found below in Theorem 6.
Revised: The pseudocode for ADA S TAR -G can be found in Algorithm 3, and the convergence guarantee for the algorithm can be found below in Theorem 6.

Original: The proof of Theorem 6 can be found in Appendix C.
Revised: The proof of Theorem 6 can be found in Appendix C.

Original: Algorithm 3:
Revised: Algorithm 3:

Original:  Data: F : sample function being optimized; K : epochs; T : number of epochs of AdaVal; n : number of iterations per epoch of AdaVal; X 1 : parameter domain; x 1 : initial parameter; β : smoothness parameter
Revised: : Adapting to algorithm Data: F : sample function being optimized; K : epochs; T : number of epochs of AdaVal; n : number of iterations per epoch of AdaVal; X 1 : parameter domain; x 1 : initial parameter; β : smoothness parameter

Original: Result: x K : estimate of optimal parameter
Revised: Result: x K : estimate of optimal parameter

Original: D 0 ← diameter of X  for k ← 1 to K do x k +1 ← ADA S TAR ( F, T, n, X k , x k , β )D k
Revised: D 1 ← diam ( X 1 ) for k ← 1 to K do x k +1 ← ADA S TAR ( F, T, n, X k , x k , β )

Original: [Equation] = D [Equation]
Revised: D k +1 ← D k / X k +1 ← { x ∈ X 1 : ∥ x − x k +1 ∥ 2 ≤ D k +1 } end return

Original: X
Revised: x

Original: [Equation] } end return x K
Revised: K

Original: Theorem 6 Let F be β -smooth and non-negative.
Revised: Theorem 6 Let F be β -smooth and non-negative.

Original: Let f have κ -growth with κ ≥ ¯ κ > 2 .
Revised: Let f have κ -growth with κ ≥ ¯ κ > 2 .

Original: Let c 0 be some absolute constant, and ε ∈ (0 , 1) .
Revised: Let c 0 be some absolute constant, and ε ∈ (0 , 1) .

Original: Let { x k } Kk =1 be generated from Algorithm 3 with K = (cid:22) log(cid:18)(cid:16)
Revised: Let { x k } Kk =1 be generated from Algorithm 3 with K such that K ≥ (cid:22) log

Original: [Equation]
Revised: [Equation]

Original: + 1 and T = log 4 ( n ) + 1 .
Revised: + 1 and T = log 4 ( n ) + 1 .

Original: With probability at least 1 − ε , treating f ⋆ as a constant
Revised: With probability at least 1 − ε , treating f ⋆ as a constant

Original: [Equation]
Revised: [Equation]

Original: To summarize, ADA S TAR -G uses n log 2 n samples to get an error ˜ O
Revised: To summarize, ADA S TAR -G uses n log 2 n samples to get an error ˜ O

Original: (1 /n κκ − 2 + ( (cid:112) f
Revised: (1 /n κκ − 2 + ( (cid:112) f

Original: ⋆ /n )
Revised: ⋆ /n )

Original: κκ − 1 ) .
Revised: κκ − 1 ) .

Original: The log factors represent the price ADA S TAR -G pays to adapt to growth and interpolation.
Revised: The log factors represent the price ADA S TAR -G pays to adapt to growth and interpolation.

Original: Appendix A. Proof of TheoremA.1. Supporting Lemmas
Revised: Appendix A. Proof of TheoremA.1. Supporting Lemmas

Original: Lemma 

Revised: Lemma

Original: For a β -smooth and non-negative function h : X → R , we have ∥ h ′ ( x ) ∥ 2 ≤ (cid:112) 4 βh ( x ) for all x ∈ X .
Revised: For a β -smooth and non-negative function h : X → R , we have ∥ h ′ ( x ) ∥ 2 ≤ (cid:112) 4 βh ( x ) for all x ∈ X .

Original: Proof
Revised: Proof

Original: See the proof of Lemma 3.1 in [17].
Revised: See the proof of Lemma 3.1 in [17].

Original: □
Revised: □

Original: Lemma 8 One step of SGM with step size α i on a function F ( · ; s ) has the following guarantee for any y ∈ X
Revised: Lemma 8 One step of SGM with step size α i on a function F ( · ; s ) has the following guarantee for any y ∈ X

Original: [Equation]
Revised: [Equation]

Original: Proof
Revised: Proof

Original: Using convexity, we have that
Revised: Using convexity, we have that

Original: [Equation]
Revised: [Equation]

Original: Since x i +1 solves a convex optimization problem x
Revised: Since x i +1 solves a convex optimization problem x

Original: i +1 =
Revised: i +1 =

Original: argmin x
Revised: argmin x

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: α i ∥
Revised: α i ∥

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: 22 , for all y ∈ X , we have the following optimality condition holds
Revised: 22 , for all y ∈ X , we have the following optimality condition holds

Original: [Equation]
Revised: [Equation]

Original: which means that
Revised: which means that

Original: [Equation]
Revised: [Equation]

Original: where the equality is the result of some algebra.
Revised: where the equality is the result of some algebra.

Original: By Fenchel-Young, we have that
Revised: By Fenchel-Young, we have that

Original: [Equation]
Revised: [Equation]

Original: Combining the two inequalities, gives us the desired result.
Revised: Combining the two inequalities, gives us the desired result.

Original: □
Revised: □

Original: Lemma 9 Let { x i } ni =1 be the iterates generated by running projected stochastic subgradient descent with a non-increasing sequence of step sizes { α i } ni =1 on Problem (1) .
Revised: Lemma 9 Let { x i } ni =1 be the iterates generated by running projected stochastic subgradient descent with a non-increasing sequence of step sizes { α i } ni =1 on Problem (1) .

Original: We have that
Revised: We have that

Original: [Equation]
Revised: [Equation]

Original: Proof
Revised: Proof

Original: See Equation (3.4.8) from Theorem 3.4.7 from [8].
Revised: See Equation (3.4.8) from Theorem 3.4.7 from [8].

Original: □
Revised: □

Original: A.1.1. P ROOF OF SUPPORTING L EMMA
Revised: A.1.1. P ROOF OF SUPPORTING L EMMA

Original: Let F t :
Revised: Let F t :

Original: = σ ( { S j,i } j ∈
Revised: = σ ( { S j,i } j ∈

Original: [ t ] ,i ∈ [ n ] ) be the σ -field generated by the first t epochs of Algorithm 1, so ˆ x t ∈ F t − 1 .
Revised: [ t ] ,i ∈ [ n ] ) be the σ -field generated by the first t epochs of Algorithm 1, so ˆ x t ∈ F t − 1 .

Original: For this proof, let x t,i (or x i ) denote the output of the i th iteration of the subalgorithm
Revised: For this proof, let x t,i (or x i ) denote the output of the i th iteration of the subalgorithm

Original: Algorithm 2 in epoch t of Algorithm 1.
Revised: Algorithm 2 in epoch t of Algorithm 1.

Original: s t,i denotes the sample drawn in the i th iteration of the subalgorithm Algorithm 2 in the t epoch of Algorithm 1.
Revised: s t,i denotes the sample drawn in the i th iteration of the subalgorithm Algorithm 2 in the t epoch of Algorithm 1.

Original: Lemma 8 gives us
Revised: Lemma 8 gives us

Original: [Equation]
Revised: [Equation]

Original: Combining Lemma 8 and Lemma 7 gives
Revised: Combining Lemma 8 and Lemma 7 gives

Original: [Equation]
Revised: [Equation]

Original: Using the fact that α i ≤ 14 β and rearranging, we get
Revised: Using the fact that α i ≤ 14 β and rearranging, we get

Original: [Equation]
Revised: [Equation]

Original: Combining this result with Lemma 9 and using the fact that for all i ∈ [ n ] , E [ ⟨ f ′ ( x t,i ) − g t,i , x t,i − x ⋆ ⟩ | F t − 1 ] = 0 ,
Revised: Combining this result with Lemma 9 and using the fact that for all i ∈ [ n ] , E [ ⟨ f ′ ( x t,i ) − g t,i , x t,i − x ⋆ ⟩ | F t − 1 ] = 0 ,

Original: [Equation]
Revised: [Equation]

Original: n (cid:88)
Revised: n (cid:88)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: 22 (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
Revised: 22 (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: (cid:35)
Revised: (cid:35)

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) F t −(cid:35) .
Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) F t −(cid:35) .

Original: We now plug in the step size α i = min (cid:18) 14 β ,
Revised: We now plug in the step size α i = min (cid:18) 14 β ,

Original: D 2 √ 2 β (cid:80) i k
Revised: D 2 √ 2 β (cid:80) i k

Original: =1 F (ˆ x t − 1 ; s t,k ) (cid:19) .
Revised: =1 F (ˆ x t − 1 ; s t,k ) (cid:19) .

Original: We proceed by cases.
Revised: We proceed by cases.

Original: If α n =
Revised: If α n =

Original: D 2 √ 2 β (cid:80) nk
Revised: D 2 √ 2 β (cid:80) nk

Original: =1 F (ˆ x t − 1 ; s t,k ) , we have that
Revised: =1 F (ˆ x t − 1 ; s t,k ) , we have that

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: (cid:113)(cid:80)
Revised: (cid:113)(cid:80)

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) F t −
Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) F t −

Original: [Equation]
Revised: [Equation]

Original: The second inequality comes from the fact that for any non-negative sequence a 1 , . .
Revised: The second inequality comes from the fact that for any non-negative sequence a 1 , . .

Original: . , a n , (cid:80) ni =
Revised: . , a n , (cid:80) ni =

Original: [Equation]
Revised: [Equation]

Original: ik
Revised: ik

Original: [Equation]
Revised: [Equation]

Original: 2 (cid:112)(cid:80) ni
Revised: 2 (cid:112)(cid:80) ni

Original: =1 a i .
Revised: =1 a i .

Original: The third inequality comes from Jensen’s inequality.
Revised: The third inequality comes from Jensen’s inequality.

Original: In the other case where α n = 14 β , we have that
Revised: In the other case where α n = 14 β , we have that

Original: [Equation]
Revised: [Equation]

Original: Taking the max of these two bounds, taking expectations, using online-to-batch conversion (i.e.,
Revised: Taking the max of these two bounds, taking expectations, using online-to-batch conversion (i.e.,

Original: Jensen’s inequality), and concavity of square root (Jensen’s inequality again), we get the desired result.
Revised: Jensen’s inequality), and concavity of square root (Jensen’s inequality again), we get the desired result.

Original: A.2. Proof of result
Revised: A.2. Proof of result

Original: Let Q := (cid:113) cD 2 β n .
Revised: Let Q := (cid:113) cD 2 β n .

Original: Further let u t =
Revised: Further let u t =

Original: E
Revised: E

Original: [ f (ˆ x t )]
Revised: [ f (ˆ x t )]

Original: − f ⋆ .
Revised: − f ⋆ .

Original: From Lemma 3 , we have that
Revised: From Lemma 3 , we have that

Original: [Equation]
Revised: [Equation]

Original: Let r ( u ) =
Revised: Let r ( u ) =

Original: u be the identity function.
Revised: u be the identity function.

Original: We note that our analysis centers around analyzing the convergence of repeatedly h .
Revised: We note that our analysis centers around analyzing the convergence of repeatedly h .

Original: h is a squareroot function, such that r (0) = 

Revised: h is a squareroot function, such that r (0) =

Original: ≤ Q √ f ⋆ + Q 2 = h (0) .
Revised: ≤ Q √ f ⋆ + Q 2 = h (0) .

Original: Since linear functions grow faster than square root functions, we there must exist a point u ⋆ such that r ( u ⋆ )
Revised: Since linear functions grow faster than square root functions, we there must exist a point u ⋆ such that r ( u ⋆ )

Original: =
Revised: =

Original: h ( u ⋆ ) and r ( u ) > h ( u ) for u ≥ u ⋆ .
Revised: h ( u ⋆ ) and r ( u ) > h ( u ) for u ≥ u ⋆ .

Original: With some algebra, we can show that
Revised: With some algebra, we can show that

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: Suppose our initial choice of ˆ x 1 is such that u 1 ≤ u ⋆ .
Revised: Suppose our initial choice of ˆ x 1 is such that u 1 ≤ u ⋆ .

Original: In this case, we have u 2 = h ( u 1 )
Revised: In this case, we have u 2 = h ( u 1 )

Original: >
Revised: >

Original: r ( u 1 )
Revised: r ( u 1 )

Original: = u 1 ; further, since h and r are monotonically increasing we have u 2 = h
Revised: = u 1 ; further, since h and r are monotonically increasing we have u 2 = h

Original: ( u 1 ) <
Revised: ( u 1 ) <

Original: h ( u ⋆ ) =
Revised: h ( u ⋆ ) =

Original: u ⋆ .
Revised: u ⋆ .

Original: By induction, we have that in this setting, u t ≤ u ⋆ ≤ ˜ u for all t ≥ 1 .
Revised: By induction, we have that in this setting, u t ≤ u ⋆ ≤ ˜ u for all t ≥ 1 .

Original: Thus, the bound of our theorem covers this setting.
Revised: Thus, the bound of our theorem covers this setting.

Original: Thus, it suffices to consider the alternative setting where ˆ x 1 is selected such that u 1 > u ⋆ .
Revised: Thus, it suffices to consider the alternative setting where ˆ x 1 is selected such that u 1 > u ⋆ .

Original: In this setting we know our error is monotonically decreasing because u t
Revised: In this setting we know our error is monotonically decreasing because u t

Original: = h ( u t − 1 ) ≤ r ( u t − 1 ) =
Revised: = h ( u t − 1 ) ≤ r ( u t − 1 ) =

Original: u t − 1 .
Revised: u t − 1 .

Original: In order to get a convergence rate, we need to do more analysis.
Revised: In order to get a convergence rate, we need to do more analysis.

Original: h is difficult to directly analyze because of the square root, but since u ⋆ is the fixed point of h , it suffices to analyze a linear upper bound of h which shares (nearly) the same fixed point as h .
Revised: h is difficult to directly analyze because of the square root, but since u ⋆ is the fixed point of h , it suffices to analyze a linear upper bound of h which shares (nearly) the same fixed point as h .

Original: In particular, by concavity, we will upper bound h by the linear approximation around ˜ u
Revised: In particular, by concavity, we will upper bound h by the linear approximation around ˜ u

Original: to get a recurrence relation.
Revised: to get a recurrence relation.

Original: For reference, we have that h ′ ( u ) =
Revised: For reference, we have that h ′ ( u ) =

Original: Q 2 √ u + f ⋆ where h
Revised: Q 2 √ u + f ⋆ where h

Original: [Equation]
Revised: [Equation]

Original: (˜ u ) =
Revised: (˜ u ) =

Original: [Equation]
Revised: [Equation]

Original: We proceed by cases.
Revised: We proceed by cases.

Original: If u t ≤ ˜ u , we are done.
Revised: If u t ≤ ˜ u , we are done.

Original: Otherwise, we use the fact that u t > ˜ u and h (˜ u ) ≤ ˜ u because ˜ u ≥ u ⋆ to obtain:
Revised: Otherwise, we use the fact that u t > ˜ u and h (˜ u ) ≤ ˜ u because ˜ u ≥ u ⋆ to obtain:

Original: [Equation]
Revised: [Equation]

Original: We choose T = log 4 ( n ) + 1 .
Revised: We choose T = log 4 ( n ) + 1 .

Original: This gives the error guarantee
Revised: This gives the error guarantee

Original: [Equation]
Revised: [Equation]

Original: We conclude by using the following fact
Revised: We conclude by using the following fact

Original: [Equation]
Revised: [Equation]

Original: to give the desired error rate.
Revised: to give the desired error rate.

Original: Appendix B. Proof of CorollaryB.1. Supporting Lemmas
Revised: Appendix B. Proof of CorollaryB.1. Supporting Lemmas

Original: Lemma 10 Let X 1 , . . .
Revised: Lemma 10 Let X 1 , . . .

Original: , X n be a martingale difference sequence adapted to the filtration F t with | X t | ≤ b .
Revised: , X n be a martingale difference sequence adapted to the filtration F t with | X t | ≤ b .

Original: Define V :=
Revised: Define V :=

Original: (cid:80)
Revised: (cid:80)

Original: nt =1 E [ X 2 t |F t − 1 ] .
Revised: nt =1 E [ X 2 t |F t − 1 ] .

Original: For any δ < 1 /e and n ≥
Revised: For any δ < 1 /e and n ≥

Original: [Equation]
Revised: [Equation]

Original: Proof
Revised: Proof

Original: See Lemma 7 in [1].
Revised: See Lemma 7 in [1].

Original: □
Revised: □

Original: Lemma 11 Let X 1 , . . .
Revised: Lemma 11 Let X 1 , . . .

Original: , X n be independent, mean 0, random variables such that | X t | ≤ b and σ 2 t :=
Revised: , X n be independent, mean 0, random variables such that | X t | ≤ b and σ 2 t :=

Original: E [ X 2 t ] .
Revised: E [ X 2 t ] .

Original: For all any 0 < δ < 1 , we have
Revised: For all any 0 < δ < 1 , we have

Original: [Equation]
Revised: [Equation]

Original: Proof
Revised: Proof

Original: We know that X t is ( 6 σ 2 t 5 , 2 b ) sub-exponential.
Revised: We know that X t is ( 6 σ 2 t 5 , 2 b ) sub-exponential.

Original: Using Corollary 3.17 from [9], we have that
Revised: Using Corollary 3.17 from [9], we have that

Original: [Equation]
Revised: [Equation]

Original: (cid:27)(cid:19)
Revised: (cid:27)(cid:19)

Original: Setting the right hand side to δ and rearranging gives the desired result.
Revised: Setting the right hand side to δ and rearranging gives the desired result.

Original: □
Revised: □

Original: Lemma 12 Let F be β -smooth, non-negative, and bounded above by B < ∞ .
Revised: Lemma 12 Let F be β -smooth, non-negative, and bounded above by B < ∞ .

Original: Let { ˆ x t } Tt =1 be the iterates generated by running Algorithm 1 on Problem (1) .
Revised: Let { ˆ x t } Tt =1 be the iterates generated by running Algorithm 1 on Problem (1) .

Original: For all t ∈ [ T ] , with probability at least 1 − ε , we have
Revised: For all t ∈ [ T ] , with probability at least 1 − ε , we have

Original: [Equation]
Revised: [Equation]

Original: Proof
Revised: Proof

Original: For this proof, let x t,i (or x i ) denote the output of the i th iteration of the subalgorithm
Revised: For this proof, let x t,i (or x i ) denote the output of the i th iteration of the subalgorithm

Original: Algorithm 2 in epoch t of Algorithm 1.
Revised: Algorithm 2 in epoch t of Algorithm 1.

Original: We begin with the result from Lemma 9 and plug in the step size schedule used by Algorithm 2.
Revised: We begin with the result from Lemma 9 and plug in the step size schedule used by Algorithm 2.

Original: After applying the same arguments from (2) and (3), we obtain the following bound
Revised: After applying the same arguments from (2) and (3), we obtain the following bound

Original: [Equation]
Revised: [Equation]

Original: Bounding the second term
Revised: Bounding the second term

Original: We bound the (cid:112)(cid:80) nk =1 F (ˆ x t − 1 ; s t,i ) term first.
Revised: We bound the (cid:112)(cid:80) nk =1 F (ˆ x t − 1 ; s t,i ) term first.

Original: Let X
Revised: Let X

Original: i :=
Revised: i :=

Original: F (ˆ x t − 1 ; s t,i ) − f
Revised: F (ˆ x t − 1 ; s t,i ) − f

Original: (ˆ x t − 1 ) .
Revised: (ˆ x t − 1 ) .

Original: We have that |
Revised: We have that |

Original: X i | ≤ B and σ 2 = E
Revised: X i | ≤ B and σ 2 = E

Original: [ X 2 i ] ≤ 2 Bf (ˆ x t − 1 ) .
Revised: [ X 2 i ] ≤ 2 Bf (ˆ x t − 1 ) .

Original: Using Lemma 11, we have that with probability at least 1 − ε/n (cid:88) i =
Revised: Using Lemma 11, we have that with probability at least 1 − ε/n (cid:88) i =

Original: X i ≤ (cid:114)
Revised: X i ≤ (cid:114)

Original: [Equation]
Revised: [Equation]

Original: (ˆ x t − 1 ) log(2 /ε )
Revised: (ˆ x t − 1 ) log(2 /ε )

Original: + 4 B log(2 /
Revised: + 4 B log(2 /

Original: [Equation]
Revised: [Equation]

Original: In particular, we have that
Revised: In particular, we have that

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: /ε )
Revised: /ε )

Original: (cid:19)n (cid:35) .
Revised: (cid:19)n (cid:35) .

Original: The first inequality comes from the fact that √ x + y ≤ √ x + √ y for non-negative x and y .
Revised: The first inequality comes from the fact that √ x + y ≤ √ x + √ y for non-negative x and y .

Original: The second inequality comes from our bound on (cid:80) ni =1 X i .
Revised: The second inequality comes from our bound on (cid:80) ni =1 X i .

Original: The third inequality comes from the fact that f (ˆ x t − 1 ) 1 / 4 n 3 /≤ max(1 /n, (cid:112) f
Revised: The third inequality comes from the fact that f (ˆ x t − 1 ) 1 / 4 n 3 /≤ max(1 /n, (cid:112) f

Original: (ˆ x t − 1 ) /n ) ; we prove this fact by analyzing by cases on whether f (ˆ x t − 1 ) ≤ 1 /n is true.
Revised: (ˆ x t − 1 ) /n ) ; we prove this fact by analyzing by cases on whether f (ˆ x t − 1 ) ≤ 1 /n is true.

Original: Bounding the third term
Revised: Bounding the third term

Original: Now we bound the last term.
Revised: Now we bound the last term.

Original: Let Y i :=
Revised: Let Y i :=

Original: ⟨ f ′ ( x i )
Revised: ⟨ f ′ ( x i )

Original: − g i , x i − x ⋆ ⟩ .
Revised: − g i , x i − x ⋆ ⟩ .

Original: We have that | Y i | ≤ ˜ B .
Revised: We have that | Y i | ≤ ˜ B .

Original: Using Lemma 7 and Cauchy-Schwartz we see that ˜ B = 4 D √ βB suffices.
Revised: Using Lemma 7 and Cauchy-Schwartz we see that ˜ B = 4 D √ βB suffices.

Original: Let
Revised: Let

Original: F ′ i :
Revised: F ′ i :

Original: = σ ( { S t,j } j ∈ [ i ] ) be the σ -field generated by the first i iterations of Algorithm 2 of the t th epoch
Revised: = σ ( { S t,j } j ∈ [ i ] ) be the σ -field generated by the first i iterations of Algorithm 2 of the t th epoch

Original: Algorithm 1; in particular, x i ∈ F ′ i − 1 .
Revised: Algorithm 1; in particular, x i ∈ F ′ i − 1 .

Original: We have σ 2 i =
Revised: We have σ 2 i =

Original: E
Revised: E

Original: [ Y 2 i |F ′
Revised: [ Y 2 i |F ′

Original: i − 1 ] ≤ 16 βD 2 f ( x t ) using Lemma 7.
Revised: i − 1 ] ≤ 16 βD 2 f ( x t ) using Lemma 7.

Original: Applying Lemma 10, we get that with probability at least 1 − ε/ 2 (where ε < 3 log( n ) /e ), n
Revised: Applying Lemma 10, we get that with probability at least 1 − ε/ 2 (where ε < 3 log( n ) /e ), n

Original: [Equation]
Revised: [Equation]

Original: i =
Revised: i =

Original: [Equation]
Revised: [Equation]

Original: In particular we have
Revised: In particular we have

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: Combining terms
Revised: Combining terms

Original: Combining everything and using a union bound, we get that with probability at least 1 − ε ,
Revised: Combining everything and using a union bound, we get that with probability at least 1 − ε ,

Original: [Equation]
Revised: [Equation]

Original: for A 1 , A 2 , A 3 defined accordingly.
Revised: for A 1 , A 2 , A 3 defined accordingly.

Original: We proceed by cases: either the left hand side is less than
Revised: We proceed by cases: either the left hand side is less than

Original: A 1 n + A 2 (cid:113) f
Revised: A 1 n + A 2 (cid:113) f

Original: (ˆ x t −
Revised: (ˆ x t −

Original: [Equation]
Revised: [Equation]

Original: n .
Revised: n .

Original: Otherwise, if we let Q :=
Revised: Otherwise, if we let Q :=

Original: A 3 / √ n , C :
Revised: A 3 / √ n , C :

Original: = A 1 /n + A 2 (cid:112) f
Revised: = A 1 /n + A 2 (cid:112) f

Original: (ˆ x t − 1 ) /n , and H :
Revised: (ˆ x t − 1 ) /n , and H :

Original: = 1 n nf ( x i ) , rearranging, we get that the above inequality implies that H − C ≤ Q √ H + f ⋆ .
Revised: = 1 n nf ( x i ) , rearranging, we get that the above inequality implies that H − C ≤ Q √ H + f ⋆ .

Original: Because
Revised: Because

Original: H ≥ C in this case, we can square both sides and use the quadratic formula, to get the following must hold
Revised: H ≥ C in this case, we can square both sides and use the quadratic formula, to get the following must hold

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: The second inequality comes from an application of the fact √ a + b ≤ √ a + √ b for a, b ≥ 0 and a reapplication of the fact that f (ˆ x t −
Revised: The second inequality comes from an application of the fact √ a + b ≤ √ a + √ b for a, b ≥ 0 and a reapplication of the fact that f (ˆ x t −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 4 n
Revised: 4 n

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Online-to-batch conversion (i.e., Jensen’s inequality) gives the desired result.
Revised: Online-to-batch conversion (i.e., Jensen’s inequality) gives the desired result.

Original: □
Revised: □

Original: B.2. Proof of result
Revised: B.2. Proof of result

Original: The proof structure mirrors Theorem 4 except we use Lemma 12 with ε/T as ε in place of Lemma 3.
Revised: The proof structure mirrors Theorem 4 except we use Lemma 12 with ε/T as ε in place of Lemma 3.

Original: The use of ε/T ensures that the probability of failure any epoch is at most ε by union bound.
Revised: The use of ε/T ensures that the probability of failure any epoch is at most ε by union bound.

Original: If we let u t = f (ˆ x t ) − f ⋆ and further set T = log 4 ( n ) + 1 , we have with probability at least 1 − ε/T .
Revised: If we let u t = f (ˆ x t ) − f ⋆ and further set T = log 4 ( n ) + 1 , we have with probability at least 1 − ε/T .

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: Using the previous notation, we have
Revised: Using the previous notation, we have

Original: [Equation]
Revised: [Equation]

Original: where Q := log (cid:16) log n ε (cid:17) / √ n and B := log (cid:16) log n ε (cid:17) /n .
Revised: where Q := log (cid:16) log n ε (cid:17) / √ n and B := log (cid:16) log n ε (cid:17) /n .

Original: This corresponds to the following value of u ⋆ and ˜ u
Revised: This corresponds to the following value of u ⋆ and ˜ u

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: The remaining steps proceed in the same way to give the desired result.
Revised: The remaining steps proceed in the same way to give the desired result.

Original: Appendix C. Proof of Theorem
Revised: Appendix C. Proof of Theorem

Original: We let
Revised: We let

Original: [Equation]
Revised: [Equation]

Original: We have by union bound that all K calls of ADA S TAR succeed with probability at least 1 − ε .
Revised: We have by union bound that all K calls of ADA S TAR succeed with probability at least 1 − ε .

Original: Throughout this proof, we will do our analysis on this event that all K calls of ADA S TAR succeeds.
Revised: Throughout this proof, we will do our analysis on this event that all K calls of ADA S TAR succeeds.

Original: From Theorem 5, we have that
Revised: From Theorem 5, we have that

Original: [Equation]
Revised: [Equation]

Original: Without loss of generality, assume that f has κ -growth for some κ > 2 .
Revised: Without loss of generality, assume that f has κ -growth for some κ > 2 .

Original: Due to growth, we have the following guarantee:
Revised: Due to growth, we have the following guarantee:

Original: [Equation]
Revised: [Equation]

Original: Colloquially, let k 0 be the last iterate where we can guarantee that there exists x ⋆ ∈ X ⋆ such that x ⋆ ∈ X k 0 .
Revised: Colloquially, let k 0 be the last iterate where we can guarantee that there exists x ⋆ ∈ X ⋆ such that x ⋆ ∈ X k 0 .

Original: Formally, k 0 is largest integer such that
Revised: Formally, k 0 is largest integer such that

Original: [Equation]
Revised: [Equation]

Original: Phrased differently, if κ > 2 , then it is the largest integer such that
Revised: Phrased differently, if κ > 2 , then it is the largest integer such that

Original: [Equation]
Revised: [Equation]

Original: in particular
Revised: in particular

Original: [Equation]
Revised: [Equation]

Original: we have that
Revised: we have that

Original: [Equation]
Revised: [Equation]

Original: Because κ ≥ ¯ κ
Revised: Because κ ≥ ¯ κ

Original: [Equation]
Revised: [Equation]

Original: we have that K ≥ k 0 .
Revised: we have that K ≥ k 0 .

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: The first term we bound as
Revised: The first term we bound as

Original: [Equation]
Revised: [Equation]

Original: The second term we can bound as
Revised: The second term we can bound as

Original: [Equation]
Revised: [Equation]

Original: where the last inequality comes from the fact that (cid:80) j = k 0 +2 r j ≤ 1 / (1 − r ) − 1 for r < 1 .
Revised: where the last inequality comes from the fact that (cid:80) j = k 0 +2 r j ≤ 1 / (1 − r ) − 1 for r < 1 .

Original: Thus, we have that
Revised: Thus, we have that

Original: [Equation]
Revised: [Equation]

Original: Note that by the definition of D k 0 , we have that
Revised: Note that by the definition of D k 0 , we have that

Original: [Equation]
Revised: [Equation]

Original: Evaluating by using the fact that max is upper bounded by addition, and examining dominating terms (assuming that f ⋆ is a constant gives the result.
Revised: Evaluating by using the fact that max is upper bounded by addition, and examining dominating terms (assuming that f ⋆ is a constant gives the result.