T HE C ONVERGENCE R ATE OF SGD’ S F INAL I TERATE : A NALYSIS ON D IMENSION D EPENDENCE
Anonymous authors
Paper under double-blind review
A BSTRACT
Stochastic Gradient Descent (SGD) is among the simplest and most popular optimization and machine learning methods.
Running SGD with a ﬁxed step size and outputting the ﬁnal iteration is an ideal strategy one can hope for, but it is still not well-understood even though SGD has been studied extensively for over 70 years.
Given the Θ(log T ) gap between current upper and lower bounds for running SGD for T steps, it was then asked by Koren & Segal (2020) how to characterize the ﬁnal-iterate convergence of SGD with a ﬁxed step size in the constant dimension setting, i.e., d = O (1) .
In this paper, we consider the more general setting for any d ≤ T , proving Ω(log d/ √ T ) lower bounds for the sub-optimality of the ﬁnal iterate of SGD in minimizing non-smooth Lipschitz convex functions with standard step sizes.
Our results provide the ﬁrst general dimension-dependent lower bound on the convergence of SGD’s ﬁnal iterate, partially resolving the COLT open question raised by Koren & Segal (2020).
Moreover, we present a new method in one dimension based on martingale and Freedman’s inequality, which gets the tight O (1 / √ T ) upper bound with mild assumptions.
I NTRODUCTION
Stochastic gradient descent (SGD) was ﬁrst introduced by Robbins & Monro (1951).
It soon became one of the most popular tools in applied machine learning, e.g., Johnson & Zhang (2013); Schmidt et al.
(2017) due to its simplicity and effectiveness.
SGD works by iteratively taking a small step in the opposite direction of an unbiased estimate of sub-gradients and is widely used in minimizing convex function f over a convex domain K .
Formally speaking, given a stochastic gradient oracle for an input x ∈ K , the oracle returns a random vector ˆ g whose expectation is equal to one of the sub-gradients of f at x .
Given an initial point x 1 , SGD generates a sequence of points x 1 , ..., x T + according to the update rule
[Equation]
where Π K denotes projection onto K and { η t } t ≥ 1 is a sequence of step sizes.
Theoretical analysis on SGD usually adopt running average step size, i.e., outputting 1 T (cid:80)
Tt =1 x t in the end, to get optimal rates of convergence in the stochastic approximation setting.
Optimal convergence rates have been achieved in both convex and strongly convex settings when averaging of iterates is used Nemirovskij & Yudin (1983); Zinkevich (2003); Kakade & Tewari (2008); CesaBianchi et al.
Nonetheless, the ﬁnal iterate of SGD, which is often preferred over the running average, as pointed out by Shalev-Shwartz et al.
(2011), has not been very well studied from the theoretical perspective, and convergence results for the ﬁnal iterate are relatively scarce compared with the running average schedule.
Standard choices of step sizes for convex functions include η t = 1 / √ t for unknown horizon T and η t = 1 / √ T for known T , and η t = 1 /t for strongly convex functions.
In these cases, it is known that the ﬁnal-iterate convergence rate of SGD is optimal when f is both smooth and strongly convex (Nemirovski et al.
However, in practice, the convex functions we want to minimize are often non-smooth.
See Cohen et al.
; Lee et al.
for more details.
The convergence rate of SGD’s ﬁnal iterate with standard step sizes in the non-smooth setting is much less explored.
Understanding this problem is essential as the ﬁnal iterate of SGD is popular and used often.
If the last iterate of SGD performs as well as the running average, it yields a very simple, implementable,
Work
Rate
Method
Convexity
Step size
Assumptions
Nemirovski et al.
O (1 /T )
SGD
Strongly 1 /t
Smooth
Jain et al.
O (1 / √ T )
SGD
Convex
Non-standard
Jain et al.
O (1 /T )
SGD
Strongly
Non-standard
Shamir & Zhang (2013)
O (log T/ √ T )
SGD
Convex 1 / √ t
Shamir & Zhang (2013)
O (log T/T )
SGD
Strongly 1 /t
Harvey et al. (2019a) Ω(log T/ √ T )
GD
Convex 1 / √ t d ≥ T
Harvey et al. (2019a)
Ω(log T/T )
GD
Strongly 1 /t d ≥ T
Ours Ω(log d/ √ T )
GD
Convex 1 / √ t, 1 / √ T d ≤ T
Ours Ω(log d/T )
GD
Strongly 1 /t d ≤ T and interpretable form of SGD. If there is a lower bound saying the last iterate of SGD is worse than the running average, we may need to compare the last iterate and running average when implementing the algorithm.
A line of works attempts to understand the convergence rate of the ﬁnal iterate of SGD.
A seminar work Shamir & Zhang (2013) ﬁrst established a near-optimal O (log T/ √ T ) convergence rate for the ﬁnal iterate of SGD with a STANDARD step size schedule η t = 1 / √ t .
Jain et al.
(2019) proved an information-theoretically optimal O (1 / √ T ) upper bound using a rather NON - STANDARD step size schedule.
Roughly speaking, the T steps are divided into log T phases, and the step size decreases by half when entering the next phase.
Many implementations take ever-shrinking step sizes, which is somewhat consistent with this theoretical result.
Harvey et al. (2019a) gave an Ω(log T/ √ T ) lower bound for the STANDARD η t = 1 / √ t step size schedule, but their construction requires the dimension d to be no less than T , which is restrictive.
See Table 1 for more details.
A natural question arises:
Question:
What’s the dependence on dimension d of the convergence rate of SGD’s ﬁnal iterate with standard step sizes when d ≤ T ?
In a recent COLT open question raised by Koren & Segal (2020), the same problem was posed but mainly for the more restrictive constant dimension setting.
Moreover, they conjectured that the right convergence rate of SGD with standard step size in the constant dimensional case is Θ(1 / √ T ) .
As preliminary support evidence for their conjecture, they analyzed a one-dimensional one-sided random walk special case.
However, this result is limited in the one-dimension setting for the particular absolute-value function and thus can not be easily generalized.
Analyzing the ﬁnal-iterate convergence rate of SGD in the general dimension for general convex functions is a more exciting and challenging question.
In particular, in Koren & Segal (2020), they wrote:
For dimension d > 1 , a natural conjecture is that the right convergence rate is Θ(log d/ √ T ) , but we have no indication to corroborate this.
Motivated by this, we mainly focus on analyzing the ﬁnal iterate of SGD with standard step size in general dimension d ≤ T without smoothness assumptions in this paper.
1. O UR CONTRIBUTIONS
Our ﬁrst main result is an Ω(log d/ √ T ) lower bound for SGD minimizing Lipschitz convex functions with a ﬁxed step size η t = 1 / √ T when dimension d ≤ T , generalizing the result in Harvey et al.
(2019a).
Our main observation is that we can let the initial point x 1 stay still for any number of steps as long as 0 is one of the sub-gradient of f at x 1 .
By modifying the original construction of Harvey et al. (2019a), we can keep x 1 at 0 for T − d steps and then ’kick’ it to start taking a similar route as in Harvey et al. (2019a) in a d -dimensional space, which incurs an Ω(log d/ √ T ) sub-optimality.
This result is generalized to Lipschitz convex functions with 1 / √ t decreasing step size schedule with the same sub-optimality, and an Ω(log d/T ) lower bound to strongly convex functions with 1 /t step size schedule is also constructed with the similar technique.
Our lower bound results imply that the last iterate with ﬁxed step size has sub-optimal convergence rate for SGD in general theoretically, which, unfortunately, is used a lot in practice.
As for the upper bound, we present a new method based on martingale and Freedman’s inequality to analyze the one-dimensional case.
Though seemingly straightforward, the convergence rate of ﬁxed-step-size SGD for one-dimensional linear functions is still open and non-trivial.
Koren & Segal (2020) considered minimizing a linear function with a restricted SGD oracle which only outputs ± 1 , reducing this problem to a one-sided random walk.
We relax the restriction on the SGD oracle and prove an O (1 / √ T ) optimal rate for a class of convex functions which we call nearly linear convex functions, with the help of martingale theory.
The class of nearly linear functions captures many common functions, such as linear functions, | x | , e x , x 2 + x, − sin( x ) on [0 , 1] .
Our contributions are summarized as follows:
• We prove an Ω(log d/ √ T ) lower bound for the sub-optimality of the ﬁnal iterate of SGD minimizing non-smooth Lipschitz convex functions with η t = 1 / √ T step size schedule.
We also generalize this bound to the η t = 1 / √ t decreasing step size schedule, and also prove an Ω(log d/T ) lower bound for non-smooth strongly convex functions with η t = 1 /t .
To the best of our knowledge, our results are the ﬁrst that characterize the general dimension dependence in analyzing the ﬁnal iterate convergence of SGD with standard step sizes.
• We prove an optimal O (1 / √ T ) upper bound for the sub-optimality of the ﬁnal iterate of SGD minimizing nearly linear Lipschitz convex functions with ﬁxed Θ(1 / √ T ) step sizes in one dimension, which captures a broad class of convex functions including linear functions.
P RELIMINARIES
Given a bounded convex set K ⊂ R d , and a convex function f : K → R deﬁned on K , our goal is to solve min x
∈K f ( x ) .
In the black-box optimization, there is no explicit representation of f .
Instead, we can use a stochastic oracle to query the sub-gradients of f at x ∈ K .
The set K is given in the form of a projection oracle, which outputs the closest point in K to a given point x in the Euclidean norm.
We introduce several standard deﬁnitions.
Deﬁnition 1 (Sub-gradient) .
A sub-gradient g ∈ R d of a convex function f : K → R at point x , is a vector satisfying that for any x (cid:48) ∈ K ,
[Equation]
We use ∂f ( x ) to denote the set of all sub-gradients of f at x .
Deﬁnition 2 (Strong Convexity) .
A function f : K → R is said to be α -strongly convex, if for any x, y ∈ K and
g ∈ ∂f ( x ) , the following holds:
[Equation]
[Equation]
Deﬁnition 3 (Lipschitz Function) .
A function f : K → R is called G -Lipschitz (with respect to (cid:96)norm), if for any x, y ∈ K , we have that:
[Equation]
Further, if we assume f is convex, the above deﬁnition is equal to (cid:107) g (cid:107) 2 ≤ G for any sub-gradient g .
Let Π K denote the projection operator on K , the (projected) stochastic gradient descent (SGD) is described in Algorithm 1.
We make the following standard assumption on the convex objective f and the SGD algorithm we consider throughout this paper:
Assumption 1 (Standard Assumption) .
We make the following assumptions for the objective f and running SGD:
• The domain K ⊂ R is convex and bounded with diameter D .
1: Given K ⊂ R d , initial point x 1 ∈ K , step size schedule η t : 2: for j = 1 , ..., T : do
Query
ˆ ˆ g t − 1 ] ∈ ∂f x t ) 4: y t − η g t 5: t
• The objective f : K → R is convex and G -Lipschitz, and not necessarily differentiable.
• The output stochastic gradients are bounded: (cid:107) ˆ g t (cid:107) 2 ≤ G , and we have E [ˆ g t | ˆ g 1 , · · · , ˆ g t − 1 ] ∈ ∂f ( x t ) .
The ﬁrst two items hold for both our lower bound and upper bound.
Our results are in the strong versions regarding the third item.
In particular, our lower bound even holds for Gradient Descent (GD), i.e., even if the gradient oracle always outputs ˆ g t ∈ ∂f ( x t ) rather than in expectation, one still has the lower bound Ω(log d/ √ T ) .
Our upper bound works for the SGD, where the oracle’s outputs can be stochastic and one only assumes their expectations are sub-gradients.
L OWER B OUNDS
In this section we prove our main result, that is the ﬁnal iterate of SGD for (non-smooth) Lipschitz convex functions with ﬁxed step sizes η t = 1 / √ T has sub-optimality Ω(log d/ √ T ) , even with deterministic oracle.
We build upon the construction in Harvey et al. (2019a), which is a variant of classical lower bound constructions Nesterov (2003) and proves an Ω(log T/ √ T )
lower bound for the high-dimensional case d ≥ T .
In a nutshell, we consider the setting d ≤ T and construct a function f along with a special subgradient oracle such that the initial point stays still for the ﬁrst T − d steps, and then start moving in Algorithm 1, in which the ﬁnal iterate satisﬁes f ( x T +1 )
= Ω(log d/ √ T ) .
Then we extend the analysis to decreasing step sizes and strongly convex functions.
Let [ j ] be the set of positive integers no larger than j .
For simplicity, we consider convex functions over the d -dimensional Euclidean unit ball.
Let 0 be the d -dimensional all-zero vector.
We present our proof for general convex functions with ﬁxed step sizes ﬁrst.
For decreasing step sizes and strongly convex functions, it is straightforward to scale our construction and get corresponding lower bounds, and we leave the proofs in the Appendix.
Theorem 4.
For any positive integer T > 0 and 1 ≤ d ≤ T , there exists a 1-Lipschitz convex function f : K → R where K ⊂ R d is the Euclidean unit ball, and a non-stochastic sub-gradient oracle satisfying Assumption 1, such that when executing Algorithm 1 on f with initial point 0 and step size schedule η t = 1 / √ T , the last iterate satisﬁes:
[Equation]
Proof.
Let B d be the Euclidean unit ball and deﬁne f : B d → R for i ∈
[ d + 1] ∪ { 0 } to be:
[Equation]
where H i ( x ) = h (cid:62) i x , and we deﬁne for i ≥h i,j = (cid:40) a j ( if 1 ≤ j < i )
− b i ( if i = j ≤ d ) ( if i < j ≤ d ) and a j = 8( d + 1 − j ) , b j = 1( for j ∈ [ d ]) in which h i,j is the j -th coordinate of h i .
Additionally, let h
= 0 and H 0 ( x ) = 0 .
It’s straightforward to check that f is 1-Lipschitz on K , with a minimum value of 0.
Furthermore, ∂f ( x ) is the convex hull of { h i | i ∈
I ( x ) } where I ( x ) =
{ i ≥
| H i ( x ) = f
( x ) } , which is a standard fact in convex analysis Hiriart-Urruty & Lemaréchal (2013).
Setting x 1 = 0 , we observe that f ( x 1 ) = 0 which attains the global minimum, and by the characterization of ∂f ( x ) from above, we know that h
= 0 is a sub-gradient at x 1 .
This observation allows our non-stochastic sub-gradient oracle to output 0 for the ﬁrst T − d steps and outputs h i (cid:48) where i (cid:48)
=
min I ( x ) \ { 0 } for the last d steps.
Deﬁne z 1 = · · · = z T − d +1 = 0 , let T ∗ =
T − d and we further deﬁne
[Equation]
We show inductively that these are precisely the ﬁrst T iterates produced by algorithm 1 when using the sub-gradient oracle deﬁned above.
The following claim is easy to verify from the deﬁnition.
Claim 5.
We have the following claims:
• z t is non-negative.
In particular, z t,j ≥ 1 4 √ T for j < t − T ∗ and z t,j = 0 for j ≥ t − T ∗ .
• z t,j ≤ 1 2 √ T for all j .
In particular, z t ∈ K .
Proof.
It is evident that z t,j = 0 for j ≥ t − T ∗ from the deﬁnition.
As b j √ T = 1 2 √ T , it sufﬁces to prove that 0 ≤ a j t − j − T ∗ − 1 √ T ≤ 1 4 √ T , which is direct as 0 ≤ t − j − T ∗ − 1 ≤ d + 1 − j .
We can now determine the value and sub-gradient at z t .
The case for the ﬁrst T ∗ steps is trivial as the sub-gradient oracle always outputs 0 and x 1 never moves a bit.
For the last d steps we have that z t is supported on its ﬁrst t − T ∗ coordinates, and h (cid:62) t − T ∗ z t = h (cid:62)
i − T ∗ z t for all i > t > T ∗ .
For the other case T ∗ + 1 ≤ i < t , one has that z
[Equation]
which means z (cid:62) t h t − T ∗ >
z (cid:62) t
h
i − T ∗ for all T ∗ + 1 ≤ i < t .
The two results together guarantee that H t − T ∗ ( z t ) ≥ H i − T ∗ ( z t ) for all T ∗ + 1 ≤ i and further f ( z t ) =
H t − T ∗ ( z t ) .
Combining with the fact I ( z t )
= { t − T ∗ , ..., d + 1 } , we conclude that the sub-gradient oracle outputs h t − T ∗ at time t .
Lemma 6.
For the function constructed in this section, the solution of t -th step in algorithm 1 equals to z t for every T ∗ < t ≤ T + 1 .
Proof.
We prove this lemma by induction.
For base case t = T ∗ + 1 , we know that z t =
= x t holds.
Next, when z t = x t holds for some t :
[Equation]
So y t +1 =
z t +1 .
Since z t +1 ∈ K , we have that x t +1 =
z t +1 .
From the above equivalence, we have that the vector x t in algorithm 1 is equal to z t for t ∈
[ T + 1] , which allows the determination of the value of the ﬁnal iterate: f
[Equation]
[Equation]
[Equation]
d
[Equation]
Remark 7.
For the case d = 1 we still have the Ω(1 / √ T ) lower bound, by not using (cid:80) di =1 1 i > log d in the last step. Remark 8.
Notably, our lower bound is valid even for GD, where one can access a noiseless sub-gradient oracle.
Theorem 4 improves the previously known lower bound by a factor of log d , implying an inevitable dependence on the dimension of the convergence of SGD’s ﬁnal iterate.
Though our proof is built upon Harvey et al. (2019a), their construction doesn’t apply directly.
Other natural ways of adaption, for example, cyclic (gradient oracle repeatedly goes over each coordinate), repeated (gradient oracle stays at one coordinate for T/d steps then go to the next), do not work here.
Next, we extend this result to Lipschitz convex functions with step sizes η t = 1 √ t and strongly convex functions with step sizes η t = 1 t , both known to be the optimal choice of learning rate schedule.
The proofs are mostly similar to that of Theorem 4, and we defer them to the Appendix.
Corollary 9.
For any T and 1 ≤ d ≤ T , there exist a 1-Lipschitz convex function f : K
→ R where
K ⊂ R d is the Euclidean unit ball, and a non-stochastic sub-gradient oracle satisfying Assumption 1, such that when executing algorithm 1 on f with initial point 0 and step size schedule η t = 1 / √ t , the last iterate satisﬁes:
[Equation]
Corollary 10.
For any T and 1 ≤ d ≤ T , there exist a 3-Lipschitz and 1-strongly convex function f : K → R where K ⊂ R d is the Euclidean unit ball, and a non-stochastic sub-gradient oracle satisfying Assumption 1, such that when executing Algorithm 1 on f with initial point 0 (the global minimum) and step size schedule η t = 1 /t , the ﬁnal iterate satisﬁes:
[Equation]
U PPER B OUND IN O NE D IMENSION
With our lower bound, it is natural to conjecture that the optimal rate should be Θ(log d/ √ T ) when d ≤ T .
In particular, it’s believed that in the one-dimensional case, the optimal rate is Θ(1 / √ T ) .
As mentioned in the introduction, Koren & Segal (2020) considered a random walk induced by a linear function as evidence for this conjecture in one dimension, which is somewhat restricted.
In this section, we relax their assumptions by considering a function class that we call nearly linear functions, which capture a broad class of functions, including linear functions, and prove an optimal rate O (1 / √ T ) .
For the general Lipschitz convex function class, our analysis also recovers the previously known best bound O (log T/ √ T ) .
4. N EARLY L INEAR F UNCTIONS
Let f ∗ =
min x ∈K f ( x ) .
We need the following deﬁnition before deﬁning nearly linear functions.
Deﬁnition 11.
We say a point x is good if f ( x ) − f ∗ ≤ 4 GD √ T , and deﬁne a set of good points by S :
[Equation]
Now we can deﬁne the convex function family.
In a nutshell, the class of nearly linear functions we consider is the function such that at any not-good point, the absolute value of its sub-gradient is not too small.
Put it formally:
Deﬁnition 12 (Nearly Linear Function) .
We call a convex function f : K → R nearly linear if there exist a constant 0 < c ≤ 1 , such that for any x t / ∈ S which does not belong to the set of good points, we have (cid:12)(cid:12) E
[ˆ g t | ˆ g 1 , · · · , ˆ g t − 1 ] (cid:12)(cid:12) ∈
[ cG, G ] .
We note that any general Lipschitz convex function is nearly linear with c = 1 / √ T , and our later analysis recovers the previously known best bound O (log T/ √ T ) under this interpretation.
Therefore our method is a strict improvement over previous results.
The family of nearly linear functions captures those functions whose sub-gradients do not change drastically outside the set of good points, for example, | x | , e x , x 2 + x, − sin( x ) .
The linear functions considered in Koren & Segal (2020) lie in this family.
The nice property of nearly linear functions allows a martingale-based analysis which gives an improved O (1 / √ T ) bound.
Our proof is based on the Martingale (difference) (See Appendix for a detailed deﬁnition), and we use Freedman’s Inequality given below.
Theorem 13 (Freedman’s Inequality, Theorem 1.6 in Freedman (1975)) .
Consider a real-valued martingale difference sequence { X t } t ≥ 0 such that X 0 = 0 , and E [ X t +1 |F t ] = 0 for all t , where {F t } t ≥ 0 is the ﬁltration deﬁned by the sequence.
Assume that the sequence is uniformly bounded, i.e., | X t | ≤ M almost surely for all t .
Now deﬁne the predictable quadratic variation process of the martingale to be W t = (cid:80) tj =1 E [ X 2 j |F j − 1 ] for all t ≥ 1 .
Then for all (cid:96) ≥ 0 and σ 2 > 0 and any stopping time τ , we have
[Equation]
Some previous works also use martingale theory to analyze SGD.
For example, Harvey et al. (2019a) generalizes Freedman’s Inequality to demonstrate a high probability (w.p. 1 − δ ) suboptimality bound
O (log(1 /δ ) log T/ √ T ) for SGD with standard step sizes, which is improved to O (log(1 /δ )
/ √ T ) by Harvey et al.
(2019b).
4. A NALYSIS
We show how to improve the convergence of the last iterate of SGD with a ﬁxed step size η = 4 D G √ T in one dimension for nearly linear functions.
The proof mainly consists of two parts.
In the ﬁrst part, we prove that for running SGD with ﬁxed step sizes for any convex function satisfying Assumption 1, with very high probability, the solution goes into the set of good points at least once.
In some sense, this is consistent with the known result that averaging scheme can achieve the optimal rate.
It is straightforward to get the following lemma by convexity.
Lemma 14.
For any x ∈ K \ S , ∀∇ f ( x ) ∈ ∂f ( x ) , one has
[Equation]
Suppose we start from an arbitrary point x 1 ∈ K and the (random) sequence of the SGD algorithm with the ﬁxed step size η is denoted by x 1 , x 2 , · · · , x T +1 , i.e. x t +
=
Π K ( x t − η ˆ g t ) .
The following lemma says that with a very high probability, the solution enters S at least once.
Lemma 15.
Given any x 1 ∈ K , and let η = 4 D G √ T .
For any nearly linear function f under the
Assumption 1, deﬁne τ t := ∞ if SGD never goes back to S in the ﬁrst t steps, and τ t :
= min i { 1 ≤ i ≤ t | x i ∈ S} otherwise.
If t ≥ T + 1 and k ≥ 10 , we have that
Pr[ τ t = ∞ | x 1 ] ≤ 2 exp( − Ω( T )) .
Lemma 15 shows that the probability that x t never entered S in the ﬁrst T steps is negligible, whose proof can be found in the Appendix.
In the second part, we bound the tail probability of the sub-optimality of the last iterate for nearly linear functions, from which we can bound the expectation of the sub-optimality.
Roughly speaking, we consider the events that f ( x T +1 ) − f ∗ ≥ GDk √ T and the last T + 1 − i steps all lie out the set of good points, and bound its probability by exp (cid:0) − Ω
( k + ( T + 1 − i ))
(cid:1) .
And by Union Bound we know that the tail probability Pr
[ f ( x T +1 )
− f ∗ ≥ GDk √ T ] ≤ exp( − Ω( k )) , which is enough to get the optimal bound O ( GD √ T ) .
Theorem 16.
Given positive integer T > 0 which is large enough, running SGD with a ﬁxed step size η = 4 D G √ T on any nearly linear function f under Assumption 1 for T steps, one has
[Equation]
where f ∗ =
min x ∈K f ( x ) .
Proof.
We try to bound the tail probability, that is Pr[ f ( x T +1 ) − f ∗ ≥ GDk √ T ] for any k ≥ 10 .
We deﬁne t := ∞ if SGD never goes in the set S and let t := max i { 1 ≤ i ≤ T + 1 | x i ∈ S} otherwise.
One has
[Equation]
where the second equality follows from the fact that Pr[ f ( x T +1 ) − f ∗ ≥ GDk √ T ∧ t = T + 1] = 0 by the deﬁnition of S and k ≥ 10 .
By Lemma 15, we have
[Equation]
which is negligible when T is large enough.
Now we begin to bound Pr[ f ( x T +1 )
− f ∗ ≥ GDk √ T ∧ t = i ] .
We use y i
=
x i − x i − 1 to capture the movement of the solution.
Let n L = inf x ∈S x and n R = sup x ∈S x , which exist because the domain is bounded and the function is continuous.
By Deﬁnition 11, there exists x ∗ ∈ arg min x ∈K f ( x ) such that either | n R − x ∗ | ≥ 4 D/ √ T or | n L − x ∗ | ≥ 4 D/ √ T .
By our setting of step size η , if x j > n R for some j , it is impossible that x j +1 < n L , and vice versa.
Consider the event t = i and assume x i +1 > n R ﬁrst. Hence x j > n R for all i < j ≤ T + 1 .
By the Assumption that f is nearly linear, we have E [ y i ] ∈ [ − ηG, − cηG ] for some constant c ∈ (0 , 1] (See Deﬁnition 11).
Let F i − 1 be the ﬁltration and ˜ y
i =
y
i − E
[ y i | F i − 1 ] .
Obviously, we know E [˜
y
i | F i − 1 ] = 0 , | ˜ y i | ≤ 2 ηG and { ˜ y i } is a martingale difference sequence.
We know that W ( i,T +1] := (cid:80) T +1 j
=
i
+1 E
[ ˜ y i 2 | F i − 1 ] ≤ (cid:80) T +1 j = i +1 E
[ y 2 i | F i − 1 ] ≤ η 2 G 2 ( T + 1 − i ) as | y i | ≤ ηG .
Let (cid:96) =
(cid:80) T +1 j
= i +1 E
[ y j | F j − 1 ] .
It is evident that (cid:96) ≤ − cηG ( T + 1 − i ) by the assumption of being nearly linear and t j > n R .
Condition on f ( x T +1 ) − f ∗ ≥ GDk √ T ∧ t =
i . It follows that (cid:80) T +1 j
= i +
y i ≥ D ( k − 4) √ T .
More speciﬁcally, as x i ∈ S and thus f ( x i ) − f ∗ ≤ 4 GD √ T , we have that f ( x T +1 )
− f ( x i ) ≥ GD ( k − 4) √ T and further x T +1 − x i = (cid:80) T +1 j = i +
y j ≥ D ( k − 4) √ T .
Moreover, we know W
{+( i,T +1] ≤ η 2 G 2 ( T + 1 − i ) and (cid:96) ≤ − cηG ( T +1 − i ) , which means (cid:80) T +
+}
j = i +1 ˜ y j = (cid:80) T +1 j = i +1 y j − (cid:80) T +1 j
= i +1 E
[ y j | F j − 1 ] ≥ D ( k − 4) √ T − (cid:96) = D ( k − 4) √ T + | (cid:96) | .
As for the case when x i +1 < n L , conditioning on f ( x T +1 ) − f ∗ ≥ GDk √ T ∧ t = i ∧ x i +1 < n L , it is similar to get (cid:80) {+T +
+}
j = i [-+
-]+1 ˜
y j ≤ − D ( k − 4) √ T − | (cid:96) | ∧ W ( i : T +1] ≤ η 2 G 2 ( T + 1 − i ) ∧ (cid:96) ≥ cηG ( T + 1 − i ) as well. Hence we have
[Equation]
where the second inequality follows from the analysis above.
Applying Freedman’s Inequality (Theorem 13) over Equation (8) [Equation] one has
[Equation]
Further [Equation] for k ≥
[Equation]
we have
[Equation]
where the last step follows from the fact that for any constant C > 0 one has (cid:80) Ti =1 exp( − Ci ) ≤ (cid:82) T − 1 i =0 exp( − Ci )d i ≤ 1 /C .
As a result, for h ≥ 10 GD/ √ T , we have that
[Equation]
where λ =
Θ( √ TGD ) .
Our conclusion follows from
[Equation]
C ONCLUSION
In this paper, we analyze the ﬁnal iterate convergence rate of SGD with standard step size schedules, proving Ω(log d/ √ T ) and Ω(log d/T ) lower bounds for the sub-optimality of SGD minimizing nonsmooth general convex and strongly convex functions respectively.
We also prove a tight O (1 / √ T ) upper bound for one-dimensional nearly linear functions, a more general setting than Koren & Segal (2020).
This work is the ﬁrst, to the best of our knowledge, that characterizes the dependence on dimension in the general d ≤ T setting, and we hope it can advance our theoretical understanding of the ﬁnal iterate convergence of SGD with standard step sizes, and guide the implementations in practice.
A M ORE P RELIMINARIES A. P RELIMINARIES ON M ARTINGALE
We demonstrate some basic deﬁnitions with a relationship to Martingale, which is used in the proof.
Deﬁnition 17 (Martingale) .
A sequence Y 1 , Y 2 , · · · is said to be a martingale with respect to another sequence X 1 , X 2 , · · · if for all n :
• E ( | Y n | ) <
∞ • E ( Y n +1 | X 1 , . . .
, X n ) =
Y n .
Deﬁnition 18 (Martingale Difference) .
Consider an adapted sequence { X t , F t } ∞−∞ on a probability space.
X t is a martingale difference sequence (MDS) if it satisﬁes the following two conditions for all t :
• E | X t | < ∞ • E [ X t | F t − 1 ] = 0 , a.s.
Deﬁnition 19 (Stopping Time) .
A stopping time with respect to a sequence of random variables
X 1 , X 2 , X 3 , · · · is a random variable τ with the property that for each t , the occurrence or nonoccurrence of the event τ = t depends only on the values of X 1 , X 2 , X 3 , · · · , X t .
B O MITTED P ROOFS FOR S ECTIONB. P ROOF OF C OROLLARY
Proof.
Deﬁne f : K = B
d
→ R and h i ∈ R d for i
∈ [ d + 1] ∪ { 0 } by f ( x ) = max 0 ≤ i ≤ d +1 H
i ( x ) where H i ( x ) = h (cid:62) i x .
For i ≥ 1 we deﬁne h
i,j = (cid:40) a j ( if 1 ≤ j < i ) − b i ( if i = j ≤ d ) ( if i < j ≤ d ) and a j = 8( d + 1 − j ) , b j = √ j + T − d 2 √ T ( for j ∈ [ d ])
Additionally, let h
= 0 and H 0 ( x ) = 0 .
It’s easy to check that f is 1-Lipschitz, with minimal value 0.
We have that ∂f ( x ) is the convex hull of { h i | i ∈
I ( x ) } where I ( x ) =
{ i ≥
| H i ( x ) = f
( x ) } .
Our non-stochastic sub-gradient oracle outputs 0 for the ﬁrst T − d steps and outputs h i (cid:48) where i (cid:48)
=
min I ( x ) \ { 0 } for the last d steps.
Deﬁne z 1 = · · · = z T − d +1 = 0 , let T ∗ =
T − d .
[Equation]
We will show inductively that these are precisely the ﬁrst T iterates produced by algorithm 1 when using the sub-gradient oracle deﬁned above.
The following claim follows from deﬁnition.
Claim 20.
We have the following claims:
• z t is non-negative.
In particular, z t,j ≥ 1 4 √ T for j < t − T ∗ and z t,j = 0 for j ≥ t − T ∗ .
• z t,j ≤ 1 2 √ T ofr all j .
In particular, z t ∈ K .
Proof.
It is obvious that z t,j = 0 for j ≥ t − T ∗ from the deﬁnition.
As b j √ j + T ∗ = 1 2 √ T , it sufﬁces to prove that 0 ≤ a j (cid:80) t − 1 k = j + T ∗ 1 √ k ≤ 1 4 √ T .
We have that
[Equation]
and further t − j − T ∗ √ t −≤ T +
[Equation]
=
[Equation]
by monotony.
Thus 0 ≤ a j (cid:80)
[Equation]
follows from the deﬁnition of a j .
We can now determine the value and sub-differential at z t .
The case for the ﬁrst T ∗ steps is trivial as the sub-gradient oracle always outputs 0 and x 1 never moves a bit.
For the last d steps we have that z t is supported on its ﬁrst t − T ∗ coordinates and h (cid:62) t − T ∗ z t = h (cid:62)
i − T ∗ z t for all i > t > T ∗ .
For T ∗ + 1 ≤ i < t , one has
[Equation]
which means z (cid:62) t h t − T ∗ >
z (cid:62) t
h
i − T ∗ for all T ∗ + 1 ≤ i < t .
The two results together guarantee that
H t − T ∗ ( z t ) ≥ H i − T ∗ ( z t ) for all T ∗ + 1 ≤ i and further f ( z t ) =
H t − T ∗ ( z t ) .
Combining with the fact I ( z t )
= { t − T ∗ , ..., d + 1 } , we conclude that the sub-gradient oracle outputs h t − T ∗ .
Lemma 21.
For the function constructed in this section, the solution of t -th step in algorithmequals to z t for every T ∗ < t ≤ T + 1 .
Proof.
We prove this lemma by induction.
For base case t = T ∗ + 1 , we know that z t =
= x t holds.
Next, when z t = x t holds for some t :
[Equation]
th
[Equation]
So y t +1 =
z t +1 .
Since z t +1 ∈ K , we have that x t +1 =
z t +1 .
From the above claim we have that the vector x t in algorithm 1 is equal to z t for t ∈
[ T + 1] , which allows determination of the value of the ﬁnal iterate: f (
[Equation]
+
[Equation]
[Equation]
T +1 )
[Equation]
H d +1 ( z T +1 ) ≥ d (cid:88)
j = h d +1 ,j z T +1 ,j ≥ d (cid:88)
j =  8( d + 1 − j ) 4 √ T > log d 32 √ T .
B. P ROOF OF C OROLLARY
Proof.
Deﬁne f : K = B
d
→ R by H i ∈ R d for i ∈
[ d + 1] ∪ { 0 } to be:
[Equation]
where H i ( x ) = h (cid:62) i x + 12 (cid:107) x (cid:107) 2 , and we deﬁne for i ≥h i,j = (cid:40) a j ( if 1 ≤ j < i ) − 1 ( if i = j ≤ d ) ( if i < j ≤ d ) and a j = 2( d + 1 − j ) ( for j ∈ [ d ]) in which h i,j is the j -th coordinate of h i .
Additionally, let h
= 0 and H 0 ( x ) = 12 (cid:107) x (cid:107) 2 .
It’s straightforward to check that f is 3-lipschitz and 1-strongly convex on K , with minimal value 0.
Furthermore, ∂f ( x ) is the convex hull of { h i + x | i ∈ I ( x ) } where I ( x ) =
{ i ≥
| H i ( x ) = f
( x ) } , a standard fact in convex analysis Hiriart-Urruty & Lemaréchal (2013).
Setting x 1 = 0 , we observation that f ( x 1 ) = 0 which attains the global minimum, and by the characterization of ∂f ( x ) from above, we know that h
+ x 1 = 0 is a sub-gradient at x 1 .
This observation allows our non-stochastic sub-gradient oracle to output 0 for the ﬁrst T − d steps and outputs h i (cid:48)
+ x where i (cid:48)
=
min I ( x ) \ { 0 } for the last d steps, since outputting 0 in the ﬁrst T − d steps makes x 1 = ...
=
x T − d +1 = 0 by the update rule of SGD.
Deﬁne z 1 = · · · = z T − d +1 = 0 , let T ∗ :
= T − d and
[Equation]
We will show inductively that these are precisely the ﬁrst T iterates produced by algorithm 1 when using the sub-gradient oracle deﬁned above.
The following claim is easy to verify from deﬁnition.
Claim 22.
We have the following claims:
• z t is non-negative.
In particular, z t,j ≥ 1 2( t − 1) for j < t − T ∗ and z t,j = 0 for j ≥ t − T ∗ .
• z t = 0 for t ∈ [ T ∗ + 1] and (cid:107) z t (cid:107) 2 ≤ 1 t − 1 for t > T ∗ + 1 .
Thus z t ∈ K for all t .
Proof.
The ﬁrst claim simply follows from the fact that t − T ∗ − j
−
d
− j + ≤ 1 .
The second claim follows from that ( t − T ∗ − 1)
[Equation]
≤
[Equation]
− 1 .
We can now determine the value and sub-differential at z t .
The case for the ﬁrst T ∗ steps is trivial as the sub-gradient oracle always outputs 0 and x 1 never moves.
For the value of last d steps we observe that z t is supported on its ﬁrst t − T ∗ coordinates by deﬁnition, and as a result h (cid:62) t − T ∗ z t = h (cid:62)
i
[Equation]
T ∗ z t for all i > t > T ∗ .
For the other case T ∗ + 1 ≤ i < t , one have that
[Equation]
which means z (cid:62) t h t − T ∗ >
z (cid:62) t
h
i − T ∗ for all T ∗ + 1 ≤ i < t .
The two results together guarantee that
H t − T ∗ ( z t ) ≥ H i − T ∗ ( z t ) for all T ∗ + 1 ≤ i and thus f ( z t ) =
H t − T ∗ ( z t ) .
Combining with the fact
I ( z t ) =
{ t − T ∗ , ..., d + 1 } , we conclude that the sub-gradient oracle outputs h t − T ∗ + z t .
Lemma 23.
For the function f and its gradient oracle constructed in the proof, the output x t of t -th step in Algorithm 1 equals to z t for every T ∗ < t ≤ T + 1 .
Proof.
We prove this lemma by induction.
For base case t = T ∗ + 1 , we know by the deﬁnition of z t that z t
=
= x t holds.
Next, when z t = x t for some t holds, we have that
[Equation]
(cid:27) −t (cid:40) a j ( if 1 ≤ j < t − T ∗ ) − 1 ( if t − T ∗ =
j ≤ d ) ( if t − T ∗ < j ≤ d ) (cid:41) =
t (cid:26) 1 − ( t − T ∗ − j − 1) a j ( for 1 ≤ j < t − T ∗ ) ( for j ≥ t − T ∗ ) (cid:27) −t (cid:40) a j ( if 1 ≤ j < t − T ∗ ) − 1 ( if t − T ∗ =
j ≤ d ) ( if t − T ∗ < j ≤ d ) (cid:41) =
  1 − ( t − T ∗ − j ) a j t ( for j < t − T ∗ )t ( for j = t − T ∗ ) ( fro j > t − T ∗ )   .
So y t +1 =
z t +1 .
Since z t +1 ∈ K , we have that x t +1 =
z t +1 .
From the above equivalence we have that the vector x t in algorithm 1 is equal to z t for t ∈
[ T + 1] , which allows the determination of the value of the ﬁnal iterate:
[Equation]
d +
[Equation]
z
[Equation]
+
[Equation]
≥ d (cid:88) j = h d +1 ,j z T +1 ,j ≥ d (cid:88)
j =  2( d + 1 − j ) 2 T > log d 5 T .
C O MITTED P ROOFS OF S ECTIONC. P ROOF OF L EMMA
Proof.
We prove this statement by contradiction.
Suppose there exists x ∈ K\S such that |∇ f ( x ) | ≤ G √ T .
By the convexity of f and the deﬁnition of sub-gradient and let x ∗ ∈ K be a minimizer (arbitrarily if the minimizers are not unique), one has
[Equation]
which implies that
[Equation]
This means x ∈ S and thus is a contradiction.
C. P ROOF OF L EMMA
Proof.
Let n L = inf x ∈S x and n R = sup x ∈S x , which exist because the domain is bounded.
By our setting of parameters and deﬁnition, we know if x j > n R , then it is impossible that x j +1 < n L , and vice versa.
As we are considering τ t = ∞ , either x i > n R for all 1 ≤ i ≤ t , or x i < n L for all 1 ≤ i ≤ t .
Without loss of generality, we consider ﬁrst the case where x i > n R for all 1 ≤ i ≤ t .
We deﬁne a random variable y i
= x i − x i − 1 to capture the movement of the solution for 1 ≤ i ≤ t .
Conditioning on τ = ∞ , i.e. x i > n R for all 1 ≤ i ≤ t , we have that E [ y i ] ≤ − cηG = − 4 D/T for i ≥ 2 by Lemma 14 (the projection only makes the expectation smaller).
By standard arguments, let F i be the ﬁltration and ˜ y
i =
y
i − E
[ y i | F i − 1 ] .
It is easy to verify that { ˜ y i } is a martingale difference sequence:
[Equation]
Obviously, one has | ˜
y
i | ≤ Gη = 4 D √ T by the third line of Assumptions 1.
As a result, E [˜ y 2 i | F i − 1 ] =
E [ y 2 i | F i − 1 ] − ( E [ y i | F i − 1 ]) 2 ≤ E
[ y 2 i | F i − 1 ] ≤ η 2 G 2 .
Hence, we get the estimation
W t = (cid:80) t i =2 E
[ ˜ y i 2 | F i − 1 ] ≤ ( t − 1) η 2 G 2 .
Let (cid:96) :
= (cid:80) t i =
2 E [ y i | F i − 1 ] .
We know that (cid:96) =
(cid:80) t i =1 E
[ y i | F i − 1 ] [-| ≥ 
 Dt/T ≥ 4 D-]≤ − tηcG .
So far, we have shown that if x i > n R for all 1 ≤ i ≤ t , then we must have (cid:96) ≤ − tηcG , and at the same time D ≥ (cid:80) ti =
y i ≥ − D which must happen.
Similarly, if x i < n L for all 1 ≤ i ≤ t , then we have (cid:96) ≥ tηcG and − D ≤ (cid:80) ti =
2 y i ≤ D . If we can show the probability that | (cid:96) | ≥ tηcG and | (cid:80) ti =2 y i | ≤ D happen simultaneously is small, we are done.
By the Freedman’s Inequality, if t = T + 1 , one has:
[Equation]
We complete the proof.