Dynamics of SGD with Stochastic Polyak Stepsizes: Truly Adaptive Variants and Convergence to Exact Solution
Abstract
Recently Loizou et al.
(2021), proposed and analyzed stochastic gradient descent (SGD) with stochastic Polyak stepsize (SPS).
The proposed SPS comes with strong convergence guarantees and competitive performance; however, it has two main drawbacks when it is used in non-over-parameterized regimes: (i) It requires a priori knowledge of the optimal mini-batch losses, which are not available when the interpolation condition is not satisﬁed (e.g., regularized objectives), and (ii) it guarantees convergence only to a neighborhood of the solution.
In this work, we study the dynamics and the convergence properties of SGD equipped with new variants of the stochastic Polyak stepsize and provide solutions to both drawbacks of the original SPS.
We ﬁrst show that a simple modiﬁcation of the original SPS that uses lower bounds instead of the optimal function values can directly solve issue (i).
On the other hand, solving issue (ii) turns out to be more challenging and leads us to valuable insights into the method’s behavior.
We show that if interpolation is not satisﬁed, the correlation between SPS and stochastic gradients introduces a bias, which effectively distorts the expectation of the gradient signal near minimizers, leading to non-convergence - even if the stepsize is scaled down during training.
To ﬁx this issue, we propose DecSPS, a novel modiﬁcation of SPS, which guarantees convergence to the exact minimizer - without a priori knowledge of the problem parameters.
For strongly-convex optimization problems, DecSPS is the ﬁrst stochastic adaptive optimization method that converges to the exact solution without restrictive assumptions like bounded iterates/gradients.
Introduction
We consider the stochastic optimization problem:
[Equation]
where each f i is convex and lower bounded.
We denote by X ⇤ the non-empty set of optimal points x ⇤ of equation (1).
We set f
⇤ :=
min x 2 R d f ( x ) , and f ⇤ i :=
inf x 2 R d f i ( x ) .
In this setting, the algorithm of choice is often Stochastic Gradient Descent (SGD), i.e. x k +
= x
k   k r f S k ( x k ) , where  k > 0 is the stepsize at iteration k ,
S k ✓ [ n ] a random subset of datapoints (minibatch) with cardinality B sampled independently at each iteration k , and r f S k ( x ) := 1 B
P i 2 S k r f i ( x ) is the minibatch gradient.
A careful choice of  k is crucial for most applications [4, 14].
The simplest option is to pick  k to be constant over training, with its value inversely proportional to the Lipschitz constant of the gradient.
While this choice yields fast convergence to the neighborhood of a minimizer, two main problems arise: (a) the optimal  depends on (often unknown) problem parameters — hence often requires heavy tuning ; and (b) it cannot be guaranteed that X ⇤ is reached in the limit [13, 16, 15].
A simple ﬁx for the last problem is to allow polynomially decreasing stepsizes (second option) [23]: this choice for  k often leads to convergence to X ⇤ , but hurts the overall algorithm speed.
The third option, which became very popular with the rise of deep learning, is to implement an adaptive stepsize.
These methods do not commit to a ﬁxed schedule, but instead use the optimization statistics (e.g. gradient history, cost history) to tune the value of  k at each iteration.
These stepsizes are known to work very well in deep learning [35], and include Adam [19], Adagrad [11], and RMSprop [29].
Ideally, a theoretically grounded adaptive method should yield fast convergence to X ⇤ without knowledge of problem dependent parameters, such as the gradient Lipshitz constant or the strong convexity constant.
As a result, an ideal adaptive method should require very little tuning by the user, while matching the performance of a ﬁne-tuned  k .
However, while in practice this is the case for common adaptive methods such as Adam and AdaGrad, the associated convergence rates often rely on strong assumptions — e.g. that the iterates live on a bounded domain, or that gradients are uniformly bounded in norm [11, 32, 31].
While the above assumptions are valid in the constrained setting, they are problematic for problems deﬁned in the whole R d .
A promising new direction in the adaptive stepsizes literature is based on the idea of Polyak stepsizes, introduced by [25] in the context of deterministic convex optimization.
Recently [22] successfully adapted Polyak stepsizes to the stochastic setting, and provided convergence rates matching ﬁnetuned SGD — while the algorithm does not require knowledge of the unknown quantities such as the gradient Lipschitz constant.
The results especially shines in the overparameterized strongly convex setting, where linear convergence to x ⇤ is shown.
This result is especially important since, under the same assumption, no such rate exists for AdaGrad (see e.g. [31] for the latest results) or other adaptive stepsizes.
Moreover, the method was shown to work surprisingly well on deep learning problems, without requiring heavy tuning [22].
Even if the stochastic Polyak stepsize (SPS) [22] comes with strong convergence guarantees, it has two main drawbacks when it is used in non-over-parameterized regimes: (i) It requires a priori knowledge of the optimal mini-batch losses, which are not often available for big batch sizes or regularized objectives (see discussion in §1.1) and (ii) it guarantees convergence only to a neighborhood of the solution.
In this work, we study the dynamics and the convergence properties of SGD equipped with new variants of SPS for solving general convex optimization problems.
Our new proposed variants provide solutions to both drawbacks of the original SPS.
1.1 Background and Technical Preliminaries
The stepsize proposed by [22] is
[Equation]
where  b , c > 0 are problem-independent constants, f
[-= 
 |S k | P i 2 S k f i , f ⇤ S k = inf x 2 R d f S k ( x ) .-][Equation]
Dependency on f ⇤ S k .
Crucially the algorithm requires knowledge of f ⇤ S k for every realization of the mini-batch S k .
In the non-regularized overparametrized setting (e.g. neural networks), f S k is often zero for every subset S [34].
However, this is not the only setting where f ⇤ S is computable: e.g., in the regularized logistic loss with batch size 1, it is possible to recover a cheap closed form expression for each f
⇤ i [22].
Unfortunately, if the batch-size is bigger than 1 or the loss becomes more demanding (e.g. cross-entropy), then no such closed-form computation is possible .
Rates and comparison with AdaGrad.
In the convex overparametrized setting (more precisely, under the interpolation condition, i.e. 9 x ⇤ 2 X ⇤ : inf x 2 R d f S ( x ) =
f S ( x ⇤ ) for all S , see also §2),
SPS max enjoys a convergence speed of O (1 /k ) , without requiring knowledge of the gradient Lipschitz constant or other problem parameters.
Recently, [31] showed that the same rate can be achieved for
AdaGrad in the same setting.
However, there is an important difference: the rate of [31] is technically
O ( dD 2 /k ) , where d is the problem dimension and D 2 is a global bound on the squared distance to the minimizer, which is assumed to be ﬁnite.
Not only does SPS max not have this dimension dependency, which dates back to crucial arguments in the AdaGrad literature [11, 21], but also does not require bounded iterates.
While this assumption is satisﬁed in the constrained setting, it has no reason to hold in the unconstrained scenario.
Unfortunately, this is a common problem of all
AdaGrad variants: with the exception of [33] (which works in a slightly different scenario), no rate can be provided in the stochastic setting without the bounded iterates/gradients [12] assumption — even after assuming strong convexity.
However, in the non-interpolated setting, AdaGrad enjoys a convergence guarantee of O (1 / p k ) (with the bounded iterates assumption).
A similar rate does not yet exist for SPS, and our work aims at ﬁlling this gap.
1.2 Main Contributions
As we already mentioned, in the non-interpolated setting SPS max has the following issues:
Issue (1) :
For B > 1 (minibatch setting), SPS max requires the exact knowledge of f ⇤ S .
This is not practical.
Issue (2) :
SPS max guarantees convergence to a neighborhood of the solution.
It is not clear how to modify it to yield convergence to the exact minimizer.
Having the above two issues in mind, the main contributions of our work (see also Table 1 for a summary of the main complexity results obtained in this paper) are summarized as follows:
• In §3, we provide a direct solution for Issue (1).
We explain how only a lower bound on f ⇤ S (trivial if all f i s are non-negative) is required for convergence to a neighborhood of the solution.
While this neighborhood is bigger that the one for SPS max , our modiﬁed version provides a practical baseline for the solution to the second issue.
• We explain why Issue (2) is highly non-trivial and requires an in-depth study of the bias induced by the interaction between gradients and Polyak stepsizes.
Namely, we show that simply multiplying the stepsize of SPS max by 1 / p k — which would work for vanilla SGD [23] — yields a bias in the solution found by SPS (§4), regardless of the estimation of f ⇤ S .
• In §5, we provide a solution to the problem (Issue (2)) by introducing additional structure — as well as the ﬁx to Issue (1) — into the stepsize.
We call the new algorithm Decreasing SPS (DecSPS), and provide a convergence guarantee under the bounded domain assumption — matching the standard AdaGrad results.
• In §5.2 we go one step further and show that, if strong convexity is assumed, iterates are bounded {+with probability 
+}
and hence we can remove the bounded iterates assumption.
To the best of our knowledge, DecSPS, is the ﬁrst stochastic adaptive optimization method that converges to the exact solution without assuming strong assumptions like bounded iterates/gradients.
• In §5.3 we provide extensions of our approach to the non-smooth setting.
• In §6, we corroborate our theoretical results with experimental testing.
Background on Stochastic Polyak Stepsize
In this section, we provide a concise overview of the results in [22], and highlight the main assumptions and open questions.
To start, we remind the reader that problem (1) is said to be interpolated if there exists a problem solution x ⇤ 2 X ⇤ such that inf Stepsize Citation
Assumptions No Knowledgeof f ⇤ S Exact Convergence Theorem
SPS max [22] convex, smooth
Thm. 2.
SPS ` max
This paper convex, smooth
Thm. 3.1, Cor. 3.
DecSPS
This paper convex, smooth, bounded iterates
Cor. 5.3, O (1 / p K ) This paper strongly-convex, smooth Thm. 5.5, O (1 / p K )
DecSPS-NS
This paper convex, bounded iterates/grads
Cor. F.2, O (1 / p K ) ⇥ [Table]
⇤ ,
¯ x K = 1 K P K  1 k =0 x k all c = O at batch size B can be quantiﬁed by the following quantity, introduced by [22] and studied also in [31, 9]: ﬁx a batch size B , and let S ✓ [ n ] with |S| = B .
[Equation]
[Equation]
It is easy to realize that as soon as problem (1) is interpolated, then  2 B = 0 for each B  n .
In addition, note that  2 B is non-increasing as a function of B .
We now comment on the main result from [22].
Theorem 2.1 (Main result of [22]) .
Let each f i be L i -smooth convex functions.
Then SGD with
SPS max , mini-batch size B , and c = 1 , converges as
[Equation]
where ↵ = min n 1 2 cL max ,  b o and ¯ x K = 1 K
P
K  1 k =0 x k .
If in addition f is µ -strongly convex, then, for any c  1 / 2 , SGD with SPS max converges as: E k x k [Equation] where again ↵ = min {
[Equation]
cL max ,  [Equation] and L max = max { L i } ni =1 is the maximum smoothness constant.
In the overparametrized setting, the result guarantees convergence to the exact minimizer, without knowledge of the gradient Lipschitz constant (as vanilla SGD would instead require) and without assuming bounded iterates (in contrast to [31]).
As soon as (1) a regularizer is applied to the loss (e.g. L 2 penalty), or (2) the number of datapoints gets comparable to the dimension, then the problem is not interpolated and SPS max only converges to a neighborhood and it gets impractical to compute f ⇤ S — this is the setting we study in this paper.
Remark 1 (What if kr f S k k = 0 ?) .
In the rare case that kr f S k ( x k ) k 2 = 0 , there is no need to evaluate the stepsize.
In this scenario, the update direction r f S k ( x k ) = 0 and thus the iterate is not updated irrespective of the choice of step-size.
If this happens, the user should simply sample a different minibatch.
We note that in our experiments (see §6), such event never occurred.
Related work on Polyak stepsize: The classical Polyak stepsize [25] has been successfully used in the analysis of deterministic subgradient methods in different settings [5, 7, 18].
First attempts on providing an efﬁcient variant of the stepsize that works well in the stochastic setting were made in [3, 24].
However, as explained in [22], none of these approaches provide a natural stochastic extension with strong theoretical convergence guarantees, and thus Loizou et al.
[22] proposed the stochastic Polyak stepsize SPS max as a better alternative.
Despite its recent appearance, SPS max has already been used and analyzed as a stepsize for SGD for solving structured non-convex problems [15], in combination with other adaptive methods [31], with a moving target [17] and in the update rule of stochastic mirror descent [9].
These extensions are orthogonal to our approach, and we speculate that our proposed variants can also be used in the above settings.
We leave such extensions for future work.
Removing f ⇤ S from SPS
As motivated in the last sections computing f ⇤ S in the non-interpolated setting is not practical.
In this section we explore the effect of using a lower bound ` ⇤ S  f ⇤ S instead in the SPS max deﬁnition.
[Equation]
Such a lower bound is easy to get for many problems of interest: indeed, for standard regularized regression and classiﬁcation tasks, the loss is non-negative hence one can pick ` ⇤ S = 0 , for any
S ✓ [ n ] .
The obvious question is: what is the effect of estimating ` ⇤ S on the convergence rates in Thm. 2.1?
We found that the proof of [22] is easy to adapt to this case, by using the following fundamental bound (see also Lemma B.2):
[Equation]
The following results can be seen as an easy extension of the main result of [22], under a newly deﬁned suboptimality measure:
[Equation]
Theorem 3.1.
Under SPS ` max , the same exact rates in Thm.
2.1 hold (under the corresponding assumptions), after replacing  2 B with ˆ  2 B .
And we also have an easy practical corollary.
Corollary 3.2.
In the context of Thm. 3.1, assume all f i s are non-negative and estimate ` ⇤ S
=for all S ✓ [ n ] .
Then the same exact rates in Thm.
2.1 hold for SPS ` max , after replacing  2 B with f ⇤ =
f ( x ⇤ ) .
A numerical illustration of this result can be found in Fig.
In essence, both theory and experiments conﬁrm that, if interpolation is not satisﬁed, then we have a linear rate until a convergence ball, where the size is optimal under exact knowledge of f ⇤ S .
Instead, under interpolation, if all the f i s are non-negative and f ⇤ = 0 , then SPS max =SPS ` max .
Finally, in the less common case in practice where f ⇤ >
but we still have interpolation, then SPS max converges to the exact solution while
SPS ` max does not.
To conclude SPS ` max does not (of course) work better than SPS max , but it is a practical variant which we can use as a baseline in §5 for an adaptive stochastic Polyak stepsize with convergence to the true x
⇤ in the non-interpolated setting.
Bias in the SPS dynamics.
In this section, we study convergence of the standard SPS max in the non-interpolated regime, under an additional (decreasing) multiplicative factor, in the most ideal setting: batch size 1, and we have knowledge of each f
⇤ i .
That is, we consider  k = min { f ik ( x k )  f
⇤ ik c k kr f
ik ( x k ) k 2 ,  b } with c k ! 1 , e.g. c k = O ( p k ) or c k = O ( k ) .
We note that, in the SGD case, simply picking e.g.  k =  0 / p k +would guarantee convergence of f ( x k ) to f ( x ⇤ ) , in expectation and with high probability [20, 23].
Therefore, it is natural to expect a similar behavior for SPS, if 1 /c k saﬁsﬁes the usual Robbins-Monro conditions [27]:
[Equation]
We show that this is not the case : quite interestingly, f ( x k ) converges to a biased solution due to the correlation between r f i k and  k .
we show this formally, in the case of non-interpolation (otherwise both SGD and SPS do not require a decreasing learning rate).
Counterexample.
Consider the following ﬁnite-sum setting: f ( x ) = 12 f 1 ( x ) + 12 f 2 ( x ) with f 1 ( x ) =
a 1 2 ( x  1) 2 , f 2 ( x ) = a 2 2 ( x + 1) 2 .
To make the problem interesting, we choose a 1 =and a 2 = 1 : this introduces asymmetry in the average landscape with respect to the origin.
During optimization, we sample f 1 and f 2 independently and seek convergence to the unique minimizer x ⇤ =
a 1  a 2 a 1 + a 2 = 1 / 3 .
The ﬁrst thing we notice is that x ⇤ is not a stationary point for the dynamics under SPS.
Indeed note that since f ⇤ i = 0 for i = 1 , 2 we have (assuming  b large enough):  k r f i k
( x ) = x  1 2 c k , if i k = 1 , and
k r f i k
( x ) =
x [-+1 
-]+12 c k if i k
= 2 .
Crucially, note that this update is curvature-independent .
The expected update is E i k
[  k r f i k ( x )]
= x  1 4 c k + x +14 c k =
c k x .
Hence, the iterates can only converge to x = 0 — because this is the only ﬁxed point for the update rule.
The proof naturally extends to the multidimensional setting, an illustration can be found in Fig. No
[Figure] Interpolation In the same picture, we show how our modiﬁed variant of the vanilla stepsize — we call this new algorithm DecSPS, see §5 — instead converges to the correct solution.
Remark 2 .
SGD with (non-adaptive) stepsize  k instead keeps the curvature, and therefore is able to correctly estimate the average E i
k [  k r f i k ( x )] =
k 2 ( a 1 + a 2 )
h x
[Equation]
[Equation]
1 + a
[Equation]
i — precisely because  k is independent from r f i k .
From this we can see that SGD can only converge to the correct stationary point x ⇤ =
a 1  a 2 a 1 + a 2 — because again this is the only ﬁxed point for the update rule.
In the appendix, we go one step further and provide an analysis of the bias of SPS in the onedimensional quadratic case (Prop. D.2).
Yet, we expect the precise characterization of the bias phenomenon in the non-quadratic setting to be particularly challenging.
We provide additional insights in §D.2.
Instead, in the next section, we show how to effectively modify  k to yield convergence to x ⇤ without further assumptions.
DecSPS: Convergence to the exact solution
We propose the following modiﬁcation of the vanilla SPS proposed in [22], designed to yield convergence to the exact minimizer while keeping the main adaptiveness properties 3 .
We call it
Decreasing SPS (DecSPS), since it combines a steady stepsize decrease with the adaptiveness of SPS.
[Equation]
for k 2 N , where c k 6 = 0 for every k 2 N .
We set c
1 = c 0 and   1 =  b > 0 (stepsize bound, similar to [22]), to get  0 :
= 1 c 0 · min
[Equation]
Lemma 5.1.
Let each f i be L
i smooth and let ( c k ) 1 k =0 be any non-decreasing positive sequence of real numbers.
Under DecSPS, we have min
[Equation]
Remark 3 .
As stated in the last lemma, under the assumption of c k non-decreasing,  k is trivially non-increasing since  k  c k  1  k  1 /c k .
The proof can be found in the appendix, and is based on a simple induction argument.
5.1 Convergence under bounded iterates
The following result provides a proof of convergence of SGD for the  k sequence deﬁned above.
Theorem 5.2.
Consider SGD with DecSPS and let ( c k ) 1 k =0 be any non-decreasing sequence such that c k  1 , 8 k 2 N .
Assume that each f i is convex and L i smooth.
We have:
[Equation]
where D 2 := max
[Equation]
x
[Equation]
k
[Equation] L : [Equation] max n max i { L i } ,
[Equation]
o and ¯
[Equation]
K
P
K
[Equation] x [Equation] If ˆ  2 B = 0 , then c k = 1 for all k 2 N leads to a rate O ( 1 K ) , well known from [22].
If ˆ  2 B > 0 , as for the standard SGD analysis under decreasing stepsizes, the choice c k = O ( p k ) leads to an optimal asymptotic trade-off between the deterministic and the stochastic terms, hence to the asymptotic rate O (1 / p k ) since
P
[-K  1 k = 
-][Equation]
 [Equation] p K .
Moreover, picking c 0 = 1 minimizes the speed of convergence for the deterministic factor.
Under the assumption that ˆ  2 B ⌧ ˜ LD 2 (e.g. reasonable distance initialization-solution and L max > 1 /  b ), this factor is dominant compared to the factor involving ˆ  2 B .
For this setting, the rate simpliﬁes as follows.
Corollary 5.3.
Under the setting of Thm.
5.2, for c k = p k + 1 ( c  1 = c 0 ) we have
[Equation]
Remark 4 (Beyond bounded iterates) .
The result above crucially relies on the bounded iterates assumption: D 2 < 1 .
To the best of our knowledge, if no further regularity is assumed, modern convergence results for adaptive methods (e.g. variants of AdaGrad) in convex stochastic programming require 4 this assumption, or else require gradients to be globally bounded.
To mention a few: [11, 26, 32, 8, 31].
A simple algorithmic ﬁx to this problem is adding a cheap projection step onto a large bounded domain [21].
We can of course include this projection step in DecSPS, and the theorem above will hold with no further modiﬁcation.
Yet we found this to be not necessary: the strong guarantees of SPS in the strongly convex setting [22] let us go one step beyond: in §5.2 we show that, if each f i is strongly convex (e.g. regularizer is added), then one can bound the iterates globally with probability one, without knowledge of the gradient Lipschitz constant.
To the best of our knowledge, no such result exist for AdaGrad — except [30], for the deterministic case.
Remark 5 (Dependency on the problem dimension) .
In standard results for AdaGrad, a dependency on the problem dimension often appears (e.g. Thm. 1 in [31]).
This dependency follows from a bound on the AdaGrad preconditioner that can be found e.g. in Thm.
4 in [21].
In the SPS case no such dependency appears — speciﬁcally because the stepsize is lower bounded by 1 / (2 c k L max ) .
5.2 Removing the bounded iterates assumption
We prove that under DecSPS the iterates live in a set of diameter D max almost surely.
This can be done by assuming strong convexity of each f i .
The result uses this alternative deﬁnition of neighborhood : ˆ  2 B, max := max S ✓ [ n ] , |S| = B [ f S ( x ⇤ )
` ⇤ S ] .
Note that trivially ˆ  2 B, max < 1 under the assumption that all f i are lower bounded and n < 1 .
Proposition 5.4.
Let each f i be µ
i -strongly convex and L i -smooth.
The iterates of SGD with
DecSPS with c k = p k + 1 (and c  1 = c 0 ) are such that k x k  x ⇤ k 2  D 2max almost surely 8 k 2 N , where D
[Equation]
The proof relies on the variations of constants formula and an induction argument — it is provided in the appendix.
We are now ready to state the main theorem for the unconstrained setting, which follows from Prop. 5.4 and Thm. 5.2.
Theorem 5.5.
Consider SGD with the DecSPS stepsize  k :=
[Equation]
for k  1 and  0 deﬁned as at the beginning of this section.
Let each f i be µ
i -strongly convex and
L i -smooth:
[Equation]
Remark 6 (Strong Convexity) .
The careful reader might notice that, while we assumed strong convexity, our rate is slower than the optimal O (1 /K ) .
This is due to the adaptive nature of DecSPS.
[-It is indeed notoriously
-]
It is indeed notoriously hard to achieve a convergence rate of O (1 /K ) for adaptive methods in the strongly convex regime.
While further investigations will shed light on this interesting problem, we note that the result we provide is somewhat unique in the literature : we are not aware of any adaptive method that enjoys a similar convergence rate without either (a) assuming bounded iterates/gradients or (b) assuming knowledge of the gradient Lipschitz constant or the strong convexity constant. Remark 7 (Comparison with Vanilla SGD) .
On a convex problem the non-asymptotic performance of SGD with a decreasing stepsize ⌘ / p k strongly depends on the choice of ⌘ .
The optimizer might diverge if ⌘ is too big for the problem at hand.
Indeed, most bounds for SGD, under no access to the gradient Lipschitz constant, display a dependency on the size of the domain and rely on projections after each step. If one applies the method in the unconstrained setting, such convergence rates technically do not hold, and tuning is sometimes necessary to retrieve stability and good performance. Instead, for DecSPS, simply by adding a small regularizer, the method is guaranteed to converge at the non-asymptotic rate we derived even in the unconstrained setting. 5.3 Extension to the non-smooth setting
For any S ✓ [ n ] , we denote in this section by g S ( x ) the subgradient of f S evaluated at x .
We discuss the extension of DecSPS to the non-smooth setting.
A straightforward application of DecSPS leads to a stepsize  k which is no longer lower bounded (see
Lemma 5.1) by the positive quantity min n
[Equation]
o .
Indeed, the gradient Lipschitz constant in the non-smooth case is formally L max = 1 .
Hence,  k prescribed by DecSPS can get arbitrarily small 5 for ﬁnite k .
One easy solution to the problem is to enforce a lower bound, and adopt a new proof technique.
Speciﬁcally we propose the following:
[Equation]
where c k 6 = 0 for every k  0 ,  `   b is a small positive number and all the other quantities are deﬁned as in DecSPS.
In particular, as for DecSPS, we set c
=
c 0 and   1 =  b .
Intuitively,  k is selected to live in the interval [ c 0  ` /c k , c 0  b /c
k ] (see proof in §F, appendix), but has subgradientdependent adaptive value.
In addition, this stepsize is enforced to be monotonically decreasing.
Theorem 5.6.
For any non-decreasing positive sequence ( c k ) 1 k =0 , consider SGD with DecSPS-NS.
Assume that each f i is convex and lower bounded.
We have
[Equation]
where D 2 :=
max k 2 [ K  1] k x k  x
⇤ k 2 and G 2 := max k 2 [ K  1] k g S k ( x k ) k 2 .
One can then easily derive an O (1 / p k ) convergence rate.
This is presented in §F (appendix).
Numerical Evaluation
We evaluate the performance of DecSPS with c k = c
p k + 1 on binary classiﬁcation tasks, with regularized logistic loss f ( x ) = 1 n
P ni =1 log(1 + exp( y i · a > i x ))
+  2 k x k 2 , where a i 2 R d is the feature vector for the i -th datapoint and y i 2 {  1 , 1 } is the corresponding binary target.
We study performance on three datasets: (1) a Synthetic Dataset, (2)
The A1A dataset [6] and (3)
the
Breast Cancer dataset [10].
We choose different regularization levels and batch sizes bigger than 1.
Details are reported in §G, and the code is available at https://github.com/aorvieto/DecSPS .
At the batch sizes and regularizer levels we choose, the problems do not satisfy interpolation.
Indeed, running full batch gradient descent yields f ⇤ > 0 .
While running SPS max on these problems (1) does not guarantee convergence to f ⇤ and (2) requires full knowledge of the set of optimal function values
{ f ⇤ S } |S| = B , in DecSPS we can simply pick the lower bound 0 =
` ⇤ S  f ⇤ S for every S .
Supported by Theorems 5.2 & 5.5 & 5.6, we expect SGD with DecSPS to converge to the minimum f ⇤ .
Stability of DecSPS.
DecSPS has two hyperparameters: the upper bound  b on the ﬁrst stepsize and the scaling constant c 0 .
While Thm.
5.6 guarantees convergence for any positive value of these hyperparameters, the result of Thm.
5.2 suggests that using c 0 = 1 yields the best performance under the assumption that ˆ  2 B ⌧ ˜ LD 2 (e.g. reasonable distance of initialization from the solution, and
L max > 1 /  b ).
In Fig.
3, we show on the synthetic dataset that (1) c
= 1 is indeed the best choice in this setting and (2) the performance of SGD with DecSPS is almost independent of  b .
Similar ﬁndings are reported and commented in Figure 7 (Appendix) for the other datasets.
Hence, for all further experiments , we choose the hyperparameters  b = 10 , c 0 = 1 .
Comparison with vanilla SGD with decreasing stepsize.
We compare the performance of DecSPS against the classical decreasing SGD stepsize ⌘ / p k + 1 , which guarantees convergence to the exact solution at the same asymptotic rate as DecSPS.
We show that, while the asymptotics are the same, DecSPS with hyperparameters c 0 = 1 ,  b = 10 performs competitively to a ﬁne-tuned ⌘ — where crucially the optimal value of ⌘ depends on the problem.
This behavior is shown on all the considered datasets, and is reported in Figures 4 ( Breast and Synthetic reported in the appendix for space constraints).
If lower regularization ( 1 e  4 , 1 e  6 ) is considered, then DecSPS can still match the performance of tuned SGD — but further tuning is needed (see Figure 14.
Speciﬁcally, since the non-regularized problems do not have strong curvature, we found that DecSPS works best with a much higher  b parameter and c 0 = 0 .
DecSPS yields a truly adaptive stepsize.
We inspect the value of  k returned by DecSPS, shown in Figures 4 & 8 (in the appendix).
Compared to the vanilla SGD stepsize ⌘ / p k + 1 , a crucial difference appears:  k decreases faster than O (1 / p k ) .
This showcases that, while the factor p k + can be found in the formula of DecSPS 6 , the algorithm structure provides additional adaptation to curvature.
Indeed, in (regularized) logistic regression, the local gradient Lipschitz constant increases as we approach the solution.
Since the optimal stepsize for steadily-decreasing SGD is 1 / ( L p k + 1) , where L is the global Lipschitz constant [13], it is pretty clear that ⌘ should be decreased over training for optimal converge (as L effectively increases).
This is precisely what DecSPS is doing.
Comparison with AdaGrad stepsizes.
Last, we compare DecSPS with another adaptive coordinateindependent stepsize with strong theoretical guarantees: the norm version of AdaGrad (a.k.a.
AdaGrad-Norm, AdaNorm), which guarantees the exact solution at the same asymptotic rate as
DecSPS [32].
AdaGrad-norm at each iteration updates the scalar b 2 k +
=
b 2 k + kr f S k
( x k ) k {+
 and then+} selects the next step as x k
{++
+} =
x k  ⌘ b k +1 r
f
i ( x k ) .
Hence, it has tuning parameters b 0 and ⌘ .
In Fig.
4 we show that, on the Breast Cancer dataset, after ﬁxing b 0 = 0 . 1 as recommended in [32] (see their Figure 3), tuning ⌘ cannot quite match the performance of DecSPS.
This behavior is also observed on the other two datasets we consider (see Fig.
9 in the Appendix).
Last, in Fig.
10&in the Appendix, we show that further tuning of b 0 likely does not yield a substantial improvement.
[Figure]
Comparison with Adam and AMSgrad without momentum.
In Figures 5&12&13 we compare DecSPS with Adam [19] and AMSgrad [26] on the A1A and Breast Cancer datasets.
Results show that DecSPS with the usual hyperparameters is comparable to the ﬁne-tuned version of both these algorithms — which however do not enjoy convergence guarantees in the unbounded domain setting. [Figure]
Conclusions and Future Work
We provided a practical variant of SPS [22] which converges to the true problem solution without the interpolation assumption in convex stochastic problems — matching the rate of AdaGrad.
If in addition strong convexity as assumed, then we show how, in contrast to current results for AdaGrad, the bounded iterates assumption can be dropped.
The main open direction is a proof of a faster rate
O (1 /K ) under strong convexity.
There are also other possible extensions of our work: using the proposed new variants of SPS with accelerated methods, studying further the effect of mini-batching and non-uniform sampling of DecSPS, and extensions to the distributed and decentralized settings. Acknowledgements
This work was partially supported by the Canada CIFAR AI Chair Program.
Simon Lacoste-Julien is a CIFAR Associate Fellow in the Learning in Machines & Brains program.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[No] Only public datasets used.
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Introduction and Conclusion (c) Did you discuss any potential negative societal impacts of your work?
[No] Theoretical work (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] Stated clearly in the main text at the corresponding theorem statements (b) Did you include complete proofs of all theoretical results?
[Yes]
In the appendix 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Discussed in the experimental section.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] In the Experiments section (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] All experiments are run at least 3 times (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] Experiments on convex optimization problems verifying the theory (run on a laptop).
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]