Dynamics of SGD with Stochastic Polyak Stepsizes: Truly Adaptive Variants and Convergence to Exact Solution
Abstract
Recently Loizou et al.
(2021), proposed and analyzed stochastic gradient descent(SGD) with stochastic Polyak stepsize (SPS).
The proposed SPS comes with strongconvergence guarantees and competitive performance; however, it has two maindrawbacks when it is used in non-over-parameterized regimes: (i) It requires apriori knowledge of the optimal mini-batch losses, which are not available whenthe interpolation condition is not satisﬁed (e.g., regularized objectives), and (ii) itguarantees convergence only to a neighborhood of the solution.
In this work, westudy the dynamics and the convergence properties of SGD equipped with newvariants of the stochastic Polyak stepsize and provide solutions to both drawbacksof the original SPS.
We ﬁrst show that a simple modiﬁcation of the original SPSthat uses lower bounds instead of the optimal function values can directly solveissue (i).
On the other hand, solving issue (ii) turns out to be more challengingand leads us to valuable insights into the method’s behavior.
We show that ifinterpolation is not satisﬁed, the correlation between SPS and stochastic gradientsintroduces a bias, which effectively distorts the expectation of the gradient signalnear minimizers, leading to non-convergence - even if the stepsize is scaled downduring training.
To ﬁx this issue, we propose DecSPS, a novel modiﬁcation of SPS,which guarantees convergence to the exact minimizer - without a priori knowledgeof the problem parameters.
For strongly-convex optimization problems, DecSPSis the ﬁrst stochastic adaptive optimization method that converges to the exactsolution without restrictive assumptions like bounded iterates/gradients.
Introduction
We consider the stochastic optimization problem:
[Equation]
where each f i is convex and lower bounded.
We denote by X ⇤ the non-empty set of optimal pointsx ⇤ of equation (1).
We set f ⇤ := min x 2 R d f ( x ) , and f ⇤ i :=
inf x 2 R d f i ( x ) .
In this setting, the algorithm of choice is often Stochastic Gradient Descent (SGD), i.e.x k {++
+} = x
k   k r f S k ( x k ) , where  k > 0 is the stepsize at iteration k , S k ✓ [ n ] a randomsubset of datapoints (minibatch) with cardinality B sampled independently at each iteration k , andr f S k ( x ) := 1 B P i 2 S k r f i ( x ) is the minibatch gradient.
A careful choice of  k is crucial for most applications [4, 14].
The simplest option is to pick  k to beconstant over training, with its value inversely proportional to the Lipschitz constant of the gradient.
While this choice yields fast convergence to the neighborhood of a minimizer, two main problemsarise: (a) the optimal  depends on (often unknown) problem parameters — hence often requiresheavy tuning ; and (b) it cannot be guaranteed that X ⇤ is reached in the limit [13, 16, 15].
A simpleﬁx for the last problem is to allow polynomially decreasing stepsizes (second option) [23]: this choicefor  k often leads to convergence to X ⇤ , but hurts the overall algorithm speed.
The third option,which became very popular with the rise of deep learning, is to implement an adaptive stepsize.
These methods do not commit to a ﬁxed schedule, but instead use the optimization statistics (e.g.gradient history, cost history) to tune the value of  k at each iteration.
These stepsizes are known towork very well in deep learning [35], and include Adam [19], Adagrad [11], and RMSprop [29].
Ideally, a theoretically grounded adaptive method should yield fast convergence to X ⇤ withoutknowledge of problem dependent parameters, such as the gradient Lipshitz constant or the strongconvexity constant.
As a result, an ideal adaptive method should require very little tuning by theuser, while matching the performance of a ﬁne-tuned  k .
However, while in practice this is the casefor common adaptive methods such as Adam and AdaGrad, the associated convergence rates oftenrely on strong assumptions — e.g. that the iterates live on a bounded domain, or that gradients areuniformly bounded in norm [11, 32, 31].
While the above assumptions are valid in the constrainedsetting, they are problematic for problems deﬁned in the whole R d .
A promising new direction in the adaptive stepsizes literature is based on the idea of Polyak stepsizes,introduced by [25] in the context of deterministic convex optimization.
Recently [22] successfullyadapted Polyak stepsizes to the stochastic setting, and provided convergence rates matching ﬁnetuned SGD — while the algorithm does not require knowledge of the unknown quantities such as thegradient Lipschitz constant.
The results especially shines in the overparameterized strongly convexsetting, where linear convergence to x ⇤ is shown.
This result is especially important since, underthe same assumption, no such rate exists for AdaGrad (see e.g. [31] for the latest results) or otheradaptive stepsizes.
Moreover, the method was shown to work surprisingly well on deep learningproblems, without requiring heavy tuning [22].
Even if the stochastic Polyak stepsize (SPS) [22] comes with strong convergence guarantees, it has twomain drawbacks when it is used in non-over-parameterized regimes: (i)
It requires a priori knowledgeof the optimal mini-batch losses, which are not often available for big batch sizes or regularizedobjectives (see discussion in §1.1) and (ii) it guarantees convergence only to a neighborhood of thesolution.
In this work, we study the dynamics and the convergence properties of SGD equipped withnew variants of SPS for solving general convex optimization problems.
Our new proposed variantsprovide solutions to both drawbacks of the original SPS.
1.1 Background and Technical Preliminaries
The stepsize proposed by [22] is
[Equation]
where  b , c > 0 are problem-independent constants, f S k :
=
|S k | P i 2 S
k f i , f ⇤ S k = inf x 2 R d f S k ( x ) .
Dependency on f S k .
Crucially the algorithm requires knowledge of f ⇤ S k for every realization ofthe mini-batch S k .
In the non-regularized overparametrized setting (e.g. neural networks), f S k isoften zero for every subset S [34].
However, this is not the only setting where f ⇤ S is computable:e.g., in the regularized logistic loss with batch size 1, it is possible to recover a cheap closed formexpression for each f ⇤ i [22].
Unfortunately, if the batch-size is bigger than 1 or the loss becomesmore demanding (e.g. cross-entropy), then no such closed-form computation is possible .
Rates and comparison with AdaGrad.
In the convex overparametrized setting (more precisely,under the interpolation condition, i.e. 9 x ⇤ 2 X ⇤ : inf x 2 R d f S ( x ) =
f S ( x ⇤ ) for all S , see also §2),
SPS max enjoys a convergence speed of O (1 /k ) , without requiring knowledge of the gradient Lipschitzconstant or other problem parameters.
Recently, [31] showed that the same rate can be achieved for
AdaGrad in the same setting.
However, there is an important difference: the rate of [31] is technically
O ( dD 2 /k ) , where d is the problem dimension and D 2 is a global bound on the squared distanceto the minimizer, which is assumed to be ﬁnite.
Not only does SPS max not have this dimensiondependency, which dates back to crucial arguments in the AdaGrad literature [11, 21], but also doesnot require bounded iterates.
While this assumption is satisﬁed in the constrained setting, it hasno reason to hold in the unconstrained scenario.
Unfortunately, this is a common problem of all
AdaGrad variants: with the exception of [33] (which works in a slightly different scenario), no ratecan be provided in the stochastic setting without the bounded iterates/gradients [12] assumption —even after assuming strong convexity.
However, in the non-interpolated setting, AdaGrad enjoys aconvergence guarantee of O (1 / p k ) (with the bounded iterates assumption).
A similar rate does notyet exist for SPS, and our work aims at ﬁlling this gap.
1.2 Main Contributions
As we already mentioned, in the non-interpolated setting SPS max has the following issues:
Issue (1) :
For B > 1 (minibatch setting), SPS max requires the exact knowledge of f ⇤ S .
This is notpractical.
Issue (2) :
SPS max guarantees convergence to a neighborhood of the solution.
It is not clear how tomodify it to yield convergence to the exact minimizer.
Having the above two issues in mind, the main contributions of our work (see also Table 1 in theappendix, for a summary of the main complexity results obtained in this paper) are summarized asfollows:
• In §3, we provide a direct solution for Issue (1).
We explain how only a lower bound on f ⇤ S (trivialif all f i s are non-negative) is required for convergence to a neighborhood of the solution.
Whilethis neighborhood is bigger that the one for SPS max , our modiﬁed version provides a practicalbaseline for the solution to the second issue.
We explain why Issue (2) is highly non-trivial and requires an in-depth study of the bias induced bythe interaction between gradients and Polyak stepsizes.
Namely, we show that simply multiplyingthe stepsize of SPS max by 1 / p k — which would work for vanilla SGD [23] — yields a bias in thesolution found by SPS (§4), regardless of the estimation of f ⇤ S
• In §5, we provide a solution to the problem (Issue (2)) by introducing additional structure — as wellas the ﬁx to Issue (1) — into the stepsize.
We call the new algorithm Decreasing SPS (DecSPS),and provide a convergence guarantee under the bounded domain assumption — matching thestandard AdaGrad results.
• In §5.2 we go one step further and show that, if strong convexity is assumed, iterates are bounded
with probability 1 and hence we can remove the bounded iterates assumption.
To the best of ourknowledge, DecSPS, is the ﬁrst stochastic adaptive optimization method that converges to the exactsolution without assuming strong assumptions like bounded iterates/gradients.
• In §5.3 we provide extensions of our approach to the non-smooth setting.
• In §6, we corroborate our theoretical results with experimental testing.
Background on Stochastic Polyak Stepsize
In this section, we provide a concise overview of the results in [22], and highlight the main assumptions and open questions
To start, we remind the reader that problem (1) is said to be interpolated if there exists a problemsolution x
⇤ 2 X ⇤ such that inf x 2 R d f i ( x ) =
f i ( x ⇤ ) for all i 2 [ n ] .
The degree of interpolationat batch size B can be quantiﬁed by the following quantity, introduced by [22] and studied alsoin [31, 9]: ﬁx a batch size B , and let S ✓ [ n ] with |S| = B .
[Equation]
It is easy to realize that as soon as problem (1) is interpolated, then  2 B = 0 for each B  n . Inaddition, note that  2 B is non-increasing as a function of B .
We now comment on the main result from [22].
Theorem 2.1 (Main result of [22]) .
Let each f i be L i -smooth convex functions.
Then SGD with
SPS max , mini-batch size B , and c = 1 , converges as: E ⇥ f (
[Equation] )  f ( x [Equation] ⇤
[Equation] K + [Equation]
2 B ↵ ,where ↵ = min n 1 2 cL max ,  b o and ¯ x K = 1 K
P
K  1 k =0 x k .
If in addition f is µ -strongly convex, then,for any c  1 / 2 , SGD with SPS max converges as: E k x k  x ⇤ k 2  (1  µ ↵ ) k
k x 0  x ⇤ k 2 + 2  b  2 B µ ↵ ,where again ↵ = min { 1 2 cL max ,  b } and L max = max { L i } ni =1 is the maximum smoothness constant.
In the overparametrized setting, the result guarantees convergence to the exact minimizer, withoutknowledge of the gradient Lipschitz constant (as vanilla SGD would instead require) and withoutassuming bounded iterates (in contrast to [31]).
As soon as (1) a regularizer is applied to the loss (e.g. L 2 penalty), or (2) the number of datapointsgets comparable to the dimension, then the problem is not interpolated and SPS max only convergesto a neighborhood
and it gets impractical to compute f ⇤ S — this is the setting we study in this paper.
Remark 1 (What if kr f S k k = 0 ?) .
In the rare case that kr f S k ( x k ) k 2 = 0 , there is no need toevaluate the stepsize.
In this scenario, the update direction r f S k ( x k ) = 0 and thus the iterate isnot updated irrespective of the choice of step-size.
If this happens, the user should simply sample adifferent minibatch.
We note that in our experiments (see §6), such event never occurred.
Related work on Polyak stepsize: The classical Polyak stepsize [25] has been successfully usedin the analysis of deterministic subgradient methods in different settings [5, 7, 18].
First attemptson providing an efﬁcient variant of the stepsize that works well in the stochastic setting were madein [3, 24].
However, as explained in [22], none of these approaches provide a natural stochasticextension with strong theoretical convergence guarantees, and thus Loizou et al.
[22] proposed thestochastic Polyak stepsize SPS max as a better alternative.
Despite its recent appearance, SPS max hasalready been used and analyzed as a stepsize for SGD for solving structured non-convex problems[15], in combination with other adaptive methods [31], with a moving target [17] and in the updaterule of stochastic mirror descent [9].
These extensions are orthogonal to our approach, and wespeculate that our proposed variants can also be used in the above settings.
We leave such extensionsfor future work.
Removing f ⇤ S from SPS
As motivated in the last sections, computing f ⇤ S in the non-interpolated setting is not practical.
In thissection, we explore the effect of using a lower bound ` ⇤ S  f ⇤ S instead in the SPS max deﬁnition.
[Equation]
Such a lower bound is easy to get for many problems of interest: indeed, for standard regularizedregression and classiﬁcation tasks, the loss is non-negative hence one can pick ` ⇤ S = 0 , for any
S ✓ [ n ] .
The obvious question is: what is the effect of estimating ` ⇤ S on the convergence rates in Thm. 2.1?
We found that the proof of [22] is easy to adapt to this case, by using the following fundamentalbound (see also Lemma B.2):
[Equation]
The following results can be seen as an easy extension of the main result of [22], under a newlydeﬁned suboptimality measure:
[Equation]
Theorem 3.1.
Under SPS ` max , the same exact rates in Thm.
2.1 hold (under the correspondingassumptions), after replacing  2 B with ˆ  2 B .
And we also have an easy practical corollary.
Corollary 3.2.
In the context of Thm. 3.1, assume all f i s are non-negative and estimate ` ⇤ S
= 0for all S ✓ [ n ] .
Then the same exact rates in Thm.
2.1 hold for SPS ` max , after replacing  2 B withf ⇤ =
f ( x ⇤ ) .
A numerical illustration of this result can be found in Fig.
In essence, both theory and experimentsconﬁrm that, if interpolation is not satisﬁed, then we have a rate until a convergence ball, wherethe size is optimal under exact knowledge of f ⇤ S .
Instead, under interpolation, if all the f i s arenon-negative and f ⇤ = 0 , then SPS max =SPS ` max .
Finally, in the less common case in practicewhere f ⇤ > {+
+} but we still have interpolation, then SPS max converges to the exact solution while
SPS ` max does not.
To conclude SPS ` max does not (of course) work better than SPS max , but it is apractical variant which we can use as a baseline in §5 for an adaptive stochastic Polyak stepsize withconvergence to the true x ⇤ in the non-interpolated setting.
Bias in the SPS dynamics.
In this section, we study convergence of the standard SPS max in the non-interpolated regime, underan additional (decreasing) multiplicative factor, in the most ideal setting: batch size 1, and we haveknowledge of each f
⇤ i . That is, we consider  k = min { f ik ( x k )  f
⇤ ik c k kr f
ik ( x k ) k 2 ,  b } with c k ! 1 , e.g.c k = O ( p k ) or c k = O ( k ) .
We note that, in the SGD case, simply picking e.g.  k =  0 / p k
+ 1would guarantee convergence of f ( x k ) to f ( x ⇤ ) , in expectation and with high probability [20, 23].
Therefore, it is natural to expect a similar behavior for SPS, if 1 /c k saﬁsﬁes the usual Robbins-Monroconditions [27]:
[Equation]
We show that this is not the case : quite interestingly, f ( x k ) converges to a biased solution due to thecorrelation between r f i k and  k .
we show this formally, in the case of non-interpolation (otherwiseboth SGD and SPS do not require a decreasing learning rate).
Counterexample.
Consider the following ﬁnite-sum setting: f ( x ) = 12 f 1 ( x ) + 12 f 2 ( x ) withf 1 ( x ) =
a 1 2 ( x  1) 2 , f 2 ( x ) = a 2 2 ( x + 1) 2 .
To make the problem interesting, we choose a 1 = 2and a 2 = 1 : this introduces asymmetry in the average landscape with respect to the origin.
Duringoptimization, we sample f 1 and f 2 independently and seek convergence to the unique minimizerx ⇤ =
a 1  a 2 a 1 + a 2 = 1 / 3 .
The ﬁrst thing we notice is that x ⇤ is not a stationary point for the dynamicsunder SPS.
Indeed note that since f ⇤ i = 0 for i = 1 , 2 we have (assuming  b large enough): k r f i k x ) = x  1 2 c k , if i k = 1 , and  k r f i k
( x ) =
x +1
c k if i k
= 2 .
Crucially, note that this update is curvature-independent .
The expected update is E i k
[  k r f i k ( x )]
=x  1 4 c k + x +14 c k =
c k x .
Hence, the iterates can only converge to x = 0 — because this is the only ﬁxedpoint for the update rule.
The proof naturally extends to the multidimensional setting, an illustrationcan be found in Fig.
In the same picture, we show how our modiﬁedvariant of the vanilla stepsize — we call this newalgorithm DecSPS, see §5 — instead converges tothe correct solution.
Remark 2 .
SGD with (non-adaptive) stepsize  kinstead keeps the curvature, and therefore is able tocorrectly estimate the average E i
k [  k r f i k ( x )] =
[Equation]
i — precisely because k is independent from r f i k .
From this we cansee that SGD can only converge to the correctstationary point x ⇤ = a 1  a 2 a 1 + a 2 — because againthis is the only ﬁxed point for the update rule.
In the appendix, we go one step further and provide an analysis of the bias of SPS in the onedimensional quadratic case (Prop. D.2).
Yet, weexpect the precise characterization of the bias phenomenon in the non-quadratic setting to be particularly challenging.
We provide additional insightsin §D.2.
Instead, in the next section, we show howto effectively modify  k to yield convergence tox ⇤ without further assumptions.
DecSPS: Convergence to the exact solution
We propose the following modiﬁcation of the vanilla SPS proposed in [22], designed to yieldconvergence to the exact minimizer while keeping the main adaptiveness properties 2 .
We call it
Decreasing SPS (DecSPS), since it combines a steady stepsize decrease with the adaptiveness of SPS.
[Equation]
for k 2 N , where c k 6 = 0 for every k 2 N .
We set c
1 = c 0 and   1 =  b > 0 (stepsize bound,similar to [22]), to get [-γ 
-] 0 :
= 1 c 0 · min
[Equation]
Lemma 5.1.
Let each f i be L
i smooth and let ( c k ) 1 k =0 be any non-decreasing positive sequence ofreal numbers.
Under DecSPS, we have min
[Equation]
Remark 3 .
As stated in the last lemma, under the assumption of c k non-decreasing,  k is triviallynon-increasing since  k  c k  1  k  1 /c k .
The proof can be found in the appendix, and is based on a simple induction argument.
5.1 Convergence under bounded iterates
The following result provides a proof of convergence of SGD for the  k sequence deﬁned above.
Theorem 5.2.
Consider SGD with DecSPS and let ( c k ) 1 k =0 be any non-decreasing sequence suchthat c k  1 , 8 k 2 N .
Assume that each f i is convex and L i smooth.
We have:
[Equation]
where
[Equation]
If ˆ  2 B = 0 , then c k = 1 for all k 2 N leads to a rate O ( 1 K ) , well known from [22].
If ˆ  2 B > 0 , as forthe standard SGD analysis under decreasing stepsizes, the choice c k = O ( p k ) leads to an optimalasymptotic trade-off between the deterministic and the stochastic terms, hence to the asymptoticrate
O (1 / p k ) since P K  1 k = {+
 p+} k +1  2 p K .
Moreover, picking c 0 = 1 minimizes the speed ofconvergence for the deterministic factor.
Under the assumption that ˆ  2 B ⌧ ˜ LD 2 (e.g. reasonabledistance initialization-solution and L max > 1 /  b ), this factor is dominant compared to the factorinvolving ˆ  2 B .
For this setting, the rate simpliﬁes as follows.
Corollary 5.3.
Under the setting of Thm.
5.2, for c k = p k + 1 ( c  1 = c 0 ) we have
[Equation]
Remark 4 (Beyond bounded iterates) .
The result above crucially relies on the bounded iteratesassumption: D 2 < 1 .
To the best of our knowledge, if no further regularity is assumed, modernconvergence results for adaptive methods (e.g. variants of AdaGrad) in convex stochastic programming require 3 this assumption, or else require gradients to be globally bounded.
To mention a few:[11, 26, 32, 8, 31].
A simple algorithmic ﬁx to this problem is adding a cheap projection step ontoa large bounded domain [21].
We can of course include this projection step in DecSPS, and thetheorem above will hold with no further modiﬁcation.
Yet we found this to be not necessary: thestrong guarantees of SPS in the strongly convex setting [22] let us go one step beyond: in §5.2 weshow that, if each f i is strongly convex (e.g. regularizer is added), then one can bound the iteratesglobally with probability one, without knowledge of the gradient Lipschitz constant.
To the best ofour knowledge, no such result exist for AdaGrad — except [30], for the deterministic case.
Remark 5 (Dependency on the problem dimension) .
In standard results for AdaGrad, a dependencyon the problem dimension often appears (e.g. Thm. 1 in [31]).
This dependency follows from a boundon the AdaGrad preconditioner that can be found e.g. in Thm.
4 in [21].
In the SPS case no suchdependency appears — speciﬁcally because the stepsize is lower bounded by 1 / (2 c k L max ) .
5.2 Removing the bounded iterates assumption
We prove that under DecSPS the iterates live in a set of diameter D max almost surely.
This can bedone by assuming strong convexity of each f i .
The result uses this alternative deﬁnition of neighborhood : ˆ  2 B, max := max S ✓ [ n ] , |S| = B [ f S ( x ⇤ )  ` ⇤ S ] .
Note that trivially ˆ  2 B, max < 1 under the assumption that all f i are lower bounded and n < 1 .
Proposition 5.4.
Let each f i be µ
i -strongly convex and L i -smooth.
The iterates of SGD with
DecSPS with c k = p k + 1 (and c  1 = c 0 ) are such that k x k  x ⇤ k 2  D 2max almost surely8 k 2 N , where D 2max := max
[Equation]
b }  , with µ min = min i 2 [ n ] µ i and
L max = max i 2 [ n ] L i .
The proof relies on the variations of constants formula and an induction argument — it is providedin the appendix.
We are now ready to state the main theorem for the unconstrained setting, whichfollows from Prop. 5.4 and Thm. 5.2.
Theorem 5.5.
Consider SGD with the DecSPS stepsize  k :=
[Equation]
for k  1 and  0 deﬁned as at the beginning of this section.
Let each f i be µ
i -strongly convex and
L i -smooth:
[Equation]
Remark 6 .
The careful reader might notice that, while we assumed strong convexity, our rate is slowerthan the optimal O (1 /K ) .
This issue is due to the adaptive nature of DecSPS.
{+It is indeed notoriously
+}
Perhaps the only exception is the result of [32], where the authors work on a different setting: i.e. they introduce the RUIG inequality.
hard to achieve a convergence rate of O (1 /K ) for adaptive methods in the strongly convex regime.
While further investigations will for sure shed light on this interesting problem, we note that theresult we provide is somewhat unique in the literature : we are not aware of any adaptive methodthat enjoys such a convergence rate without either (a) assuming bounded iterates/gradients or (b)assuming knowledge of the gradient Lipschitz constant and or the strong convexity constant.
5.3 Extension to the non-smooth setting
For any S ✓ [ n ] , we denote in this section by g S ( x ) the subgradient of f S evaluated at x .
We discussthe extension of DecSPS to the non-smooth setting.
A straightforward application of DecSPS leads to a stepsize  k which is no longer lower bounded (see
Lemma 5.1) by the positive quantity min
[Equation]
o . Indeed, the gradient Lipschitz constantin the non-smooth case is formally L max = 1 .
Hence,  k prescribed by DecSPS can get arbitrarilysmall 4 for ﬁnite k .
One easy solution to the problem is to enforce a lower bound, and adopt a newproof technique.
Speciﬁcally we propose the following:
[Equation]
where c k 6 = 0 for every k  0 ,  `   b is a small positive number and all the other quantities aredeﬁned as in DecSPS.
In particular, as for DecSPS, we set c
= c 0 and   1 =  b .
Intuitively,  k isselected to live in the interval [ c 0  ` /c k , c 0  b /c
k ] (see proof in §F, appendix), but has subgradientdependent adaptive value.
In addition, this stepsize is enforced to be monotonically decreasing.
Theorem 5.6.
For any non-decreasing positive sequence ( c k ) 1 k =0 , consider SGD with DecSPS-NS.
Assume that each f i is convex and lower bounded.
We have
[Equation]
where D 2 := max k 2 [ K  1] k x k  x ⇤ k 2 and G 2 := max k 2 [ K  1] k g S k ( x k ) k 2 .
One can then easily derive an O (1 / p k ) convergence rate.
This is presented in §F (appendix).
Numerical Evaluation
We evaluate the performance of DecSPS with c k = c {+
 p+} k + 1 on binary classiﬁcation tasks, withregularized logistic loss f ( x ) = 1 n
[Equation]
2 , where a i 2 R d is thefeature vector for the i -th datapoint and y i 2 {  1 , 1 } is the corresponding binary target.
We study performance on three datasets: (1) a Synthetic Dataset, (2)
The A1A dataset [6] and (3)
the
Breast Cancer dataset [10].
We choose different regularization levels and batch sizes bigger than 1.
Details are reported in §G, and the code is available in the supplementary material.
At the batch sizes and regularizer levels we choose, the problems do not satisfy interpolation.
Indeed,running full batch gradient descent yields f ⇤ > 0 .
While running SPS max on these problems (1) doesnot guarantee convergence to f ⇤ and (2) requires full knowledge of the set of optimal function values { f ⇤ S } |S| = B , in DecSPS we can simply pick the lower bound 0 =
` ⇤ S  f ⇤ S for every S .
Supported by 310 Theorems 5.2 & 5.5 & 5.6, we expect SGD with DecSPS to converge to the minimum f ⇤ .
Stability of DecSPS.
DecSPS has two hyperparameters: the upper bound  b on the ﬁrst stepsizeand the scaling constant c 0 .
While Thm.
5.6 guarantees convergence for any positive value of thesehyperparameters, the result of Thm.
5.2 suggests that using c 0 = 1 yields the best performance underthe assumption that ˆ  2 B ⌧ ˜ LD 2 (e.g. reasonable distance of initialization from the solution, and L max > 1 /  b ).
In Fig.
3, we show on the synthetic dataset that (1) {+c 
 = 1+} is indeed the best choicein this setting and (2) the performance of SGD with DecSPS is almost independent of  b .
Similar ﬁndings are reported and commented in Figure 9 (Appendix) for the other datasets.
Hence, for allfurther experiments , we choose the hyperparameters  b = 10 , c 0 = 1 .
Comparison with vanilla SGD with decreasing stepsize.
We compare the performance of DecSPS againstthe classical decreasing SGD stepsize ⌘ / p k + 1 ,which guarantees convergence to the exact solutionat the same asymptotic rate as DecSPS.
We show that,while the asymptotics are the same, DecSPS with thetheoretically inspired hyperparameters c 0 = 1 ,  b = 10performs competitively to a ﬁne-tuned ⌘ — where crucially the optimal value of ⌘ depends on the problem.
This behavior is shown on all the considered datasets,and is reported in Figures 4 ( Breast and Synthetic reported in the appendix for space constraints).
DecSPS yields a truly adaptive stepsize.
We inspect the value of  k returned by DecSPS, shownin Figures 4 & 10 (in the appendix).
Compared to the vanilla SGD stepsize ⌘ / p k + 1 , a crucialdifference appears:  k decreases faster than O (1 / p k ) .
This showcases that, while the factor p k + 1can be found in the formula of DecSPS 5 , the algorithm structure provides additional adaptation tocurvature.
Indeed, in (regularized) logistic regression, the local gradient Lipschitz constant increasesas we approach the solution.
Since the optimal stepsize for steadily-decreasing SGD is 1 / ( L p k + 1) ,where L is the global Lipschitz constant [13], it is pretty clear that ⌘ should be decreased over trainingfor optimal converge (as L effectively increases).
This is precisely what DecSPS is doing.
Comparison with AdaGrad stepsizes.
Last, we compare DecSPS with another adaptive coordinateindependent stepsize with strong theoretical guarantees: the norm version of AdaGrad (a.k.a.
AdaGrad-Norm, AdaNorm), which guarantees the exact solution at the same asymptotic rate as
DecSPS [32].
AdaGrad-norm at each iteration updates the scalar b 2 k +
=
b 2 k + kr f S k
( x k ) k 2 andthen selects the next step as x k
+1 =
x k  ⌘ b k +1 r f i ( x k ) .
Hence, it has tuning parameters b 0 and⌘ .
In Fig.
4 we show that, on the Breast Cancer dataset, after ﬁxing b 0 = 0 . 1 as recommendedin [32] (see their Figure 3), tuning ⌘ cannot quite match the performance of DecSPS.
This behavior isalso observed on the other two datasets we consider (see Fig. 12 in the Appendix).
Last, in Fig.
11 inthe Appendix, we show that further tuning of b 0 likely does not yield a substantial improvement.
[Figure]
Comparison with Adam and AMSgrad.
In Figures 13&14 we compare DecSPS with Adam [19]and AMSgrad [26] on the A1A and Breast Cancer datasets.
The results show that DecSPS with theusual hyperparameters is comparable to the ﬁne-tuned version of both these algorithms — whichhowever do not enjoy comparable convergence guarantees in the unbounded domain setting.
Conclusions and Future Work
We provided a practical variant of SPS [22] which converges to the true problem solution withoutthe interpolation assumption in convex stochastic problems — matching the rate of AdaGrad.
If in5 We pick c k = c 0 p k + 1 , as suggested by Cor.
5.3 & F. addition strong convexity as assumed, then we show how, in contrast to current results for AdaGrad,the bounded iterates assumption can be dropped.
The main open direction is a proof of a faster rate
O (1 /K ) under strong convexity.
There are also other possible extensions of our work: using theproposed new variants of SPS with accelerated methods, studying further the effect of mini-batchingand non-uniform sampling of DecSPS, and extensions to the distributed and decentralized settings.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[No] Only public datasets used.•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Introduction and Conclusion
(c) Did you discuss any potential negative societal impacts of your work?
[No] Theoretical work
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Stated clearly
in the main text at the corresponding theorem statements
(b) Did you include complete proofs of all theoretical results?
[Yes]
In the appendix
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] Discussed in
the experimental section.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
In the Experiments section
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes] All experiments are run at least 3 times
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] Experiments on convex optimization
problems verifying the theory (run on a laptop).
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A](b)
Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
[Figure]
[Figure]
[Figure]