C ONTINUIZED A CCELERATION FOR Q UASAR C ONVEX F UNCTIONS IN N ON -C ONVEX O PTIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Quasar convexity is a condition that allows some ﬁrst-order methods to efﬁciently minimize a function even when the optimization landscape is non-convex.
Previous works develop near-optimal accelerated algorithms for minimizing this class of functions, however, they require a subroutine of binary search which results in multiple calls to gradient evaluations in each iteration, and consequently the total number of gradient evaluations does not match a known lower bound.
In this work, we show that a recently proposed continuized Nesterov acceleration can be applied to minimizing quasar convex functions and achieves the optimal bound with a high probability.
Furthermore, we ﬁnd that the objective functions of training generalized linear models (GLMs) satisfy quasar convexity, which broadens the applicability of the relevant algorithms, while known practical examples of quasar convexity in non-convex learning are sparse in the literature.
We also show that if a smooth and one-point strongly convex, Polyak-Łojasiewicz, or quadratic-growth function satisﬁes quasar convexity, then attaining an accelerated linear rate for minimizing the function is possible under certain conditions, while acceleration is not known in general for these classes of functions.
I NTRODUCTION
Momentum has been the main workhorse for training machine learning models (Kingma & Ba, 2015;
Wilson et al., 2017; Loshchilov & Hutter, 2019; Reddi et al., 2018;
He et al., 2016; Simonyan &
Zisserman, 2015; Krizhevsky et al., 2012).
In convex learning and optimization, several momentum methods have been developed under different machineries, which include the ones built on Nesterov’s estimate sequence (Nesterov, 1983; 2013), methods derived from ordinary differential equations and continuous-time techniques, (Krichene et al., 2015; Scieur et al., 2017; Attouch et al., 2018;
Su et al., 2014; Wibisono et al., 2016; Shi et al., 2018; Diakonikolas & Orecchia, 2019), approaches based on dynamical systems and control (Hu & Lessard, 2017; Wilson et al., 2021), algorithms generated from playing a two-player zero-sum game via no-regret learning strategies (Wang et al., 2021a; Wang &
Abernethy, 2018; Cohen et al., 2021), and a recently introduced continuized acceleration (Even et al., 2021).
On the other hand, in the non-convex world, despite numerous empirical evidence conﬁrms that momentum methods converge faster than gradient descent (GD) in several applications, see e.g.,
Sutskever et al.
; Leclerc & Madry (2020), ﬁrst-order accelerated methods that provably ﬁnd a global optimal point are sparse in the literature.
Indeed, there are just few results showing acceleration over GD that we are aware.
Wang et al.
(2021b) show Heavy Ball has an accelerated linear rate for training an over-parametrized ReLU network and a deep linear network, where the accelerated linear rate has a square root dependency on the condition number of a neural tangent kernel matrix at initialization, while the linear rate of GD depends linearly on the condition number.
When the goal is not ﬁnding a global optimal point but a ﬁrst-order stationary point, some beneﬁts of incorporating the dynamic of momentum can be shown (Cutkosky & Orabona, 2019; Cutkosky & Mehta, 2021; Levy et al., 2021).
Nevertheless, theoretical-grounded momentum methods in non-convex optimization are still less investigated to our knowledge.
With the goal of advancing the progress of momentum methods in non-convex optimization in mind, we study efﬁciently solving min w f ( w ) , where the function f ( · ) satisﬁes quasar-convexity (Hinder et al., 2020; Hardt et al., 2018; Nesterov et al., 2019; Guminov & Gasnikov, 2017; Bu & Mesbahi, 2020), which is deﬁned in the following.
Under quasar convexity, it can be shown that GD or certain momentum methods can globally minimize a function even when the optimization landscape is non-convex (Hinder et al., 2020; Hardt et al., 2018; Nesterov et al., 2019; Guminov & Gasnikov, 2017; Bu & Mesbahi, 2020).
Deﬁnition 1.
( Quasar convexity ) Let ρ > 0 .
Denote w ∗ a global minimizer of f ( · ) : R d → R .
The function f ( · ) is ρ -quasar convex if for all w ∈ R d , one has:
[Equation]
For µ > 0 , the function f ( · ) is ( ρ, µ ) -strongly quasar convex if for all w ∈ R d , one has:
[Equation]
For more characterizations of quasar convexity, we refer the reader to Hinder et al.
(2020) (Appendix D in the paper), where a thorough discussion is provided.
Recall that a function f ( · ) is L -smooth if
[Equation]
for any x and y , where L > 0 is the smoothness constant.
For minimizing L -smooth and ρ -quasar convex functions, the algorithm of Hinder et al.
[Equation] [Equation]
[Table]
of function and gradient evaluations for getting an (cid:15) -optimality gap.
For L -smooth and ( ρ, µ ) -strongly quasar convex functions, the algorithm of Hinder et al.
(2020) takes O (cid:18)
√
L/µ ρ log (cid:0) V(cid:15) (cid:1)(cid:19) number of iterations and O (cid:18)
√ L/µ ρ log (cid:0) V(cid:15) (cid:1) log (cid:16) L/µρ (cid:17)(cid:19) number of function and gradient evaluations, where
V :=
f ( w 0 ) − f ( w ∗ ) + µ 2 (cid:107) z 0 − w ∗ (cid:107) 2 , and w 0 and z 0 are some initial points.
Both results of
Hinder et al.
(2020) improve those in the previous works of Nesterov et al.
and Guminov & Gasnikov (2017) for minimizing quasar and strongly quasar convex functions.
A lower bound Ω (cid:16) L 1 / 2 (cid:107) w 0 − w ∗ (cid:107) ρ(cid:15) 1 /(cid:17) on the number of gradient evaluations for minimizing quasar convex functions via any ﬁrst-order deterministic methods is also established in Hinder et al.
The additional logarithmic factors in the (upper bounds of the) number of gradient evaluations, compared to the iteration complexity, result from a binary-search subroutine that is executed in each iteration to determine the value of a speciﬁc parameter of the algorithm.
1 A similar concern applies to Bu &
Mesbahi (2020), where the algorithm assumes an oracle is available but its implementation needs a subroutine which demands multiple function and gradient evaluations in each iteration.
Hence, the open questions are whether the additional logarithmic factors in the total number of gradient evaluations can be removed and whether function evaluations are necessary for an accelerated method to minimize quasar convex functions.
We answer them by showing an accelerated randomized algorithm that avoids the subroutine, makes only one gradient call per iteration, and does not need function evaluations.
Consequently, the complexity of gradient calls does not incur the additional logarithmic factors as the previous works, and, perhaps more importantly, the computational cost per iteration is signiﬁcantly reduced.
The proposed algorithms are built on the continuized discretization technique that is recently introduced by Even et al.
(2021) to the optimization community, which offers a nice way to implement a continuous-time dynamic as a discrete-time algorithm.
Speciﬁcally, the technique allows one to use differential calculus to design and analyze an algorithm in continuous time, while the discretization of the continuized process does not suffer any discretization error thanks to the fact that the Poisson process can be simulated exactly.
Our acceleration results in this paper champion the approach, and provably showcase the advantage of momentum over GD for minimizing quasar convex functions.
While previous works of quasar convexity are theoretically interesting, a lingering issue is that few examples are known in non-convex machine learning.
While some synthetic functions are shown in previous works (Hinder et al., 2020; Nesterov et al., 2019; Guminov & Gasnikov, 2017), the only practical non-convex learning applications that we are aware are given by Hardt et al.
(2018), where they show that for learning a class of linear dynamical systems, a relevant objective function over a convex constraint set satisﬁes quasar convexity, and by Foster et al.
(2018), where they show that a robust linear regression with Tukey’s biweight loss and a GLM with an increasing link function satisfy quasar convexity, under the assumption that the link function has a bounded second derivative (which excludes the case of Leaky-ReLU).
In this work, we ﬁnd that the objective functions of learning GLMs with link functions being logistic, quadratic, ReLU, or Leaky-ReLU satisfy (strong) quasar convexity, under mild assumptions on the data distribution.
We also establish connections between strong quasar convexity and one-point convexity (Guille-Escuret et al., 2022;
Kleinberg et al., 2018), the Polyak-Łojasiewicz (PL) condition (Polyak, 1963; Karimi et al., 2016), and the quadratic-growth (QG) condition (Drusvyatskiy & Lewis, 2018).
Our ﬁndings suggest that investigating minimizing quasar
convex functions is not only theoretically interesting, but is also practical for certain non-convex learning applications.
To summarize, our contributions include:
• For minimizing functions satisfying quasar convexity or strong quasar convexity, we show that the continuized Nesterov acceleration not only has the optimal iteration complexity, but also makes the same number of gradient calls required to get an expected (cid:15) -optimality gap or an (cid:15) -gap with high probability.
The continuized Nesterov acceleration avoids multiple gradient calls in each iteration, in contrast to the previous works.
We also propose an accelerated algorithm that uses stochastic pseudo-gradients for learning a class of GLMs.
We ﬁnd that GLMs with various link functions satisfy quasar convexity.
Moreover, we show that if a smooth one-point convex, PL, or QG function satisﬁes quasar convexity, then acceleration for minimizing the function is possible under certain conditions, while acceleration over GD is not known for these classes of functions in general in the literature.
P RELIMINARIES
Related works of gradient-based algorithms for structured non-convex optimization:
Studying gradient-based algorithms under some relaxed notions of convexity has seen a growing interest in non-convex optimization, e.g., (Gower et al., 2021; Vaswani et al., 2019; 2022; Jin, 2020).
These variegated notions include one-point convexity (Guille-Escuret et al., 2022; Kleinberg et al., 2018), the PL condition (Polyak, 1963; Karimi et al., 2016), the QG condition (Drusvyatskiy &
Lewis, 2018), the error bound condition (Luo & Tseng, 1993; Drusvyatskiy & Lewis, 2018), local quasi convexity (Hazan et al., 2016), the regularity condition (Chi et al., 2019), variational coherence (Zhou et al., 2017), and quasar convexity (Hinder et al., 2020; Hardt et al., 2018; Nesterov et al., 2019; Guminov & Gasnikov, 2017; Bu & Mesbahi, 2020).
For more details, we refer the reader to the references therein.
The continuized technique of designing optimization algorithms:
The continuized technique was introduced in Aldous & Fill (2002) under the subject of Markov chain and was recently used in optimization by Even et al.
(2021), where they consider the following random process and build a connection to Nesterov’s acceleration (Nesterov, 1983; 2013):
[Equation]
in which η t , η (cid:48) t , γ t , γ (cid:48) t are parameters to be chosen and dN ( t ) is the Poisson point measure.
More precisely, one has dN ( t ) =
(cid:80) k ≥ 1 δ T k
( dt ) , where the random times T 1 , T 2 , . .
, T k , . .
. are such that the increments T 1 , T 2 − T 1 , T 3 − T 2 , . . .
follow i.i.d.
from the exponential distribution with mean 1 (so E [ T k ] = k ).
Between the random times, the continuized process (3) reduces to a system of ordinary differential equations:
[Equation]
At the random time T k , the dynamic (3) is equivalent to taking GD steps:
[Equation]
A nice feature of this continuized technique is that one can implement the dynamic (3) without causing any discretization error, thanks to the fact that the Poisson process can be simulated exactly.
In contrast, other continuous-time approaches (Krichene et al., 2015; Scieur et al., 2017; Attouch et al., 2018; Su et al., 2014; Wibisono et al., 2016; Shi et al., 2018; Diakonikolas & Orecchia, 2019) do not enjoy such a beneﬁt.
The formal statement of the continuized discretization is replicated as follows.
Lemma 1.
(Theorem 3 in Even et al.
The discretization of the continuized Nesterov acceleration (3) can be implemented as ˜ w k :=
w T k , ˜ v k :=
w T k +1 − , ˜
z k :=
z T k .
Furthermore, the update of the discretized process is in the following form:
[Equation]
where τ k , τ (cid:48) k , ˜ γ k , ˜ γ (cid:48) k are random parameters that are functions of η t , η (cid:
) t , γ t , and γ (cid:48) t .
We replicate the proof of Lemma 1 in Appendix B. Using the continuized technique, Even et al.
(2021) analyze the continuized Nesterov acceleration (3) for minimizing smooth convex functions and smooth strongly convex functions with an application in asynchronous distributed optimization.
M AIN RESULTS : APPLICATION ASPECTS 3. E XAMPLES OF QUASAR CONVEXITY
We start by identifying a class of functions that satisfy quasar convexity.
To get the ball rolling, we need to introduce two notions ﬁrst.
Deﬁnition 2.
( C v -generalized variational coherence w.r.t.
a function h ( · , · ) )
Denote w ∗ ∈ R d a global minimizer of a function f ( · ) .
We say that the function f ( · ) is generalized variational coherent with the parameter C v > 0 if for all w ∈ R d , one has: (cid:104)∇ f ( w ) , w − w ∗ (cid:105) ≥ C v h ( w, w ∗ ) , where h ( w, w ∗ ) :
R d × R d → R + is a non-negative function whose inputs are w and w ∗ .
Observe that if a function is generalized variational coherent, then it is variational coherent, i.e., (cid:104)∇ f ( w ) , w − w ∗ (cid:105) ≥ 0 , which is a condition that allows an almost-sure convergence to w ∗ via mirror descent (Zhou et al., 2017).
Also, when the non-negative function h ( w, w ∗ ) is a squared l 2 norm, i.e., h ( w, w ∗ )
=
(cid:107) w − w ∗ (cid:107) 22 , it becomes one-point convexity, i.e., (cid:104)∇ f ( w ) , w − w ∗ (cid:105) ≥ C v (cid:107) w − w ∗ (cid:107) 22 .
In the literature, a few non-convex learning problems have been shown to exhibit one-point convexity, see e.g., Yehudai & Shamir (2020); Sattar & Oymak (2022); Li & Yuan (2017); Kleinberg et al.
However, Guille-Escuret et al.
(2022) recently show that for minimizing the class of functions that are one-point convex w.r.t.
a global minimizer w ∗ and have gradient Lipschitzness in the sense that (cid:107)∇ f ( w ) − ∇ f ( w ∗ ) (cid:107) 2 ≤ L (cid:107) w − w ∗ (cid:107) 2 for any w ∈ R d (which is called the upper error bound condition in their terminology), GD is optimal among any ﬁrst-order methods, which suggests that a different condition than the upper error bound condition might be necessary to show acceleration over GD for functions satisfying one-point convexity.
Deﬁnition 3.
( C l -generalized smoothness w.r.t.
a function h ( · , · ) )
Denote w ∗ ∈ R d a global minimizer of a function f ( · ) .
We say that the function f ( · ) is generalized smooth with the parameter
C l > 0 if for all w ∈ R d , one has: f ( w ) − f ( w ∗ )
≤ C l h ( w, w ∗ ) , where h ( w, w ∗ ) :
R d × R d → R + is a non-negative function whose inputs are w and w ∗ .
We see that if a function f ( · ) is L -smooth w.r.t.
a norm (cid:107) · (cid:107) , then it is L 2 -generalized smooth w.r.t.
the square norm, i.e., h ( w, w ∗ ) = (cid:107) w − w ∗ (cid:107) 2 .
Lemma 2.
If f ( · ) is C v -generalized variational coherent and C l -generalized smooth w.r.t.
the same non-negative function h ( · , · ) , then the function satisﬁes ρ -quasar convexity with ρ = C v C l .
Proof.
Using the deﬁnitions, we have f ( w ) − f ( w ∗ ) ≤ C l h ( w, w ∗ ) ≤ C l C v (cid:104)∇ f ( w ) , w − w ∗ (cid:105) .
Lemma 2 could be viewed as a modiﬁed result of Lemma 5 in Foster et al.
(2018), where the authors show that a GLM with the link function having a bounded second derivative and a positive ﬁrst derivative satisﬁes quasar convexity.
In the following, we provide three more examples of quasar convexity, while the proofs are deferred to Appendix C. For these examples, we assume that each sample x ∈ R d is i.i.d.
from a distribution D , and that there exists a w ∗ ∈ R d such that its label is generated as y = σ ( w (cid:62)∗ x ) , where σ ( · ) :
R → R is the link function of a GLM.
We consider minimizing the square loss function:
[Equation]
3.1. E XAMPLE 1: (GLM S WITH INCREASING LINK FUNCTIONS )
Lemma 3.
Suppose that the link function σ ( z ) is L 0 -Lipschitz and α -increasing, i.e., σ (cid:48) ( z ) ≥ α >for all z > R .
Then, the loss function (11) is α 2 -generalized variational coherent and L 20 2 -generalized smooth w.r.t.
h ( w, w ∗ ) =
E x ∼D (cid:2) (( w − w ∗ ) (cid:62) x ) 2 (cid:3) .
Therefore, it is ρ = 2 α 2 L 20 -quasar convex.
An example of the link functions that satisfy the assumption is the Leaky-ReLU, i.e., σ ( z ) = max( αz, z ) , where α > 0 .
If we further assume that the models w , w ∗ , and the features x in (11) have ﬁnite length so that the input to the link function σ ( · ) is bounded, then the logistic link function, i.e., σ ( z ) = (1 + exp( − z )) − 1 , is another example.
3.1. E XAMPLE 2: (P HASE RETRIEVAL )
When the link function is quadratic, i.e., σ ( z ) =
z 2 , the objective function becomes that of phase retrieval, see e.g., Yonel & Yazici (2020); Chi et al.
White et al.
, Yonel & Yazici (2020) show that in the neighborhood of the global minimizers ± w ∗ , the function satisﬁes one-point convexity in terms of the l 2 norm when the data distribution D follows a Gaussian distribution, for which a specialized initialization technique called the spectral initialization ﬁnds a point in the neighborhood (Ma et al., 2020).
As discussed earlier, one-point convexity is equivalent to generalized coherence w.r.t.
the square norm, i.e., h ( w, w ∗ ) = (cid:107) w − w ∗ (cid:107) 22 .
Therefore, by Lemma 2, to show quasar convexity for all w in the neighborhood of ± w ∗ , it remains to show that the objective function is generalized smooth w.r.t.
the square norm (cid:107) w − w ∗ (cid:107) 22 .
Lemma 4.
Assume that there exists a ﬁnite constant C R > 0 such that all w ∈ R d in the balls of radius R centered at ± w ∗ satisfy E x ∼ D (cid:104)(cid:0) ( w + w ∗ ) (cid:62) x (cid:1) 2 (cid:107) x (cid:107) 22 (cid:105) ≤ C R .
Then, the loss function (11) is 12 C R -generalized smooth w.r.t.
h ( w, w ∗ ) = (cid:107) w − w ∗ (cid:107) 22 .
An example of the distribution D that satisﬁes the assumption in Lemma 4 is a Gaussian distribution.
3.1. E XAMPLE 3: (L EARNING A SINGLE R E LU)
When the link function is ReLU, i.e., σ ( z ) = max { 0 , z } , Theorem 4.2 in Yehudai & Shamir (2020) shows that under mild assumptions of the data distribution, e.g., D is a Gaussian, the objective function is one-point convex in terms of the l 2 norm.
Therefore, as the case of phase retrieval, it remains to show generalized smoothness w.r.t.
the square norm (cid:107) w − w ∗ (cid:107) 22 for showing quasar convexity.
Lemma 5.
When the link function is ReLU, the loss function (11) is 12 E x ∼ D
[ (cid:107) x (cid:107) 22 ] -generalized smooth w.r.t.
h ( w, w ∗ ) = (cid:107) w − w ∗ (cid:107) 22 .
3. E XAMPLES OF STRONG QUASAR CONVEXITY
In this subsection, we switch to investigate strong quasar convexity.
We establish its connections to one-point convexity, the PL condition, and the QG condition.
3.2. O NE - POINT CONVEX FUNCTIONS WITH QUASAR CONVEXITY
It turns out that if a C v -one-point convex function f ( · ) satisﬁes ρ -quasar convexity, then it also satisﬁes strong quasar convexity.
Speciﬁcally, we have the following lemma.
Lemma 6.
Suppose that the function f ( · ) satisﬁes C v -one-point convexity and ˆ ρ -quasar convexity.
Then, it is also (cid:16)
[Equation]
-strongly quasar convex for any θ > 1 .
The proof is deferred to Appendix C.4.
By Lemma 6, phase retrieval and ReLU regression illustrated in the previous subsection can also be strongly quasar convex.
3.2. P OLYAK -Ł OJASIEWICZ (PL) OR Q UADRATIC - GROWTH (QG) FUNCTIONS WITH QUASAR CONVEXITY
Recall that a function f ( · ) satisﬁes ν -QG w.r.t.
a global minimizer w ∗ ∈ R d if f ( w ) − f ( w ∗ ) ≥ ν 2 (cid:107) w − w ∗ (cid:107) 2 for some ν > 0 and all w ∈ R d (Drusvyatskiy & Lewis, 2018; Karimi et al., 2016).
Recall also that a function f ( · ) satisﬁes ν -PL if 2 ν ( f ( w ) − f ( w ∗ )) ≤ (cid:107)∇ f ( w ) (cid:107) 2 for some ν >and all w ∈ R d (Karimi et al., 2016).
It is known that a ν -PL function satisﬁes ν -QG, see e.g.,
Appendix A in Karimi et al.
The notion of PL has been discovered in various non-convex problems recently (Altschuler et al., 2021; Oymak & Soltanolkotabi, 2019; Chizat, 2021;
Merigot et al., 2021).
We show in Lemma 7 below that if a ν -QG function f ( · ) satisﬁes quasar convexity, then it also satisﬁes strong quasar convexity.
Lemma 7.
Suppose that the function f ( · ) is ν -QG and ˆ ρ -quasar convex w.r.t.
a global minimizer w ∗ .
Then, it is also ( ρ = ˆ ρθ, µ = ν (1 − θ ) θ ) -strongly quasar convex for any θ < 1 .
Lemma 8 in the following shows that GLMs with increasing link functions satisfy QG under certain distributions D , e.g., Gaussian, and hence they are strongly quasar convex by Lemma 7 and Lemma 3.
Lemma 8.
Following the setting of Lemma 3, assume that the smallest eigenvalue of the matrix E
[Equation]
] satisﬁes λ min ( E x ∼D [ xx (cid:62) ]) > 0 .
Then, the function (11) is α 2 λ min ( E x ∼D
[ xx (cid:62) ]) -QG.
The proofs of Lemma 7 and Lemma 8 are available in Appendix C.5.
M AIN RESULTS : ALGORITHMIC ASPECTS
We ﬁrst analyze the continuized Nesterov acceleration (3) and its discrete-time version (8)-(10) for minimizing quasar convex functions.
Theorem 1.
Assume that the function f ( · ) is L -smooth and ρ -quasar convex.
Let η t = 2 ρt , η (cid:48) t = 0 , γ t = 1 L , and γ (cid:48) t
= ρt 2 L .
Then, the update w t of the continuized algorithm (3) satisﬁes
[Equation]
Furthermore, for the update ˜ w k of the discrete-time algorithm (8)-(10), if the parameters are chosen as τ k = 1 − (cid:16) T k T k +1 (cid:17) 2 /ρ , τ (cid:48) k
= 0 , ˜ γ k = 1 L , and ˜ γ (cid:48) k
= ρT k 2 L , then
[Equation]
It is noted that the expectation E is with respect to the Poisson process, which is the only source of randomness in the continuized Nesterov acceleration.
By applying some concentration inequalities, we can get a bound on the optimal gap with a high probability from Theorem 1.
Corollary 1.
The update ˜ w k of the algorithm (8)-(10) with the same parameters indicated in
Theorem 1 satisﬁes f
[Equation]
˜ z
[Equation]
w ∗ (cid:107) 2 (1 − c )
[Equation]
2 k 2 , with probability at least 1 − 1 c 2 k − 1 c 0 for any c ∈ (0 , 1) and c 0 > 1 .
Corollary 1 implies that K
= O (cid:16) L 1 / 2 (cid:107)
[Equation] [Equation]
w
[Equation]
(cid:17) number of gradient calls is sufﬁcient for the discretetime algorithm to get an (cid:15) -optimality gap with a high probability, since the discrete-time algorithm only queries one gradient in each iteration k .
Next we analyze the convergence rate for minimizing ( ρ, µ )-strongly quasar-convex functions.
Theorem 2.
Assume that the function f ( · ) is L -smooth and ( ρ, µ )-strongly quasar convex, where µ > 0 .
Let γ t = 1 L , γ (cid:48) t = 1 √ µL , η (cid:48)
t = ρ (cid:112) µL , and η t = (cid:112) µL .
Then, the update w t of the continuized algorithm (3) satisﬁes
[Equation]
Furthermore, for the update ˜ w k of the discrete-time algorithm (8)-(10), if the parameters are chosen as
[Equation]
γ k = 1 L , and ˜ γ (cid:48) k
= 1 √ µL , then
[Equation]
Corollary 2.
The update ˜ w k of the algorithm (8)-(10) with the same parameters indicated in
Theorem 2 satisﬁes
[Equation]
with probability at least 1 − 1 c 2 k − 1 c 0 for any c ∈ (0 , 1) and c 0 > 1 .
The proof of the above theorems and corollaries are available in Appendix D. Denote V :
= f
( ˜ w 0 ) − f ( w ∗ ) + µ 2 (cid:107) ˜
z 0 − w ∗ (cid:107) 2 .
Theorem 2 and Corollary 2 show that the proposed algorithm takes
O (cid:18) √
[Equation]
/µ
[Equation]
log (cid:0) V(cid:15) (cid:1)(cid:19) number of iterations with the same number of gradient evaluations to get an (cid:15) -expected optimality gap and an (cid:15) -optimality gap with a high probability respectively.
Together with Theorem 1 and Corollary 1, these theoretical results show that the continuized Nesterov acceleration has an advantage compared to the existing algorithms of minimizing quasar and strongly quasar-convex functions (Hinder et al., 2020; Bu & Mesbahi, 2020; Nesterov et al., 2019; Guminov & Gasnikov, 2017), as it avoids multiple gradient calls in each iteration and does not need function evaluations to have an (cid:15) -gap with a high probability.
On the other hand, it should be emphasized that the guarantees in the aforementioned works are deterministic bounds, while ours is an expected one or a high-probability bound.
Recall Lemma 2 suggests that C v -one-point convexity and L -smoothness implies ρ = 2 C v L quasar convexity.
Furthermore, Lemma 6 states that ˆ ρ quasar convexity and C v -one-point convexity actually implies (cid:16)
ρ
= ˆ ρθ , µ = 2 C v ( [Equation] [Equation] ρ (cid:17) -strongly quasar convex for any θ > 1 .
By combining Lemma 2 and
Lemma 6, we ﬁnd that C v -one-point convexity and L -smoothness implies ( ρ = 2 C v Lθ , µ =
L ( θ − 1)) - strong quasar convexity, for any θ > 1 .
By substituting ( ρ = 2 C v Lθ , µ =
L ( θ − 1)) into the complexity
O (cid:18) √ L/µ ρ log (cid:0) V(cid:15) (cid:1)(cid:19) indicated by Corollary 2, we see that the required number of iterations to get an (cid:15) gap with high probability for minimizing functions that satisfy C v -one-point convexity and
L -smoothness via the proposed algorithm is O (cid:16) LC v
[Equation]
, where we simply let θ = 2 .
On the other hand, Guille-Escuret et al.
(2022) consider minimizing a class of functions that satisﬁes C v -one-point convexity and a condition called the L -upper error bound condition ( L -EB + ).
A function satisﬁes L -EB + if (cid:107)∇ f ( w ) − ∇ f ( w ∗ ) (cid:107) 2 ≤ L (cid:107) w − w ∗ (cid:107) 2 for a ﬁxed minimizer w ∗ and any w ∈ R d .
Guille-Escuret et al.
(2022) show that the optimal iteration complexity k to have (cid:107) w k − w ∗ (cid:107) 22 (cid:107) w 0 − w ∗ (cid:107)
= ˆ (cid:15) for minimizing the class of functions via any ﬁrst-order algorithm is k = Θ (cid:18)(cid:16) LC v (cid:17) 2 log (cid:0) 1ˆ (cid:15) (cid:1)(cid:19) and that the optimal complexity is simply attained by GD.
Our result does not contradict to this lower bound result, because L -smoothness and L -EB + are different.
L -EB + is about gradient Lipschitzness between a minimizer w ∗ and any w , not for any pair of points in R d , and hence does not imply L -smoothness.
Also, L -smoothness does not imply L -EB + .
Lastly, it is noted that both theorems (and corollaries) require the L -smoothness condition.
The reader might raise a question whether the smoothness holds for the case when a GLM with the link function is ReLU or Leaky-ReLU.
For this case, it has been shown that the objective (11) satisﬁes smoothness when the data distribution is a Gaussian distribution, e.g., Lemma 5.2 in Zhang et al.
Continuized accelerated algorithm with stochastic pseudo-gradients for GLMs: We also propose a stochastic algorithm to recover an unknown GLM w ∗
∈ R d that generates the label y of a sample x ∈ R d via y =
σ ( w (cid:62)∗ x ) , where σ ( · ) is the link function.
A natural metric for this task is the distance to the unknown target w ∗ , i.e., f ( w ) :
= 12 (cid:107) w − w ∗ (cid:107) 22 ,
However, since we do not have access to w ∗ , we cannot use the gradients of f ( · ) for the update.
Instead, let us consider using stochastic pseudo-gradients , deﬁned as g ( w ; ξ ) :
= (cid:0) σ ( w (cid:62) x ) − y (cid:1) x , where ξ := ( x, y ) represents a random sample drawn from the data distribution.
Assume that the ﬁrst derivative of the link function is positive, i.e., σ (cid:48) ( · ) ≥ α > 0 .
Then, the expectation of the dot product between the pseudo-gradient and the gradient ∇ f ( w ) over the data distribution satisﬁes
[Equation]
which implies that taking a negative of pseudo-gradient step should make progress on minimizing the distance 12 (cid:107) w − w ∗ (cid:107) 22 on expectation when w has not converged to w ∗ .
That is, the update w t +1 =
w t − ηg ( w t ; ξ ) could be shown to converge to the target w ∗ ∈ R d
under certain conditions, where η > 0 is the step size.
In fact, this algorithm is called (stochastic) GLMtron in the literature (Kakade et al., 2011).
We introduce a continuized acceleration of it in the following.
But before that, let us provide some necessary ingredients ﬁrst.
Denote a matrix H ( w ) :=
E x [ ψ ( w (cid:62) x, w (cid:62)∗ x ) xx (cid:62) ] , where ψ ( a, b ) :
= σ ( a ) − σ ( b ) a − b .
When the data matrix satisﬁes E x [ xx (cid:62) ] (cid:23) θI d for some θ > 0 and when the derivative of the link function σ ( · ) satisﬁes σ (cid:48) ( · ) ≥ α > 0 , one has H ( w ) (cid:23) µI d (cid:31) 0 , where µ := αθ .
We assume that for any w ∈ R d , it holds that E x (cid:2) ψ ( w (cid:62) x, w (cid:62)∗ x ) 2 (cid:107) x (cid:107) 22 xx (cid:62) (cid:3) (cid:22) R 2 H ( w ) for some constant R 2 > 0 , and also that E x (cid:104) ψ ( w (cid:62) x, w (cid:62)∗ x ) 2 (cid:107) x (cid:107) 2 H ( w ) − 1 xx (cid:62) (cid:105) (cid:22) ˜ κH ( w ) for some constant ˜ κ > 0 , where we denote (cid:107) x (cid:107) 2 H ( w ) − 1 := x (cid:62) H ( w ) − 1 x and H ( w ) − 1 is the inverse of H ( w ) .
Deﬁne κ :=
R 2 µ .
Then, we have ˜ κ ≤ κ , because E x (cid:104) ψ ( w (cid:62) x, w (cid:62)∗ x ) 2 (cid:107) x (cid:107) 2 H ( w ) − 1 xx (cid:62) (cid:105) (cid:22) 1 µ E x (cid:2) ψ ( w (cid:62) x, w (cid:62)∗ x ) 2 (cid:107) x (cid:107) 22 xx (cid:62) (cid:3) (cid:22)
R 2 µ H ( w ) = κH ( w ) .
The assumptions can be viewed as a generalization of the assumptions made in
Jain et al.
Even et al. (2021) for the standard least-square regression, in which case one has σ ( z ) =
z and hence ψ ( · , · ) = 1 .
Our continuized acceleration with stochastic pseudo-gradient steps can be formulated as:
[Equation]
(13) where η t , η (cid:48) t , γ t , γ (cid:48) t are parameters, ξ ∈ Ξ represents an i.i.d.
random variable associated with a sample used to compute a stochastic pseudo-gradient g ( w ; ξ ) , and dN ( t, ξ )
= Σ k ≥ 1 δ ( T k ,ξ k ) ( dt, dξ ) is the
Poisson point measure on R ≥ 0 × Ξ .
We have Theorem 3 in the following, and its proof is available in Appendix E, where we also provide a convergence guarantee of the discrete-time algorithm.
Theorem 3.
(Continuized algorithm (13) for GLMs)
Choose η (cid:48) t
= (cid:112) µ ˜ κR 2 , η t = (cid:112) µ ˜ κR 2 , γ t = 1 R 2 , and γ (cid:48) t = 1 √ µ ˜ κR 2 .
Then, the update w t of (13) satisﬁes
[Equation]
E XPERIMENTS
We compare the proposed continuized acceleration with GD and the accelerated method of Hinder et al.
(2020) (AGD).
For the method of Hinder et al.
(2020), we use their implementation available online (Hinder et al., 2021).
Our ﬁrst set of experiments consider optimizing the empirical risks of GLMs with link functions being logistic, ReLU, and quadratic, i.e., solving min w 1 n (cid:80) ni
=1 (cid:104) 12 (cid:0) σ ( w (cid:62) x i ) − y i (cid:1) 2 (cid:105) , where n is the number of samples.
Each data point x i is sampled from the normal distribution N (0 , I d ) and the label y
i is generated as y i =
σ ( w (cid:62)∗ x i ) , where w ∗ ∼ N (0 , I d ) is the true vector and σ ( · ) is the link function.
In the experiments, we set the number of samples n = 1000 and the dimension d = 50 .
The initial point of all the algorithms w 0 ∈ R d is a close-to-zero point, and is sampled as w 0 ∼ 10 − 2 ζ , where ζ ∼ N (0 , I d ) .
Since the continuized acceleration has randomness due to the Poisson process, it was replicated 10 runs in the experiments, and the averaged results over these runs are reported.
Both the continuized acceleration and AGD of Hinder et al.
(2021) need the knowledge of L , ρ , and µ for setting their parameters theoretically.
We instead use the grid search and report the result under the best conﬁguration of these parameters for each method.
More precisely, we search L and µ over { . . .
, 10 q , 5 × 10 q , 10 q +1 , . . .
} with the constraint that L > µ , where q ∈ {− 2 , − 1 , . . .
, 4 } , and search ρ ∈ { 0 . 01 , 0 . 1 , 0 . 5 } .
Figure 1 shows the results, where we compare the performance of the algorithms in terms of the function value versus iteration, the number of gradient calls, and CPU time (seconds).
From the ﬁrst column of the ﬁgure, one can see that the proposed continuized acceleration is competitive with
AGD of Hinder et al.
(2020) in terms of the number of iterations.
From the middle and the last column, the continuized acceleration shows its promising results over AGD and GD when they are
[Figure] [Figure]
[Figure] [Figure]
measured in terms of the number of gradient calls and CPU time, which conﬁrms that the cost of
AGD per iteration is indeed higher than the continuized acceleration and showcases the advantage of the continuized acceleration.
Our second set of experiments compare stochastic GLMtron and the proposed continuized acceleration of it (accelerated stochastic GLMtron), in which both algorithms randomly select a sample to compute a stochastic pseudo-gradient at each step of the update.
We consider learning a GLM with a Leaky-ReLU, i.e., σ ( z ) = max( αz, z ) under different values of α .
Figure 2 shows the effectiveness of accelerated stochastic GLMtron, as it is signiﬁcantly faster than stochastic GLMtron for recovering the true vector w ∗ .
C ONCLUSION
We show that the continuized Nesterov acceleration outperforms the previous accelerated methods for minimizing quasar convex functions.
Compared to the previous approaches, the continuized discretization technique provides a relatively easy way to design and analyze an accelerated algorithm for quasar convex functions.
Hence, it would be interesting to check whether this technique could offer any other beneﬁts in non-convex optimization.
Speciﬁcally, can the technique help design fast algorithms for minimizing other classes of non-convex functions?
On the other hand, while examples of quasar convex functions are provided in this paper, a natural question is if this property holds more broadly in modern machine learning applications.
Exploring the possibilities might be another interesting direction.
A A LGORITHMS OF H INDER ET AL . (2020)
We replicate the algorithms in Hinder et al.
(2020) using our notations for the reader’s reference.
Their algorithms use a subroutine of binary search to determine the “mixing” parameter τ k .
[Table] [Table]
= ρ(cid:15) 2 (cid:17) .
4: τ k ← 1 − α k .
5: v k = w k + τ k
( z k − w k ) .
6: w k +1 =
v k − ˜
γ k +1 ∇ f
( v k ) .
7: z k +1 =
z k + τ (cid:48) k
( v k − z k ) − ˜ γ (cid:48)
k +1 ∇ f ( v k ) .
B P ROOF OF L EMMA
Lemma 1 (Theorem 3 in Even et al.
The discretization of the continuized Nesterov acceleration (3) can be implemented as ˜ w k :=
w T k , ˜ v k :=
w T k +1 − , ˜
z k :=
z T k .
Furthermore, the update of the discretized process is in the following form:
[Equation]
[Equation]
where τ k , τ (cid:48) k , ˜ γ k , ˜ γ (cid:48) k are random parameters that are functions of η t , η (cid:
) t , γ t , and γ (cid:48) t .
Proof.
We replicate the proof in (Even et al., 2021) for completeness.
Recall between random times, we have the ODEs
[Equation]
Integrating from T k to T k +1 − ,
[Equation]
where τ k and τ (cid:48)(cid:48) k depend on η t and η (cid:48) t respectively.
Combing the above two equations, we have
[Equation] [Equation]
where τ (cid:48) k :
= τ (cid:48)(cid:48) k 1 − τ k .
Furthermore, from (6) and (7), we have
[Equation]
Hence, ˜ γ k +1 =
γ T k
+1 and ˜ γ (cid:48)
k +1 = γ (cid:48)
T k +1 .
C M ISSING PROOFS IN S ECTIONC. P ROOF OF L EMMA
Lemma 3 Suppose that the link function σ ( z ) is L 0 -Lipschitz and α -increasing, i.e., σ (cid:48) ( z ) ≥ α >for all z > R .
Then, the loss function (11) is α 2 -generalized variational coherent and L 20 2 -generalized smooth w.r.t.
h ( w, w ∗ ) =
E x ∼D (cid:2) (( w − w ∗ ) (cid:62) x ) 2 (cid:3) .
Therefore, the function (11) is ρ =
α 2 L 20 -quasar convex.
Proof.
We ﬁrst show generalized variational coherence.
We have
[Equation]
where (a) uses that y
= σ ( w (cid:62)∗ x ) , (b) uses that σ ( w (cid:62) x ) − σ ( w (cid:62)∗ x ) w (cid:62) x − w (cid:62)∗ x ≥ 0 as σ (cid:48) ( · ) ≥ α > 0 .
Now let us switch to show generalized smoothness.
We have
[Equation]
where the inequality is due to L 0 -Lipschitzness of σ ( · )
We can now invoke Lemma 2 to conclude that the objective function is ρ = 2 α 2 L 20 -quasar convex.
C. P ROOF OF L EMMA
Lemma 4 Assume that there exists a ﬁnite constant C R > 0 such that all w ∈ R d in the balls of radius R centered at ± w ∗ satisfy E x ∼ D (cid:104)(cid:0) ( w + w ∗ ) (cid:62) x (cid:1) 2 (cid:107) x (cid:107) 22 (cid:105) ≤ C R .
Then, the loss function (11) is 12 C R -generalized smooth w.r.t.
h ( w, w ∗ ) = (cid:107) w − w ∗ (cid:107) 22 .
Proof.
We have
[Equation]
C. P ROOF OF L EMMA
Lemma
When the link function is ReLU, the loss function (11) is 12 E x ∼ D
[ (cid:107) x (cid:107) 22 ] -generalized smooth w.r.t.
h ( w, w ∗ ) = (cid:107) w − w ∗ (cid:107) 22 .
Proof.
We have
[Equation]
where the ﬁrst inequality uses that ReLU is 1 -Lipschitz.
C. P ROOF OF L EMMA
Lemma 6 Suppose that the function f ( · ) satisﬁes C v -one-point convexity and ˆ ρ -quasar convexity.
Then, it is also (cid:16) ρ
= ˆ ρθ , µ = 2 C v ( θ − 1) ˆ ρ (cid:17) -strongly quasar convex for any θ > 1 .
Proof.
We have
[Equation]
(28) where the last inequality uses the deﬁnition of C v -one-point convexity.
Rearranging the above inequality, we get f ( w ∗ ) ≥ f ( w ) + 1ˆ ρ/θ (cid:104)∇ f ( w ) , w ∗ − w (cid:105) + 2 C v ( θ − 1) / ˆ ρ(cid:107)
w ∗ − w (cid:107) 2 .
C. P ROOF OF L EMMA 7 AND L EMMA
Lemma 7 Suppose that the function f ( · ) is ν -QG and ˆ ρ -quasar convex w.r.t.
a global minimizer w ∗ .
Then, it is also ( ρ = ˆ ρθ, µ = ν (1 − θ ) θ ) -strongly quasar convex for any θ < 1 .
Proof.
By ˆ ρ -quasar convexity, we have
[Equation]
where the last inequality uses the deﬁnition of ν -QG.
Rearranging the above inequality, we get f ( w ∗ ) ≥ f
[Equation]
+
[Equation]
, which shows the result.
Lemma
Following the setting of Lemma 3, assume that the smallest eigenvalue of the matrix E x ∼D [ xx (cid:62) ] satisﬁes λ min ( E x ∼D [ xx (cid:62) ]) > 0 .
Then, the function (11) is α 2 λ min ( E x ∼D
[ xx (cid:62) ]) -QG.
Proof.
We have
[Equation]
where the second-to-last inequality uses that the derivative of the link function satisﬁes σ (cid:48) ( · ) ≥ α .
D P ROOF OF T HEOREM 1 AND T HEOREM
Theorem 1 Assume that the function f ( · ) is L -smooth and ρ -quasar convex.
Let η t = 2 ρt , η (cid:48) t = 0 , γ t = 1 L , and γ (cid:48) t
= ρt 2 L .
Then, the update w t of the continuized algorithm (3) satisﬁes
[Equation]
Furthermore, for the update ˜ w k of the discrete-time algorithm (8)-(10), if the parameters are chosen as τ k = 1 − (cid:16) T k T k +1 (cid:17) 2 /ρ , τ (cid:48) k
= 0 , ˜ γ k = 1 L , and ˜ γ (cid:48) k
= ρT k 2 L , then
[Equation]
Theorem 2 Assume that the function f ( · ) is L -smooth and ( ρ, µ )-strongly quasar convex, where µ > 0 .
Let γ t = 1 L , γ (cid:48) t = 1 √ µL , η (cid:48)
t = ρ (cid:112) µL , and η t = (cid:112) µL .
Then, the update w t of the continuized algorithm (3) satisﬁes
[Equation]
(cid:114)
[Equation]
Lt (cid:19) .
Furthermore, for the update ˜ w k of the discrete-time algorithm (8)-(10), if the parameters are chosen as τ
[Equation] [Equation]
and ˜ γ (cid:48) k = 1 √ µL , then
[Equation]
The proof follows that of Theorem 2 in Even et al.
(2021) with some modiﬁcations to account for (strong) quasar convexity.
We will consider a Lyapunov function for the continuized process (3), deﬁned as:
[Equation]
We will show that φ t is a super-martingale under certain choices of parameters η t , η (cid:48) t , γ t , γ
[Equation]
t , A t , and
B t .
Let us ﬁrst denote the process ¯ w t :=
( t, w t , z t ) , whose dynamic is:
[Equation]
Then, by Proposition 2 of Even et al.
[Equation]
we have
[Equation]
where M t is a martingale.
Therefore, to show φ t is a supermartingale, it sufﬁces to show:
[Equation]
For the ﬁrst term of I t .
we have
[Equation]
By ( ρ, µ ) -strongly quasar-convexity
[Equation]
we have
[Equation]
Furthermore, the following inequality holds,
[Equation]
(37) because
[Equation]
Combining (35)-(37) gives
[Equation]
For the second term of I t
[Equation]
we have
[Equation]
Since by smoothness
[Equation]
we have
[Equation] [Equation]
So the second term can be bounded as
[Equation]
[Equation]
Combining (34), (38), (41), we have
[Equation]
Now let us determine the parameters η t , η (cid:48) t , γ t , γ (cid:48) t , A t and B t .
We start by taking γ t = 1 L .
Since we need I t ≤ 0 , we want to satisfy
[Equation] [Equation]
Let us choose
[Equation]
44) which ensures that the last three conditions of (43) are satisﬁed.
It remains to show that the ﬁrst two hold:
[Equation]
We have
[Equation]
46) where (a) uses that dA t dt = ρA t η t = ρA t (cid:113) B t LA t from (45), and (b) uses dB t dt = ρµ (cid:113)
A t B t L from (45).
The equations on (46) imply that
[Equation]
D. ρ - QUASAR - CONVEX
Proof.
(of Theorem 1)
For the case of µ = 0 , we choose A 0 = 0 and B 0 = 1 .
From (46), we have ddt ( √ B t )
=
so that
B t = 1 and that ddt ( √ A t ) =
ρ 2 √ L , consequently, √ A t = ρ t 2 √ L .
From (44), we conclude γ t = 1 L , γ (cid:48) t = ρt 2 L , η (cid:48) t = 0 , and η t = 2 ρt .
Therefore, as φ t is a super-martingale, we get
[Equation] [Equation]
So we have
[Equation] [Equation]
This proves the ﬁrst part of Theorem 1.
The ODEs (4)-(5) become
[Equation]
Integrating the ODEs from time t 0 to t ,
[Equation]
[Equation]
Using Lemma 1 with t 0 =
T k , t =
T k +1 − , (52) becomes
[Equation]
This together with (8) implies that τ k = 1 − (cid:16) T k T k +1 (cid:17) 2 /ρ , while comparing (20) and (53) leads to τ (cid:48)(cid:48) k
= 0 and τ (cid:48) k
= τ (cid:48)(cid:48)
k 1 − τ k = 0 .
Moreover, we have ˜ γ k = γ T k = 1 L and ˜ γ (cid:48) k
= γ (cid:48)
T k =
ρT k 2 L .
This proves the second part of Theorem 1.
D. ( ρ, µ ) - STRONGLY QUASAR - CONVEX
Proof.
(of Theorem 2)
From (44), we choose γ t = 1 L , γ (cid:48) t =
[Equation]
, η
[Equation]
t
[Equation]
We have
[Equation]
Then, we can conclude that
[Equation] [Equation]
So we have
[Equation]
This proves the ﬁrst part of Theorem
[Equation]
The ODEs (4)-(5) become
[Equation]
The solutions are
[Equation]
Taking t
= T k
[Equation]
t =
T k
+1 −
[Equation]
the above becomes
[Equation] [Equation]
Comparing equation (64) and (14), we know that τ k
=
+ ρ (cid:0) 1 − exp (cid:0) − (1 + ρ ) (cid:112) µL ( T k +1 − T k ) (cid:1)(cid:1) .
Furthermore, by using (64), we get
[Equation]
Based on (65), (66), and (21), we conclude that τ (cid:48) k
= ρ ( 1 − exp ( − (1+ ρ ) √ µL
[Equation] [Equation]
( −
[Equation]
+
[Equation] [Equation]
µL
[Equation]
T
[Equation]
+1 − T k ) ) .
Moreover, we have ˜ γ k = γ T k = 1 L and ˜ γ (cid:48) k
= γ (cid:48)
T k = 1 √ µL .
This proves the second part of Theorem 2.
D. P ROOF OF C OROLLARY 1 AND C OROLLARY
Corollary 1:
The update ˜ w k of the algorithm (8)-(10) with the same parameters indicated in
Theorem 1 satisﬁes f ( ˜ w k ) − f ( w ∗ ) ≤ 2 c 0 L (cid:107)
[Equation]
2 k 2 , with probability at least 1 − 1 c 2 k − 1 c 0 for any c ∈ (0 , 1) and c 0 > 1 .
Proof.
Using Markov’s inequality and Theorem 1, we get
[Equation]
Let C 0 :
= c 0 2 L (cid:107) ˜
z 0 − w ∗ (cid:107) 2 /ρ 2 , where c 0 > 1 is a universal constant.
Then, with probability 1 − 1 c 0 ,
[Equation]
By Chebyshev’s inequality, we have Pr ( |
[Equation]
E [ T k ]) 2 , where c > 0 is a universal constant.
Hence, we have T k ≥ (1 − c ) E
[ T k ] =
(1 − c ) k with probability at least 1 − 1 c 2 k , where we used the fact that E [ T k ] =
Var[ T k ] =
k as T k is the sum of k Poisson random variables with mean 1 .
Combining this lower bound of T k and (68) leads to the result.
Corollary 2:
The update ˜ w k of the algorithm (8)-(10) with the same parameters indicated in
Theorem 2 satisﬁes f
[Equation]
with probability at least 1 − 1 c 2 k − 1 c 0 for any c ∈ (0 , 1) and c 0 > 1 .
Proof.
Using Markov’s inequality and Theorem 2, we get
[Equation]
Let C 0 :
=
c 0 (cid:0) f
( ˜ w 0 ) − f ( w ∗ ) + µ 2 (cid:107) ˜
z 0 − w ∗ (cid:107) 2 (cid:1) , where c 0 > 1 is a universal constant.
Then, with probability 1 − 1 c 0 ,
[Equation]
By Chebyshev’s inequality, we have Pr ( | T k − E
[ T k ] | ≥ c E [ T k ])
≤ Var( T k ) c 2 ( E [ T k ]) 2 , where c > 0 is a universal constant.
Hence, we have T k ≥ (1 − c ) E
[ T k ] =
(1 − c ) k with probability at least 1 − 1 c 2 k , where we used the fact that E [ T k ] =
Var[ T k ] =
k as T k is the sum of k Poisson random variables with mean 1 .
Combining this lower bound of T k and (70) leads to the result.
E P ROOF OF T HEOREM
Theorem 3 (Continuized algorithm (13) for GLMs)
Choose η (cid:48) t
= (cid:112) µ ˜ κR 2 , η t = (cid:112) µ ˜ κR 2 , γ t = 1 R 2 , and γ (cid:48) t = 1 √ µ ˜ κR 2 .
Then, the update w t of (13) satisﬁes
[Equation]
Proof.
Let us denote H t :
= H ( w t ) =
E x [ ψ ( w (cid:62) t x, w (cid:62)∗ x ) xx (cid:62) ] and consider a Lyapunov function for the continuized process (3), deﬁned as:
[Equation]
We ﬁrst show that φ t is a super-martingale under certain values of parameters η t , η (cid:48) t , γ t , γ (cid:48) t , A t , and
B t .
Let us denote the process ¯ w t :=
( t, w t , z t ) , which satisﬁes the following equation:
[Equation]
Then, by Proposition 2 of Even et al.
[Equation]
we have
[Equation]
where M t is a martingale and I t is
[Equation]
For the ﬁrst term of I t
[Equation]
we have
[Equation] [Equation]
Since H t = E x [ ψ ( w (cid:62) t x, w (cid:62)∗ x ) xx (cid:62) ] (cid:23) µI d , we have
[Equation]
Using (76), we have
[Equation] [Equation]
Furthermore, the following inequalities hold,
[Equation]
This is because (cid:104) z t
[Equation] [Equation]
≤
[Equation]
w
[Equation]
− w ∗
[Equation]
Combining (75)-(78), we get (cid:104)∇ φ ( ¯ w
[Equation]
) , b (
[Equation]
w
[Equation]
) (cid:105) ≤ dA
[Equation]
dt (cid:107)
[Equation]
For the second term of I t
[Equation]
we have
[Equation] [Equation]
Let us upper-bound the ﬁrst two terms in (80).
We have
[Equation]
where used
[Equation]
Furthermore, we have
[Equation]
Therefore
[Equation]
the ﬁrst two terms in (80) can be bounded as
[Equation]
Now let us switch to upper-bound the last two terms in (80).
We have
[Equation]
where in the last inequality
[Equation]
we used the assumption that
[Equation]
We also have
[Equation]
So the last two terms in (80) can be bounded as
[Equation] [Equation]
Therefore, combining (84)
[Equation] [Equation]
and (80), we have
[Equation] [Equation]
Combining (74)
[Equation] [Equation]
and (89), we have:
[Equation]
Now let us determine η t , η (cid:48) t , γ t , γ (cid:
) t , A t , and B t .
We start by taking γ t = 1 R 2 .
We want I t ≤ 0 , so we want to satisfy dA t
[Equation]
Let us choose
[Equation] [Equation]
which ensures that the last three conditions of (91) are satisﬁed.
It remains to show that the ﬁrst two hold:
[Equation]
We have
[Equation]
˜
[Equation] [Equation]
where (a) uses that dA t dt =
A t η t =
A t (cid:113) B t ˜ κR 2 A t and (b) uses that dB t dt = B t η (cid:48) t
= µ (cid:113)
A t ˜ κR 2 B t from (92) and (93).
The equations on (94) imply that
[Equation]
Let us choose γ t = 1 R
[Equation]
=
[Equation]
We have
[Equation]
Then
[Equation]
we can conclude that
[Equation]
So we have
[Equation]
and we have B
A 0 = µ from (96) and we can choose A
= 1 .
This proves Theorem
[Equation]
Now let us switch to determine the corresponding parameters of the discrete-time algorithm (8)-(10), where the gradient ∇ ( w t ) is now replaced with the stochastic pseudo-gradient g ( w t ; ξ t ) .
The ODEs (4)-(5) become
[Equation]
The solutions are
[Equation]
Taking t
= T k , t =
T k
+1 −
[Equation]
the above becomes
[Equation]
Comparing equation (105) and (14), we know τ k = 12 (cid:0) 1 − exp (cid:0) − 2 (cid:112) µ ˜ κR 2 ( T k +1 − T k ) (cid:1)(cid:1) and τ (cid:48)(cid:48) k
= 12 (cid:0) 1 − exp (cid:0) − 2 (cid:112) µ ˜ κR 2 ( T k +1 − T k ) (cid:1)(cid:1) .
Furthermore, by using (105), we get
[Equation]
Based on (106), (107), and (21), we conclude that τ (cid:48) k
= 1 − exp ( −
[Equation]
1+
[Equation] [Equation]
˜
[Equation]
( T
[Equation]
+1 − T k ) ) .
Moreover, we have ˜ γ k = γ T k = 1 R 2 and ˜ γ (cid:48) k
= γ (cid:48)
T k = 1 √ µ ˜ κR 2 .
We can now conclude that the corresponding discrete-time algorithm under the above choice of parameters satisﬁes
[Equation]