Approaching Quartic Convergence Rates for Quasi-Stochastic Approximation with Application to Gradient-Free Optimization
Abstract
Stochastic approximation is a foundation for many algorithms found in machine learning and optimization.
It is in general slow to converge: the mean square error vanishes as O ( n − 1 ) .
A deterministic counterpart known as quasi-stochastic approximation is a viable alternative in many applications, including gradient-free optimization and reinforcement learning.
It was assumed in prior research that the optimal achievable convergence rate is O ( n − 2 ) .
It is shown in this paper that through design it is possible to obtain far faster convergence, of order O ( n − 4+ δ ) , with δ > 0 arbitrary.
Two techniques are introduced for the ﬁrst time to achieve this rate of convergence.
The theory is also specialized within the context of gradient-free optimization, and tested on standard benchmarks.
The main results are based on a combination of novel application of results from number theory and techniques adapted from stochastic approximation theory.
Introduction
Stochastic approximation (SA) was introduced in the seminal work of Robbins and Monro [38].
The goal is to solve the root ﬁnding problem
[Equation]
of the form
[Equation]
where ζ is a random vector taking values in R m .
The basic algorithm is expressed as the d -dimensional recursion,
[Equation]
in which { α n } is the step-size sequence and ζ n +1 d −→ ζ as n → ∞ (convergence in distribution).
SA theory has attracted a great deal of attention over the past twenty years, motivated in large part by applications to reinforcement learning and optimization [49, 18, 32, 16, 7].
Convergence theory is couched in the ODE Method in which trajectories of (2) are compared to solutions of the ODE ˙ ϑ
= s f ( ϑ ) (the mean ﬂow ).
The major assumption required to ensure convergence to θ ∗ , for each initial condition θ 0 ∈ R d , is that the mean ﬂow is globally asymptotically stable—see [7] for minimal assumptions on the “noise sequence” ζ .
Establishing sharp rates of convergence is a far greater challenge.
There is however a rich theory available to achieve the optimal rate of convergence for the mean-square error (MSE), which is in general E [ k θ n − θ ∗ k 2 ]
= O ( n − 1 ) .
There are many applications for which the designer of the algorithm also designs the noise.
Notable examples include the introduction of exploration in reinforcement learning or gradient-free optimization.
This motivates the use of quasi-stochastic approximation (QSA) in which the sequence ζ is deterministic (e.g. mixtures of sinusoids or pseudo-random numbers).
The idea was introduced in 36th Conference on Neural Information Processing Systems (NeurIPS 2022). [20, 22], but has a much longer history in the context of gradient-free optimization—see [48, 26] for a survey of extremum seeking control (ESC).
Theory supporting rates of convergence of nonlinear QSA appeared only recently [9, 30].
Analysis and algorithms are posed in continuous time to simplify analysis. This setting is also motivated by recent success stories justifying algorithm design in continuous time, followed by a careful translation to obtain a discrete time algorithm.
See for example theory surrounding acceleration methods of Polyak and Nesterov [2, 46, 19].
The notation adopted in [30, Chap. 4] will be used here: the QSA ODE is deﬁned as
[Equation]
The deterministic continuous time process ξ will be called the probing signal , and plays the role of ζ in SA; a is called the gain process.
The motivation for QSA is two-fold:
(i) It will be seen that the rate of convergence is far faster than SA, subject to careful choice of algorithm architecture. (ii)
In on-line applications the introduction of independent noise may not be realizable, or may impose unnecessary stress on equipment.
In QSA design the components of the probing signal might be chosen to be sinusoidal signals of appropriate frequency and magnitude to ensure learning takes place without stress on physical devices.
What is the optimal rate of convergence for QSA?
Consider the most basic one-dimensional problem in which f ( Θ t , ξ t ) =
− Θ t + ξ t , with ξ a zero-mean signal.
The special case a t = 1 / (1 + t ) results in an approximate average:
[Equation]
If for example ξ t = sin( ωt ) then the right hand side converges at rate O
( T − 1 ) , which translates to
O ( T − 2 ) for the “MSE”; this is far faster than the rate O
( T − 1 ) expected for SA.
[Equation]
AVG
[Equation]
[Figure]
The special case of pure averaging is illustrated at the top in Fig.
The bold question mark in the ﬁgure refers to a question regarding a natural extension of the linear example in (4): can we obtain the same rate of convergence for general non-linear QSA ?
This question is posed and answered in the afﬁrmative in [30, § 4.9], achieved through the averaging technique of Polyak and Ruppert, but only under conditions on the QSA ODE that could not be veriﬁed a priori.
The current paper not only provides ways to ensure such conditions are always met, but also answers this question in {+a much more optimistic manner: the presumption that MSE rates of order O ( T − 2 ) are optimal for general QSA is fallacy since for any δ > 0 , rates of order k Θ T − θ ∗ k 
+}
= O ( T − 4+ δ ) can be achieved through design. The astonishingly fast MSE rates obtained for general QSA are a result of the perturbative mean ﬂow (p-mean ﬂow) representation for the QSA ODE.
This representation expresses (3) in terms of the average vector ﬁeld
[Equation]
where {W it : i = 0 , 1 , 2 } are smooth functions of a larger state process, and s Υ t
[-Based on this bound, it isshown that Polyak-Ruppert averaging results in the bound k Θ t − θ ∗ k 
-]
=
s Υ ( Θ t ) with s Υ also smooth (see Thm. 2.1).
This representation is valuable only after boundedness of { Θ t } is established, for which sufﬁcient conditions are provided in [30, Prop. 4.33 and 4.34] (these conditions form part of Assumption (QSA3) in the supplementary material).
A linearization of (5) around θ ∗ gives
[Equation]
{+3 1010          10 T 10 4         10 T 
 T 2 ρ (cid:31) Θ+} t − θ ∗ (cid:31)
T 2 ρ (cid:31) Θ t − θ ∗ (cid:31)    ρ = 0 .
6 ρ = 0 .ρ = 0 . Polyak-Ruppert Forward-Backward where A ∗ =
∂
s f ( θ ∗ ) and s Υ ∗ =
s
Υ ( θ ∗ ) .
The upper bound k Θ t − θ ∗ k
= O ( a t ) (previously obtained in [9, 30]) follows easily from (5).
Given estimates { Θ t : t ≥ 0 } , Polyak-Ruppert (PR) averaging deﬁnes the new estimates as follows,
[Equation]
) where the interval [0 , T 0 ] is known as the burn-in period. The time-average of {W t [-− θ ∗ k 
-]} is of order
O ( a 2 T ) , implying the rate k Θ PR T − θ ∗ k 2 = O ( T − 4+ δ ) if and only if s Υ ∗ = 0 . This brings us to the main contributions: (i)
The p-mean ﬂow representation (5) is introduced for the ﬁrst time in Thm.
(ii) Convergence rates arbitrarily close to O
( T − 2 ) can be obtained (so that the MSE is arbitrarily close to O ( T − 4 ) ), which is far faster than the bound O ( T − 1 ) assumed in prior research.
(iii) The near quartic rates for MSE are established for PR averaging in Thm. 2.2, subject to the assumption that s Υ ∗ = 0 . Sufﬁcient conditions on the probing signal ξ are provided in Thm. 2.
to ensure that
[Equation]
(iv) An alternative to PR averaging is introduced: forward-backward ﬁltering achieves near quartic rates without the special conditions imposed in Thm.
2.2 (see Thm. 2.3).
The theory is reﬁned in the context of gradient free-optimization, and the general theory is illustrated through numerical examples in this setting. [Figure] [Figure] [Figure]
(cid:31)
[Figure]
While theory is developed in continuous time, simulation studies using Euler approximations are consistent with theory.
The plots shown in Fig.
2 are based on a two-dimensional example, whose details can be found in Section 2.1.
The change of notation is also explained there: (2 ρ ) 2 = 4 − δ , where ρ is a design parameter subject to ρ < 1 .
[-In this example ¯ Y 6 = 
 so that Polyak-Ruppert averaging fails (see (16)).-] The forwardbackward algorithm is a new algorithm introduced in this paper, that achieves the O ( t − 4+ δ ) convergence rate; we ﬁnd in these experiments that the same rates hold for the Euler approximation.
Polyak-Ruppert averaging (7) cannot obtain this fast rate of convergence, because the frequencies used in this experiment violate a critical assumption in Thm. 2.2.
Fig.
3 illustrates how the techniques introduced in this paper specialize to quasi-Monte Carlo (QMC), based on two instances of the algorithm: one designed using β = √ 4 − δ = 1 . 4 , and the other with β = 1 . 6 .
Details are found in the supplementary material.
1.1 Preliminaries
A short literature survey and notation glossary are provided here.
Literature review: Polyak-Ruppert (PR) averaging was introduced in [39, 35, 36], and is now a standard workhorse in machine learning [14, 32, 31, 30].
Quasi-Monte Carlo (QMC) remains an active area of research for applications to estimation and optimization [33].
One aspect of the theory concerns techniques to construct a sequence { ζ i : i ∈
Z + } ⊂ [0 , 1] K so that sample path averages are convergent, n − P [Equation]
) µ [Equation] ) as n → ∞ for functions g : R K
→ R , with µ uniform; theory predicts a convergence rate of O (log( n ) K /n ) (see e.g. [1, Section 9.3]). It is shown in Thm. B.1 of the present paper that the convergence rate O (1 /n ) holds for QMC, subject to a smoothness condition on the function, and careful design of the probing signal.
The main results of this paper hinge on these new bounds for QMC, and further reﬁnements . These bounds are based in part on reﬁnements of Baker’s Theorem [28, 8].
QSA was ﬁrst applied to ﬁnance in [20, 22].
Stability and convergence theory for QSA is recent [9, 30], following preliminary results in [3]; this work [-the convergence rate k Θ t − θ ∗ k 
-]was motivated by applications to extremum seeking control, and successful application of QSA to Q-learning in [29].
The linear QSA ODE is treated in [40], where the ﬁrst bounds on the rate of convergence of order
O ( a t ) were obtained.
This bound was extended to general nonlinear QSA in [9, 30], and it was also shown that the use of PR averaging results in a convergence rate of O ( T − 1 ) subject to (up to now) unveriﬁable conditions. QSA with ﬁxed step-size is the topic of the contemporaneous work [24, 25]. An analog of the p-mean ﬂow representation (5) is obtained, along with both steady-state and transient error bounds
Gradient-free optimization (GFO) concerns minimization of an objective function Γ : R d → R based solely on measurements of Γ ( θ ) for selected values of θ ∈ R d .
The algorithms of Keifer and Wolfowitz are early examples [17, 6, 4, 15, 1].
The QSA ODEs for GFO considered in this paper are inspired by the simultaneous perturbation stochastic approximation (SPSA) algorithms of Spall [41, 42, 43]. Two examples are of the form (2), with
[Equation]
where ε > 0 and { ζ n } is a zero-mean i.i.d
sequence.
These algorithms are biased in general unless the probing gain ε is a vanishing function of n .
There is substantial research in this setting: the best possible convergence rate for the mean square error is
O ( n − β ) with β = ( p − 1) / (2 p ) , provided the objective function is p -fold differentiable at θ ∗ [37]. Upper bounds appeared earlier in [13]. See [12, 11, 45, 34, 21] for more recent history.
Deterministic versions of SPSA were analyzed in [5] without sharp rates of convergence.
The present paper follows the approach of [30, § 4.9], in which the applications to gradient-free optimization result in algorithms that resemble simple versions of the ESC algorithms surveyed in [48, 26] (see [25] for more on the relationship between ESC and QSA).
Notation: We restrict to probing signals that are nonlinear functions of sinusoids, of the form ξ t
=
G 0 ( ξ 0 t ) with ξ 0 t = [cos(2 π [ ω 1 t + φ 1 ]) , · · · , cos(2 π [ ω K t + φ K ])] | (9) for which G 0 : R K → R m is analytic on C K , { φ i } are arbitrary, and assumptions on the distinct frequencies { ω i } will be imposed in the main results.
When G 0 is linear we obtain the mixture of sinusoids
[Equation]
vectors
[Equation]
Denote Φ it = exp(2 πj [ ω i t + φ i ]) for each i and t , which deﬁnes the K -dimensional vector-valued Φ t function of time evolving on Ω := { z ∈ C K : | z i | = 1 , 1 ≤ i ≤ K } .
Writing G ( z ) :
= G 0 (( z + z − 1 ) / 2) for non-zero z ∈ C K , we obtain ξ t =
G (Φ t ) .
For two real-valued functions of time { ζ it : t ≥ 0 , i = 1 , 2 } we denote [Equation]
[Equation]
The notation is extended to vector or matrix valued functions.
In particular, h ξ i = 0 ∈ R m , and Σ ξ :=
h ξξ | i is an m × m matrix.
The functions of time used in (11) are frequently obtained as functions of ( Θ t , ξ t ) [or more generally, ( Θ t , Φ t ) ].
If h : R d × R m → R we often write h t ≡ h ( Θ t , ξ t ) and h ∗ t ≡ h ( θ ∗ , ξ t ) to save space, and use the alternative notation h :
=
h h i ; identical notation is used for functions of ( Θ t , Φ t ) . Let ˆ h denote a solution of Poisson’s equation (12) with forcing function h : R m → R , which satisﬁes the deﬁning equations, [Equation] [Equation]
It is assumed throughout that the solution is normalized so that its mean is zero.
Thm.
B.1 in the supplementary material provides conditions ensuring a smooth solution.
For a continuously differentiable ( C 1 ) function g : R d × Ω → R , the directional derivative in the direction of the QSA vector ﬁeld f is denoted
[Equation]
[Equation] For a scalar-valued function δ t > 0 , we use the notation O ( δ t ) to denote a function of t ; the notation indicates that there is a constant B such that k O ( δ t ) k ≤ Bδ t for all t .
That is, we are only asserting an upper bound .
For example, if δ t = 1 / (1 + t ) we are not claiming that k O ( δ t ) k ≤ Bδ 2 t is not possible.
Organization: Section 2 summarizes the main contributions of the paper, and Section 3 provides illustrations of the theory in application to GFO; in particular, convergence rate bounds are validated for GFO algorithms based on QSA.
Conclusions and directions for future research are summarized in Section 4.
Technical proofs and additional numerical results can be found in the supplemental material.
Towards Quartic Rates 2.1 Quasi-Stochastic Approximation
We begin with foundations regarding the QSA ODE (3) from [30, Ch. 4.5, 4.9].
The mean vector ﬁeld is deﬁned by the sample path average,
[Equation]
Theory is based on comparison with the ODE,
[Equation]
The time t 0 is ﬁxed, but chosen suitably large in analysis of convergence rates. The full list of assumptions (QSA0)–(QSA5) may be found in the supplementary material.
We settle for a brief summary here: (QSA0) speciﬁes conditions on the frequencies in (9).
The assumptions allow the following special case: ﬁx ω 1 > 0 and an increasing sequence of positive integers { n i } , and choose the probing signal of the form (9) with
[Equation]
See Prop. A.1 for explanation.
(QSA1) concerns the gain process; in the body of the paper we take a t = a 0 (1+ t ) − ρ for ρ ∈ (1 / 2 , 1] and a constant a 0 > 0 .
(QSA2) imposes global Lipschitz bounds on f and s f .
(QSA3) imposes global asymptotic stability of the mean ﬂow ddt ϑ t =
s f ( ϑ t ) and minor additional assumptions.
(QSA4): s f is C 1 , ¯ A ( θ ) =
∂ θ s f ( θ ) is bounded and Lipschitz continuous, and A ∗ =
A ( θ ∗ ) is Hurwitz.
(QSA5): existence of functions ˆ f , and b Υ that are Lipschitz continuous in θ solving for each 0 ≤ t 0 ≤ t 1 , ˆ f
( θ, Φ t 0 ) =
Z tt˜ f
( θ, ξ t ) dt
[Equation] f ( [Equation] t
[Equation]
[Equation] f
[Equation]
s
[Equation]
) ˆˆ f ( θ, Φ t 0 ) =
Z ttˆ f ( [-T − T 0 ) on[ T 0 , T ] and zero elsewhere. General expressions for Θ t and Θ PR T obtained in [20, § 4.9] are summarized by the following: Theorem 2.2. Suppose that (QSA1)–(QSA5) hold and the following additional assumptions areimposed: { Θ t } is obtained using a t = 1 / (1 + t ) ρ with ρ ∈ (1 / 2 , 1) , and PR averaging is obtainedusing T 
-]θ
[Equation]
ξ t )
dt
[Equation]
ˆˆ f (
[Equation]
t 1 )
b Υ ( θ, Φ t 0 ) = Z tte [Equation]
[Equation]
b [Equation] [Equation] e
[Equation]
[Equation]
s [Equation]
[Equation]
where for
[Equation]
If A ( θ, ξ ) =
¯ A ( θ ) for all θ and ξ then s Υ ( θ ∗ ) = 0 .
This is a very restrictive special case, though it does hold for QMC for which A ( θ, ξ ) =
− I .
In broad generality, we can always ensure s Υ ( θ ∗ ) =by designing an algorithm with ξ satisfying (QSA0) as shown in part (ii) of Thm. 2.1.
The solutions to Poisson’s equation in (17) are used to deﬁne the terms in (5):
Theorem 2.1.
Suppose that (QSA1) holds and a t
= (1 + t ) − ρ , with ρ ∈ (0 , 1] .
(i) Under (QSA5), the p-mean ﬂow representation (5) holds with
[Equation] where r t = ρ/ ( t + 1) (19c) (ii) If (QSA0) and (QSA5) hold, then
[Equation]
0 for each θ ∈ R d [Equation]
(iii) If (QSA2) and (QSA3) hold, then { Θ t } is ultimately bounded: there exists b < ∞ such that for any Θ 0 , lim sup t →∞ k Θ t k ≤ b .
Proof: The proof of part (i) is given at the end of Section B.3, while part (iii) follows from [30, Prop. 4.33 and 4.34]. S = { g = h ◦ G 0 : h analytic }
S = { ˆ g : g ∈ S }
Part (ii) is based on the geometry illustrated in Fig.
4: S denotes the set of functions g : R m → R that are analytic functions of the probing signal, and b S the set of functions h : Ω → R that have zero mean and solve Poisson’s equation for some g ∈ S . Corollary B.6 establishes orthogonality of these two function classes in L 2 : h g, h i = 0 for each g ∈ S and h ∈ b
S (in the notation (11)), which indicates [Equation] The deﬁnition (18) gives [Equation]
for each ξ ∈ R m and each Φ ∈ Ω .
Since b
[Equation]
∈ b
S and f j ( θ, · ) ∈ S for each i, j and θ , it follows that
[Equation]
2.2 Acceleration
Two approaches to obtain the convergence rate bounds k Θ T − θ ∗ k = O ( a 2 ρT ) are described here.
Polyak-Ruppert averaging Fast convergence can be obtained using standard averaging, but only under restrictive conditions.
In Thm. 2.2 we adopt the notation,
[Equation]
∗ , where [Equation] The proof of Thm. 2.2 is postponed to the supplementary material.
Theorem 2.2. Suppose that (QSA1)–(QSA5) hold, and the following additional assumptions are imposed: { Θ t } is obtained using a t = (1+ t ) − ρ with ρ ∈ (1 / 2 , 1) , and estimates using PR averaging (7) are obtained using T {+
 = (1+} − 1 /κ )
T with κ ≥ 1 ﬁxed. Then,
[Equation] [Equation] with c ( κ, ρ ) = κ [Equation]
[Equation]
/κ
[Equation] Thm. 2.1 and the deﬁnition (20) imply that ¯ Y ∗ = 0 under (QSA0).
Eq. (21b) of Thm.
2.2 then implies that PR averaging leads to a convergence rate bounded by O ( T − 2 ρ ) , subject to (QSA0)–(QSA5).
It was discovered recently that the vector s Υ ∗ also appears in representations of the SA recursion (2) when { ζ n } is a smooth function of a Markov chain.
This introduces bias and potentially large variance for ﬁxed step-size algorithms ( α n +1 ≡ α > 0 ), even when using averaging [24, §2.5.3],[23].
Forward-backward ﬁltering We present a technique to achieve near quartic MSE convergence rate without imposing additional structure on the probing signal.
Motivation is two-fold: ﬁrst, it is hoped that this technique can be extended to SA for which there is no known analog to (QSA0); second, in some applications of QSA it may not be possible to ensure that (QSA0) is satisﬁed.
Forward-backward (FB) ﬁltering is deﬁned by a pair of QSA ODEs:
[Equation]
(22) where ξ − t := ξ − t , Θ PR T is deﬁned in (7), and Θ PR − T is deﬁned analogously using { Θ − t } .
Theorem 2.3.
If the assumptions of Thm.
2.2 hold, then, Θ FB T
= θ ∗ + O
( T − 2 ρ ) .
Overview of proof: The analogous vector ¯ Y ∗ for the QSA ODE (22) is denoted ¯ Y ∗− .
The identity ¯ Y ∗− = − ¯ Y ∗ is established through consideration of (20), and the desired bound is then obtained by combining (21b) with the deﬁnition (22).
ut Assumption (QSA0) implies that the frequencies { ω i } are irrationally related.
This might appear far stronger than needed: for one, the full rank condition for Σ ξ can be achieved under far weaker assumptions.
In two dimensions, Σ ξ = 12 I for each of the following special cases: ξ t = [cos( t ) , cos(2 t )] | and ξ t = [cos( t ) , sin( t )] | . However, in each case we can construct a second order polynomial h : R 2 → R such that h ( ξ t ) = 0 for all t , so that the excitation implied by the full rank condition is lost through a simple nonlinearity.
This is only the ﬁrst sign of trouble.
We consider next a numerical example designed with four positive frequencies that are linearly independent over the rationals, denoted [ ω 1 , ω 2 , ω 3 , ω 4 ] .
Consider the linear QSA ODE,
[Equation]
with A ◦ t = (cid:20)
[Equation]
sin( ω 1 t ) sin( ω 2 t )
[Equation]
cos
[Equation]
and A ∗ Hurwitz.
We have s f ( θ )
=
A ∗ θ and hence θ ∗ = 0 .
To ﬁt the notation (9) we must take ξ t ∈ R 6 , which can be chosen so that Σ ξ = 12 I .
The expression ¯ Y ∗ = − 20[2 /ω 1 , 1 /ω 4 ] | follows from (20).
The conclusions of Thm.
2.3 are illustrated in Fig.
The model was simulated with A ∗ = − 0 . 8 I and frequencies [ π , √ 3 , 4 , √ 5] / 5 . The QSA ODEs were constructed using three values of the parameter in the gain process, ρ ∈ { 0 . 6 , 0 . 7 , 0 . 8 } . The plots shown on the left in Fig. 2 were obtained using PR averaging with T 0 =
(1 − 1 /κ )
T and κ = 4 . The convergence rate is O ( T − ρ ) because ¯ Y ∗ 6 = 0 (recall (21b)). FB ﬁltering achieves the convergence rate of O ( T − 2 ρ ) , for each value of ρ tested, as predicted by Thm. 2.2.
Gradient-Free Optimization
We now turn to applications to gradient-free optimization (GFO).
It is assumed that the objective Γ : R d → R is C 2 and that it has a unique minimizer denoted θ opt , and that Σ ξ := h ξξ | i is full rank.
3.1 Gradient-Free Optimization and QSA
Two approaches are considered in the following: For ε > 0 and each Θ 0 ∈ R d ,
[Equation] [Equation]
The mean vector ﬁelds s f 1Q , s f 2Q are identical, provided the probing signal is symmetric:
Proposition 3.1.
Suppose (QSA1)-(QSA5) hold, that Γ is C 2 with unique minimizer θ opt .
Assume moreover that the probing signal is of the form (10) with m = d and distinct frequencies { ω i } .
Then, the average vector ﬁelds for 1 qSGD and 2 qSGD are equal:
[Equation]
For the special case of a strongly convex objective Γ , bounds on bias of 2 qSGD are well known [44].
An application of Prop. 3.1 implies that the bias for 1 qSGD is identical:
Corollary 3.2.
Suppose the assumptions of Prop. 3.1 hold, and Γ is strongly convex.
Then, k θ ∗ − θ opt k ≤ O ( ε 2 ) for either 1 qSGD or 2 qSGD
[Figure] 3.2 Numerical Examples
The numerical results surveyed here and in the supplementary material used a crude approximation of the QSA ODE: an Euler scheme with sampling interval of 1 sec.
It is found that the results remain consistent with theory for moderate dimension.
A metric for testing this theory was via the empirical covariance, obtained using M ≥ 50 independent runs to obtain Θ iT : { 1 ≤ i ≤ M } , and then
[Equation]
Thm. 2.2 and Thm. 2.3 predict that the root mean square error T 2 ρ q tr (Σ T ) is bounded in T , subject to conditions.
[Figure]
Rastrigin: The algorithms were tested in this standard benchmark [47], for which a plot of the objective can be found on the upper left in Fig. In each qSGD
ODE the values ε = 0 .
25 and ρ = 0 . 85 were used, with κ = 5 in
PR averaging.
In the SPSA stochastic counterparts the exploration sequence { ζ n } was chosen i.i.d. and zero-mean, with values in {±√ 2 } (these values were chosen so that the covariance matrices for both SPSA and qSGD would equal).
Further details on the experiment are postponed to the supplementary material.
As predicted by theory, the root-MSE T 2 ρ tr ( s Σ T ) 12 is bounded in T when using FB ﬁltering or PR averaging: see the plot on the lower left in Fig.
Histograms are shown only for 1 qSGD and 1 SPSA using PR averaging.
The results obtained for 1 qSGD with FB ﬁltering were similar.
Results obtained using 2 qSGD and 2 SPSA were similar qualitatively.
Outliers, identiﬁed with Matlab’s isoutlier function, are excluded in these plots.
Outliers were found in 20% of the independent runs for 1 SPSA, and none for 1 qSGD.
Outliers for deterministic algorithms were observed in other experiments, but fewer than in their stochastic counterparts.
Fig.
6 shows part of the trajectory of { Θ T } for a short run.
We see that { Θ T } oscillates between saddle points and local extrema before settling around the minimizer θ opt = 0 near the end of the run.
The corresponding PR estimates very closely approximate θ opt for the ﬁnal 33% of the run.
[Figure] Ackley: This is another standard benchmark [47]. Its objective function is shown on the right in
Fig.
7 in the special case d = 2 .
Fig.
7 shows the scaled root-MSE for dimensions d = 2 and d = 30 with ρ = 0 . 85 .
Both PR averaging and FB ﬁltering are successful in achieving O
( T − 2 ρ ) convergence rates.
We observe much larger values of tr ( s Σ T ) 12 at the end of the run for d = 30 .
It is possible that better results will be improved by with a different choice of probing signal ξ . [Figure] 1   - -1 -2Γ( θ )
[Figure]
Three-Hump Camel
: This is another standard benchmark [47], whose objective function is plotted on the right in Fig.
Results from experiments using PR averaging, surveyed in Section C.2, show that the root-MSE is similar to what is obtained for the Ackley objective with d = 2 .
In this example the value Γ ( Θ PR t ) converges rapidly towards its optimal value of zero.
Conclusions
While it is convenient to design exploration around i.i.d.
signals, and this approach opens the doors to many powerful tools from probability theory, we have shown that deterministic “noise” has signiﬁcant beneﬁts.
Convergence rates can be accelerated dramatically provided the algorithm and deterministic probing signals are chosen with care.
There are many avenues for future research: 4 The impact of dimension on convergence rate appears to be understood for QMC [1].
We currently do not know how to extend this theory to QSA because our analysis is rooted in properties of the K -dimensional clock process Φ , which is far removed from the QMC setting.
[-Future reﬁnements of these algorithms will bemotivated by these shortcomings.
-] The optimal convergence rate for QSA is unknown for the approaches described in this paper
and we currently lack universal bounds that are not restricted to a particular algorithm. (i) Can we improve the upper bounds for PR averaging and the FB algorithm? It may be possible to achieve the bound O ( T − 1 − ρ ) , rather than O ( T − 2 ρ ) in current theory. (ii) Can we ﬁnd fundamental bounds for any algorithm?
We haven’t ruled out the creation of an algorithm with rate of convergence of order O
( T − 10 ) . This question is motivated by a long history of success in the [-rate O ( T − 1 − ρ ) is conjectured based on recent analysis.
-]stochastic approximation literature [10, 39, 35, 36, 37]
4 It is not clear that the constraint ρ > 12 is required in this deterministic setting, and may be removed if we can improve the bounds in our analysis.
We have found in some experiments that the use of PR averaging results in very fast convergence even when this constraint is violated.
4 Under what conditions is q tr (Σ T ) strictly smaller using the FB algorithm as compared to PR averaging? We ﬁnd in experiments that the FB algorithm usually outperforms averaging, but we have yet to ﬁnd tools to obtain bounds that are rich enough to compare the two approaches. Acknowledgements
Many thanks to Profs. Yann Bugeaud (U. Strasbourg) and Gisbert Wüstholz (ETH) for guidance on the literature surrounding Baker’s Theorem. Financial support from ARO award W911NF2010055 and National Science Foundation award EPCN 1935389 is gratefully acknowledged