Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Functions
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We study the performance of Stochastic Cubic Regularized Newton (SCRN) on aclass of functions satisfying gradient dominance property which holds in a widerange of applications in machine learning and signal processing.
This conditionensures that any ﬁrst-order stationary point is a global optimum.
We prove that
SCRN improves the best-known sample complexity of stochastic gradient descentin achieving ✏ -global optimum by a factor of O ( ✏  1 / 2 ) .
Even under a weak versionof gradient dominance property, which is applicable to policy-based reinforcement learning (RL), SCRN achieves the same improvement over stochastic policygradient methods.
Additionally, we show that the sample complexity of SCRNcan be improved by a factor of O ( ✏  1 / 2 ) using a variance reduction method withtime-varying batch sizes.
Experimental results in various RL settings showcase theremarkable performance of SCRN compared to ﬁrst-order methods
Introduction
Consider the following unconstrained stochastic non-convex optimization problem:
[Equation]
where the random variable ⇠ is sampled from an underlying distribution P ⇠ .
In order to optimize theobjective function F ( x ) , we have access to the ﬁrst and second derivatives of stochastic functionf ( x , ⇠ ) .
The above optimization problem covers a wide range of problems, from the ofﬂine settingwhere the objective function is minimized over a ﬁxed number of samples, to the online setting wherethe samples are drawn sequentially.
In the deterministic case (where we have access to the derivatives of F ( x ) ), the gradient descent (GD)algorithm in the non-convex setting only guarantees convergence to a ﬁrst-order stationary point(FOSP) (i.e., a point x such that kr F ( x ) k = 0 ), which can be a local minimum, a local maximum, ora saddle point.
In contrast, second-order methods accessing the Hessian of F (or Hessian of f ( x , ⇠ )in the stochastic setting) can exploit the curvature information to effectively escape saddle points andconverge to a second-order stationary point (SOSP)
(i.e., such that kr F ( x ) k = 0 , r 2 F ( x ) ⌫ 0 ).
In their seminal work, Nesterov and Polyak [25] proposed the so-called cubic-regularized Newton(CRN) algorithm which exploits Hessian information and globally converges to a SOSP at a sub-linearrate of O (1 /k 2 / 3 ) , where k is the number of iterations.
In recent years, the performance of stochastic CRN (SCRN) for general non-convex functions hasbeen the focus of several studies (for more details, see the related work in Section 1.1).
A variancereduced version of SCRN [3] can ﬁnd ( ✏ ,  ) -SOSP
(i.e., a point x such that kr F ( x ) k  ✏ andr 2 F ( x ) ⌫   I ) with the sample complexity of ˜ O ( ✏  3 ) .
Moreover, this rate is optimal for achieving✏ -approximate FOSP (i.e., a point x such that kr F ( x ) k  ✏ ) and it cannot be improved using anystochastic p -th order methods for p  2 [3].
[Table]
Nesterov and Polyak [25] studied CRN under the gradient dominance property (See Assumption2).
Under Assumption 2 with ↵ = 2 , they showed that iterates of F ( x t )  min x F ( x ) converges tozero super-linearly.
The case of ↵ = 2 (commonly called Polyak- Ł ojasiewicz (PL) condition) wasoriginally introduced by Polyak in [28], who showed that GD achieves linear convergence rate.
Thegradient dominance property or its weak variations are satisﬁed in a quite wide range of machinelearning applications such as neural networks with one hidden layer [18] or ResNet with linearactivation [12] (for more details, see Section 1.1).
A particularly important application of weakversion of gradient dominance property with ↵ = {+
+} (Assumption 6) is in policy-based reinforcementlearning (RL) [41].
Khaled et al.
[16] showed that under PL condition, the stochastic GD (SGD) with time-varyingstep-size returns a point ˆ x with a sample complexity of O (1 / ✏ ) , to reach E
[ F (ˆ x )]  min x F ( x )  ✏ .
Furthermore, the dependency of the sample complexity of SGD on ✏ is optimal [26].
Recently,
Fontaine et al.
[10] obtained a sample complexity of O ( ✏  4 ↵ +1 ) for SGD under gradient dominanceproperty with 1  ↵  2 .
This shows that the worst sample complexity occurs for ↵ = 1 , which isprecisely the value of ↵ that ﬁnds important applications in policy-based RL.
Indeed, under a weakversion of gradient dominance property with ↵ = 1 , it has been shown that stochastic policy gradient(SPG) converges to the optimum point with a sample complexity of ˜ O ( ✏  3 ) [41, 9].
We know that in the deterministic case, CRN outperforms GD under gradient dominance propertyfor all ↵ 2 [1 , 2] [25, 50] 1 .
Therefore, a natural question that arises is whether this holds true inthe stochastic setting as well?
That is, does SCRN improve upon SGD under gradient dominanceproperty?
Herein, we address this question.
Speciﬁcally, our main contributions are as follows:
• We analyze the sample complexity of SCRN under gradient dominance property for 1  ↵  2 in order to return an ✏ -global stationary point ˆ x satisfying F (ˆ x )  min x F ( x )  ✏
(in expectation or with high probability).
As stated in Table 1, SCRN improves upon the
best-known sample complexity of SGD for all 1  ↵ < 2 .
The largest improvement is for↵ = 1 , and is O ( ✏  0 . 5 ) .
• In the setting of policy-based RL, under the weak version of gradient dominance propertywith ↵ = 1 (Assumption 6), we show that SCRN achieves a sample complexity of ˜ O ( ✏  2 . 5 ) ,improving over the best-known sample complexity of SPG by a factor of ˜ O ( ✏  0 . 5 ) .
• We show that an adaptation of a variance reduced SCRN [47] with time-varying batch sizes further improves the sample complexity of SCRN.
In particular, for ↵ = 1 , the averagesample complexity is reduced to O ( ✏  2 ) .
1.1 Related work
Gradient dominance property and its applications: The gradient dominance property with ↵ = 2(commonly called PL condition) was originally introduced by Polyak in [28].
It was shown by the literature of machine learning [19, 23, 42].
The gradient dominance property is also satisﬁed(sometimes locally rather than globally, and also under distributional assumptions) for the populationrisk in some learning models including neural networks with one hidden layer [18], ResNet withlinear activation [12], and generalized linear model and robust regression [11].
Moreover, in policybased reinforcement learning (RL), a weak version of gradient dominance property with ↵ = 1 (see
Assumption 6) holds for some classes of policies (such as Gaussian policy and log-linear policy).
Variants of cubic regularized Newton method: For non-convex optimization, Nesterov and Polyak[25] proposed the CRN algorithm, which converges to a SOSP with the convergence rate of O (1 /k 2 / 3 )(where k is the number of iterations) by solving a cubic sub-problem in each iteration.
Cartis etal.
[5] presented an adaptive framework for cubic regularization method.
In [17, 39], sub-sampledversions of gradient and Hessian were used in CRN to overcome the computational burden of Hessianmatrix evaluations in high dimensional settings.
In context of stochastic optimization, Tripuraneniet al.
[33] proposed a stochastic cubic regularization algorithm that obtains ✏ -SOSP with samplecomplexity of ˜ O ( ✏  3 . 5 ) .
Arjavani et al.
[3] improved the sample complexity to ˜ O ( ✏  3 ) using variancereduction.
In the convex setting, Song et al.
[30] presented a proximal CRN and its acceleratedversion, and proved a sample complexity of ˜ O ( ✏  2 ) to reach ✏ -global stationary point as long as theapproximated Hessian in each iteration satisﬁes certain properties.
In ﬁnite-sum non-convex setting,
Zhou et al.
[48] proposed an adaptive sub-sampled CRN method that requires ˜ O ( N + N 4 / 5 ✏  3 / 2 )to ﬁnd ✏ -SOSP, where N is the total number of samples.
Sample complexity was further reducedto ˜ O
( N + N 2 / 3 ✏  3 / 2 ) using various variance reduction methods [36, 45, 49].
To the best of ourknowledge, no previous work on analyzing SCRN for gradient-dominant functions exists.
1.2 Notations
We adopt the following notation in the sequel.
Calligraphic letters (e.g., S ) denote spaces.
Upper-casebold letters (e.g., A ) denote matrices, and the lower-case bold letters (e.g., x ) denote vectors.
k · kdenote the ` 2 -norm for vectors and the operator norm for matrices ( k A k :=
max ( A T A ) where max ( X ) is the maximum eigenvalue of matrix X ), respectively.
A ⌫ B indicates that A  B ispositive semi-deﬁnite.
We use the notation O to hide constants, and the notation ˜ O to hide bothconstants and logarithmic factors.
X  1   Y denotes that random variable X is less than or equal torandom variable Y with the probability at least 1   .
Setup
[Figure] Recall the stochastic non-convex optimization problem in (1), where the goal is to minimize theobjective function F ( x ) having access to stochastic gradients
r f ( x , ⇠ ) and stochastic Hessianmatrices r 2 f ( x , ⇠ ) .
We make the following assumption about the objective function in (1).
Assumption 1.
The Hessian of F is Lipschitz continuous with constant L 2 , i.e.,
[Equation]
Consider the empirical estimators g t := 1 n
P
[Equation]
r f ( x t , ⇠ i ) , and H t := 1 n P [Equation]
[Equation]
⇠
[Equation]
)where n 1 and n 2 are the numbers of samples used for estimating the gradient vector and Hessianmatrix, respectively.
Deﬁnition 1 (Total sample complexity) .
Given ✏ ,  > 0 , the total sample complexity is the number ofcalls (queries) of stochastic gradient and stochastic Hessian along the iterations until reaching apoint x that satisﬁes one of the following: (1) for high probability analysis: F ( x )  F ( x ⇤ )
 ✏ withprobability at least 1   ; or (2) for analysis in expectation: E [ F ( x )]  F ( x ⇤ )  ✏ .
Our goal is to study the performance of stochastic cubic regularized Newton (SCRN) for objectivefunctions that satisfy the gradient dominance property.
Algorithm 1 describes the steps in SCRN.
Ateach iteration t , we take batches of stochastic gradient vectors and Hessian matrices (lines 4 and 5)and then solve the following sub-problem to obtain  t (line 6):
[Equation]
We assume that there is an oracle that returns a global solution for this sub-problem (This assumptionwill be relaxed subsequently, see Remark 6).
Finally, we update x t in line 7.
Input: Batch sizes n 1 , n 2 , initial point x 0 , accuracy ✏ , cubic penalty parameter M , maximum number of iterations T 1: t 2: k  0 k = 1 3: while k  t
[Equation]
1 k
[Equation]
p ✏ or t  T do 4: g t 1 n
{+P n 
+}
i
=1 r f ( x t , ⇠ i ) 5:
H t 1 n
P n 2 i =1 r 2 f ( x t , ⇠ i ) 6:  t arg min  2 R d h g t ,  i + 12 h  , H t  i + M 6 k  k 3 7: x t + x t +  t 8: t t +9: end while 10: return x t
SCRN under gradient dominance property
We shall study the performance of SCRN for functions satisfying gradient dominance property,deﬁned as follows.
Assumption 2.
Function F ( x ) satisﬁes gradient dominance property when
[Equation]
where x ⇤ 2 arg min x F ( x ) , ⌧ F > 0 , and ↵ 2 [1 , 2] are two constants.
The case ↵ = 2 is often referred to as PL condition [28, 15].
In this paper, we consider all ↵ ’s inthe interval [1 , 2] .
The gradient dominance property holds for a large class of functions includingsub-analytic functions, logarithm and exponential functions, and semi-algebraic functions.
Thesefunction classes cover some of the most common non-convex objectives used in practice (see relatedwork in Section 1.1).
In the following lemma, we present a recursion inequality that captures the behaviour of the function
F ( x t )  F ( x ⇤ )
at each iteration t for SCRN under gradient dominance property.
Lemma 1.
Assume that function F satisﬁes Assumption 1 (Lipschitz Hessian) and Assumption 2(gradient dominance property) for ↵  1 .
Then the resulting update x t +1 in Algorithm 1 (line 7)after plugging in  t , the solution of sub-problem in (3) , satisﬁes the following:
[Equation]
where C, C g , C H > 0 are constants depending on M, L 2 , and ⌧ F , and deﬁned in (34) .
Due to space limitation, all proofs are moved to the appendix.
In the following, we ﬁrst provide an analysis in expectation of SCRN under gradient dominanceproperty with ↵ 2 [1 , 3 / 2] .
Next, we study the same algorithm for ↵ 2 (3 / 2 , 2] using a highprobability analysis.
3.1 SCRN under gradient dominance property with
↵
We make the following assumption on the stochastic gradients and Hessians.
Assumption 3.
For a given ↵ 2 [1 , 3 / 2] and for each query point x 2 R d :
[Equation]
where  1 and  2 , ↵ are two constants.
Remark 1.
The assumption E [ kr 2 f ( x , ⇠ )  r 2 F ( x ) k 2 ↵ ]   22 , ↵ for 1  ↵  3 / 2 is slightlystronger than the usual assumption
E
[ kr 2 f ( x , ⇠ )  r 2 F ( x ) k 2 ]   22 .
We need this assumptionbecause of the speciﬁc form of the error of Hessian estimator in recursion inequality in (5) .
As a result of assumption in (7) , using a version of matrix moment inequality (See Lemma 3 in Appendix),we show that the dependency of the Hessian sample complexity in Theorem 1 on dimension d is in theorder of log d .
Theorem 1.
Let F ( x ) satisfy Assumptions 1 and 2 for a given ↵ and the stochastic gradient and
Hessian satisfy Assumption 3 for the same ↵ .
Moreover, assume that an exact solver for sub-problem(3) exists.
Then Algorithm 1 outputs a point x T such that E [ F ( x T )]
F ( x ⇤ )  ✏ after T iterations,where(i) if ↵ 2 [1 , 3 / 2) , T
= O ( ✏  3  2 ↵ 2 ↵ ) , with access to the following numbers of samples of thestochastic gradient and Hessian per iteration:
[Equation]
where C 0 H is deﬁned in (38) and depends on log( d ) .(ii)
if ↵ = 3 / 2 , T =
O (log(1 / ✏ )) with the same numbers of samples per iteration as in (35) .
3.2 SCRN under gradient dominance property with ↵ 2 (3 / 2 , 2]
Deﬁnition 2 (Bernstein’s condition for matrices) .
A zero-mean symmetric random matrix X satisﬁesthe Bernstein condition with parameter b > 0 if
[Equation]
where Var( X ) :
= E [ X 2 ]  ( E [ X ]) 2 .
Assumption 4.
We assume that the symmetric version of each centered gradient estimator G ( x , ⇠ ) := 0 1 ⇥g ( x , ⇠ ) T g ( x , ⇠ )
d ⇥ d  where g ( x , ⇠ ) :=
r f ( x , ⇠ )  r F ( x ) and each centered Hessian estimator
H ( x , ⇠ ) :
= r 2 f ( x , ⇠ )  r 2 F ( x ) satisfy Bernstein’s condition (9) with parameters M 1 and M 2 ,respectively.
Remark 2.
It is noteworthy that most previous work analyzing SCRN [33, 49, 36] assumed thatcentered gradient and centered Hessian estimators are bounded, i.e., kr f ( x , ⇠ )  r F ( x ) k 2 a.s. 
M 1 , kr 2 f ( x , ⇠ )  r 2 F ( x ) k a.s.  M 2 .
This is a stronger assumption than Assumption 4 as it implies
Bernstein’s condition (9) for g ( x , ⇠ ) and H ( x , ⇠ ) .
Theorem 2.
Suppose that F ( x ) satisﬁes Assumptions 1, 2, the stochastic gradient and Hessiansatisfy Assumption 3 (with ↵ = 1 ) and Assumption 4, and there exists an exact solver for sub-problem(3) . Then, Algorithm 1, with probability 1   , outputs a solution x T such that F ( x T )  F ( x ⇤ )
 ✏after T
= O (log (log(1 / ✏ ))) iterations with the following numbers of samples for the stochasticgradient and Hessian, respectively per iteration:
[Equation]
where ˜ C
= 1 + ⌧ F
M + L 2 +4 ↵ .
Remark 3.
It is noteworthy that the assumptions in Theorem 2 sufﬁce to obtain the sample complexities in Theorem 1 for 1  ↵  3 / 2 up to a logarithmic factor.
However, the assumptions of Theorem2 are stronger than those in Theorem 1.
Remark 4.
Theorem 2 implies that the sample complexity of Algorithm 1 for ↵ = 2 is
O (log log log(1 / ✏ ) · log(log(1 / ✏ )) / ✏ ) .
We summarized the sample complexity of SCRN in Theorems 1 and 2 and the best-known samplecomplexity of SGD under gradient dominance property in Table 1.
The optimal sample complexityof SGD under gradient dominance property with ↵ = 2 is discussed in details in [16, Section 5.2].
For general case of ↵ 2 [1 , 2] , the sample complexity of SGD under gradient dominance propertywas derived in [10, Theorem 10].
This rate is achieved with a time-varying step-size.
In the fourthcolumn of Table 1, we provide the improvement of the sample complexity of SCRN with respect tothat of SGD.
The improvement is O  ✏  1 / (2 ↵ )  for ↵ 2 [1 , 3 / 2) and is ˜ O
( ✏  2 / ↵ +1 ) for ↵ 2 [3 / 2 , 2] ,which are decreasing functions of ↵ .
The largest improvement is for ↵ = 1 and is O ( ✏  0 . 5 ) .
Remark 5.
The special case ↵ = 2 generalizes strong convexity.
For strongly convex objectives,the dependence of sample complexity of SGD on ⌧ F is O ( ⌧ 2 F / ✏ ) [16, Corollary 2] while the samplecomplexity of SCRN is O ( ⌧ 7 / 4 F / ✏ ) .
This is an improvement by a factor of ⌧ 1 / 4 F .
In Appendix A.2.3, weprovide simulation results comparing the performance of SCRN and SGD over synthetic functions byvarying ↵ and ⌧ F .
Remark 6.
In our analysis, we assumed that we have access to the exact solution of sub-problemin (3) .
Although no closed-form solution nor exact solver exists for this sub-problem, there arealgorithms that approximate the exact solution with high probability [2, 4].
We emphasize thatsolving the sub-problem in (3) requires extra computation, but extra gradient and Hessian queriesare not needed.
In particular, Carmon and Duchi [4] proposed a perturbed GD-based algorithm thatreturns an approximate solution ˜
t such that m t ( ˜  t )  1   0 m t (  t ) + ✏ 0 with O (log(1 /  0 ) / ✏ 0 )iterations, for any given  0 , ✏ 0 > 0 .
In Appendix A.2.2, we prove that Theorems 1 and 2 are stilltrue if we were to use an inexact sub-solver which returned an approximate solution ˜
t such thatkr m t ( ˜  t ) k  ✏ 1 / ↵ .
Moreover, under some mild assumptions (same as those in [4]), a GD-basedalgorithm indeed returns such a solution in O ( ✏  2 / ↵ ) iterations.
Moreover, in the iterations of GDbased algorithm, instead of directly computing the Hessian matrix (which could be computationallyexpensive in high-dimensions), we could compute Hessian-vector products by running Pearlmutter’salgorithm [27].
Thus, the total computational complexity of evaluating gradients and Hessian vectorproducts of SCRN is in the order of O ( dT ✏  3 / ↵ ) while the one of SGD is O ( d ✏  4 / ↵ +1 ) , where T isthe number of iterations of SCRN and d is the dimension of x . For ↵ 2 [1 , 3 / 2) , it can be seen thatthe computational complexity of SGD is less than the one of SCRN by a factor of O ( ✏  1 / (2 ↵ ) ) . For↵ 2 [3 / 2 , 2] , this factor is ˜ O ( ✏  (1  1 / ↵ ) ) .
3.3 Further improvements
In our analysis of SCRN, we set the batch sizes such that the stochastic error terms C g kr F ( x t )  g t k ↵and C H kr 2 F ( x t )
H t k 2 ↵ in (5) are in the order of ✏ (either in expectation or with high probability).
However, for 1  ↵ < 3 / 2 , it is just needed to make sure that the error terms at iteration t are
O ( t  (2 ↵ ) / (3  2 ↵ ) ) , which equals the convergence rate of the function values F ( x t )  F ( x ⇤ )
(see
Lemma 10 in Appendix A.3).
Moreover, as stated in the following theorem, incorporating timevarying batch sizes in conjunction with variance reduction improves sample complexity results.
Assumption 5.
We assume that f ( x , ⇠ ) satisﬁes L
1 -average smoothness and L 0 2 -average Hessian Lipschitz continuity, i.e., E [ kr f ( x , ⇠ )  r f
( y , ⇠ ) k 2 ]  L 0 21 k x  y k 2 and E [ kr 2 f ( x , ⇠ ) r 2 f
( y , ⇠ ) k 2 ]  L 0 22 k x  y k 2 for all x , y 2 R d .
Theorem 3.
Suppose that F ( x ) satisﬁes the gradient dominance property with ↵ = 1 .
Assume that
Assumptions 1, 3 (for ↵ = 1 ), and 5 hold. Then variance-reduced SCRN (See Algorithm 2 in Appendix A.3) achieves ✏ -global stationary point in expectation by making O ( ✏  2 ) stochastic gradients and O ( ✏  1 ) stochastic Hessian queries on average, and the iteration complexity is O ( ✏  1 / 2 ) .
SCRN under weak gradient dominance property in RL Setting
In this section, we showcase practical relevance of our result in Section 3.1 by applying SCRN tomodel-free RL.
Speciﬁcally, in Theorem 4, we prove that as long as the expected return satisﬁes theweak version of gradient dominance property (Assumption 6), SCRN improves upon the best-knownsample complexity of stochastic policy gradient (SPG) [41] by a factor of O (1 / p ✏ ) (see Appendix
A.4.1 for the related work).
4.1 RL Setup
Consider a discrete Markov decision process (MDP) M
= ( S , A , P, R, ⇢ ,  ) , where S is the statespace and A is the action space.
P ( s 0 | s, a ) denotes the probability of state transition from s to s 0after taking action a and R ( · , · ) : S ⇥ A !
[  R max , R max ] is a bounded reward function, where R max is a positive scalar.
⇢ represents the initial distribution on state space S and  2 (0 , 1) is thediscount factor.
The parametric policy ⇡ ✓ is a probability distribution over S ⇥
A with parameter ✓ 2 R d , and 236 ⇡ ✓ ( a | s ) denotes the probability of taking action a at a given state s .
Let ⌧ = { s t , a t } t  0 ⇠ p ( ⌧ | ⇡ ✓ ) 237 be a trajectory generated by the policy ⇡ ✓ , where p ( ⌧ | ⇡ ✓ ) :
= ⇢ ( s 0 )
Q 1 t =0 ⇡ ✓ ( a t | s t ) P ( s t +1 | s t , a t ) .
The expected return of ⇡ is deﬁned as J ( ⇡ ✓ ) :
= E ⌧ ⇠ p ( ·| ⇡ ✓ ) [
P 1 t =
t R ( s t , a t )] .
In the sequel, weconsider a set of parameterized policies { ⇡ ✓ : ✓ 2 R d } , with the assumption that ⇡ ✓ is differentiablewith respect to ✓ .
For ease of presentation, we denote J ( ⇡ ✓ ) by J ( ✓ ) .
The goal of policy-based RL is to ﬁnd ✓
⇤ = arg max ✓ J ( ✓ ) .
However, in many cases, J ( ✓ ) is anon-concave function and instead we settle for obtaining an ✏ -FOSP, ˆ ✓ , such that kr J (ˆ ✓ ) k  ✏ .
Itcan be shown that: r J ( ✓ ) =
E [
P [Equation]
⇡ ✓ ( [Equation] h | s h )] .
In practice, the fullgradient cannot be computed due to the inﬁnite horizon length.
Instead, it is commonly truncatedto a length H horizon as follows.
r J H ( ✓ ) =
E [
P
[Equation]
h ( ⌧
[Equation]
r log ⇡ ✓ ( a h | s h )] , where h ( ⌧ [Equation] =
P
[Equation]
Assume that we sample m trajectories ⌧ i =
{ s it , a it } t  0 , 1  i  m , and then compute ˆ r m J ( ✓ ) =1 m
P m i =
P
H  1 h
= h ( ⌧ i ) r log ⇡ ✓ ( a i h | s i h ) , which is an unbiased estimator for r J H ( ✓ ) .
The vanilla
SPG method is based on the following update: ✓ ✓ + ⌘ ˆ r m J ( ✓ ) , where ⌘ is the learning rate.
It can be shown that the Hessian matrix of J H ( ✓ ) can be obtained as follows [29, Appendix 7.2]:
[Equation]
where  ( ✓ ; ⌧ )
[Equation]
[Equation]
⇡ ✓ ( a h | s h ) .
As a result, for trajectories ⌧ i = { s it , a it } t  0 , 1  i  m ,
[Equation]
✓
[Equation]
⌧ i ) is an unbiased estimator of Hessianmatrix r 2 J H ( ✓ ) .
4.2 Sample Complexity of SCRN
In our analysis, we consider the recently introduced relaxed weak gradient dominance property with↵ = 1 [41].
Assumption 6 (Weak gradient dominance property with ↵ = 1 ) .
J satisﬁes the weak gradientdominance property if for all ✓ 2 R d , there exist ⌧ J > 0 and ✏ 0 > 0 such that
[Equation]
(12) where J ⇤ := max ✓ J ( ✓ ) .
Remark 7.
Two commonly made assumptions in the literature: non-degenerate Fisher matrix [20, 9]and transferred compatible function approximation error [35, 1] imply Assumption 6.
See Appendix
A.4.1 for more details.
Further, we also make Lipschitz and smooth policy (LS) assumptions which are widely adopted inthe analysis of vanilla policy gradient (PG) [46] as well as variance reduced PG methods, e.g. in [29].
Assumption 7 (LS) .
There exist constants G 1 , G 2 > 0 such that for every state s 2 S , the gradientand Hessian of log ⇡ ✓ ( ·| s ) satisfy kr ✓ log ⇡ ✓ ( a | s ) k  G 1 and kr 2 ✓ log ⇡ ✓ ( a | s ) k  G 2 .
Lemma 2.
Under Assumption 7, we have kr J ( ✓ ) r J H ( ✓ ) k  D g  H and kr 2 J ( ✓ ) r 2 J H ( ✓ ) k 
D H  H , where D g =
G 1 R
[Equation]
1
[Equation]
Assumption 8 (Lipschitz Hessian) .
There exists a constant ¯ L 2 such that the Hessian of log ⇡ ✓ ( a | s )satisﬁes
[Equation]
The Lipschitz Hessian assumption is commonly used to ﬁnd SOSP in policy gradient algorithms [40].
For the Gaussian policy (133), r 2 log ⇡ ✓ ( a | s ) reduces to the matrix   ( s )  ( s ) T /  2 , which is aconstant function of ✓ and thus satisﬁes condition (13).
Soft-max policy also satisﬁes this assumption(See appendix A.4.4).
Theorem 4.
For a policy ⇡ ✓ satisfying Assumptions 7, 8, and the corresponding objective function
J ( ✓ ) satisfying Assumption 6, SCRN outputs the solution ✓ T such that J ⇤  E
[ J ( ✓ T )]  ✏ + ✏ 0 and [Figure] [Figure] the sample complexity (the number of observed state-action pairs) is: T ⇥ m ⇥ H =
˜ O ( ✏  2 . 5 ) for✏ 0 = 0 and T ⇥ m ⇥ H =
˜ O ( ✏  0 .
5 ✏ 0 2 ) for ✏ 0 > 0 .
Remark 8.
Under weak gradient dominance property with ↵ = 1 (Assumption 6), it has been shownthat the sample complexity of SPG is ˜ O ( ✏  3 ) in case of ✏ 0 = 0 and ˜ O
( ✏  1 ✏ 0 2 ) in case of ✏ 0 > 0 [41,
Theorem C.1].
Therefore, SCRN improves upon the best-known sample complexity of SPG in bothcases ✏ 0 = 0 and ✏ 0 > 0 by a factor of O ( ✏  0 . 5 ) .
Remark 9.
Having access to exact gradient and Hessian (the deterministic case), under Assumption6, PG algorithm achieves global convergence (i.e., J ⇤  J ( ✓ T )  ✏ ) with ˜ O
( ✏  1 ) iterations [41]while CRN requires ˜ O ( ✏  0 . 5 ) iterations.
Remark 10.
Under the same assumptions as in Theorem 4 and Assumption 5 and bounded varianceof importance sampling weights (See Assumption 15), a variance reduced version of SCRN (See Algorithm 3) achieves global convergence (i.e., J ⇤  E
[ J ( ✓ T )]  ✏ ) with a sample complexity of O ( ✏  2 ) .
See Appendix A.4.6 for a proof.
Experiments
In this section, we evaluate the performance of SCRN in the two following RL settings.
First, weconsider grid world environments with ﬁnite state and action spaces, and next some robotic controltasks with continuous state and action spaces.
The details of the implementations for all methods andsome additional experiments appear in the appendix and the codes are available in the supplementarymaterial.
Environments with ﬁnite state and action spaces: We consider two grid world environments in ourexperiments: cliff walking [31, Example 6.6], and random mazes [51].
In cliff walking, the agent’saim is to reach a goal state from a start state, avoiding a region of cells called “cliff”.
The episode isterminated if the agent enters the cliff region, or the number of steps exceeds 100 without reachingthe goal.
Moreover, we consider a soft-max tabular policy in the experiments of this part.
Indeed, ithas been shown that a variant of gradient dominance property with ↵ = 1 holds for soft-max tabularpolicy in environments with ﬁnite state and action spaces [22]. [Figure]
We compare SCRN with two existing ﬁrst-order methods: vanilla SPG and REINFORCE [37].
Forthe ﬁrst-order methods, we use a time-varying learning rate and tune the parameters.
To improvethe performance of the ﬁrst-order methods, we also add an entropy regularization term to the rewardfunction.
In Fig.
1, the average length of paths traversed by the agent and the average episode return aredepicted against the number of episodes for each method.
The results are averaged over 64 instancesand the shaded region shows the 90% conﬁdence interval.
As can be seen in Fig.
1 (a), all thealgorithms have a phase at the beginning during which the agent falls off the cliff in most episodesand the average length of paths are small.
Then, the agent learns to avoid the cliff but could still notreach the goal in most cases.
Finally, it ﬁnds a path to the goal and tries to reduce the path length.
Note that SCRN ﬁnds the path very quickly and signiﬁcantly outperforms the other algorithms.
Infact, SPG and REINFORCE get stuck in the start state for some period of time, while SCRN easilyescapes the ﬂat plateau (for more details, please see the demonstrations in the supplementary ﬁle).
The performance of SPG and REINFORCE improves with entropy regularization, but SCRN stilloutperforms them.
Moreover, SPG and REINFORCE fail to reach the goal in some instances while
SCRN is successful in almost all instances.
In the captions of ﬁgures, for each algorithm, we alsoprovide the percentage of instances in which the agent reached the goal.
Please note that thesepercentages are obtained based on the parameters from the last update of each algorithm.
Additionally, we studied the performance of the three aforementioned algorithms on a random mazeand a random shape maze [51].
In the random shape maze, random shape blocks are placed on a gridand the agent tries to reach the goal state ﬁnding the shortest path.
As shown in Fig.
2, SCRN againoutperforms the ﬁrst-order methods in both environments.
In Appendix A.5, we also provided resultsfor SPG and REINFROCE with entropy regularization which have slightly better performance.
Environments with continuous state and action spaces: We consider the following control tasksin MuJoCo simulator [32]: Walker, Humanoid, Reacher, and HalfCheetah.
We compare SCRNwith ﬁrst-order methods such as REINFORCE, and two state-of-the-art representatives of variancereduced PG methods, HAPG [29] and MBPG [13], both with guaranteed convergence to ✏ -FOSPin general non-convex settings.
HAPG uses second order information (Hessian vector products)for variance reduction and MBPG is a recent work based on STORM, a batch-free state-of-the-artvariance reduction approach [8].
We report average episode return against system probes as our performance measure.
That is, thenumber of observed state-action pairs (see Fig. 3).
At each point, we run the trained policy 10 timesand compute the empirical estimate of the mean and the 90% conﬁdence interval of the episode return.
As seen in Fig.
3, SCRN outperforms the other methods, especially in more complex environmentssuch as HalfCheetah and Humanoid.
In Appendix A.5, we also provided results for variance-reduced
SCRN which improves upon SCRN in Humanoid and Reacher environments.
Conclusion
We studied performance of SCRN for objectives satisfying the gradient dominance property for1  ↵  2 , which holds in various machine learning applications.
We showed that SCRN improvesthe best-known sample complexity of SGD.
The largest improvement is in the case of ↵ = 1 .
Indeed,a weak version of gradient dominance for ↵ = 1 is satisﬁed in some policy-based RL settings.
In the
RL setting, we showed that SCRN achieves the same improvement over SPG under the weak versionof gradient dominance property for ↵ = 1 .
Moreover, for ↵ = 1 , the average sample complexity of
SCRN can be reduced by utilizing a variance reduction method with time-varying batch sizes.