G LOBAL C ONVERGENCE OF P OLICY G RADIENT M ETHODS FOR L INEARIZED C ONTROL P ROBLEMS
Anonymous authors
Paper under double-blind review
A BSTRACT
Direct policy gradient methods for reinforcement learning and continuous control problems are a popular approach for a variety of reasons: 1) they are easy to implement without explicit knowledge of the underlying model 2) they are an “end-to-end” approach, directly optimizing the performance metric of interest 3) they inherently allow for richly parameterized policies.
A notable drawback is that even in the most basic continuous control problem (that of linear quadratic regulators), these methods must solve a non-convex optimization problem, where little is understood about their efﬁciency from both computational and statistical perspectives.
In contrast, system identiﬁcation and model based planning in optimal control theory have a much more solid theoretical footing, where much is known with regards to their computational and statistical properties.
This work bridges this gap showing that (model free) policy gradient methods globally converge to the optimal solution and are efﬁcient (polynomially so in relevant problem dependent quantities) with regards to their sample and computational complexities.
I NTRODUCTION
Recent years have seen major advances in the control of uncertain dynamical systems using reinforcement learning and data-driven approaches; examples range from allowing robots to perform more sophisticated controls tasks such as robotic hand manipulation (Tassa et al., 2012;
Al Borno et al., 2013; Kumar et al., 2016; Levine et al., 2016;
Tobin et al., 2017; Rajeswaran et al., 2017a), to sequential decision making in game domains, e.g. AlphaGo (Silver et al., 2016) and Atari game playing (Mnih et al., 2015).
Deep reinforcement learning (DeepRL) are becoming increasingly popular for tackling such challenging sequential decision making problems.
Many of these successes have relied on sampling based reinforcement learning algorithms such as policy gradient methods, including the DeepRL approaches; here, there is little theoretical understanding of their efﬁciency, either from a statistical or a computational perspective.
In contrast, control theory (optimal and adaptive control) has a rich body of tools, with provable guarantees, for related sequential decision making problems, particularly those that involve continuous control.
These latter techniques are often model-based — they estimate an explicit dynamical model ﬁrst (e.g. system identiﬁcation) and then design optimal controllers.
This work builds bridges between these two lines of work, namely, between optimal control theory and sample based reinforcement learning methods, using ideas from mathematical optimization.
1. T HE OPTIMAL CONTROL PROBLEM
In the standard optimal control problem, the dynamics model f t , where f t is speciﬁed as
[Equation]
, maps a state x t ∈ R d , a control (the action) u t ∈ R k , and a disturbance w t , to the next state x t +1 ∈ R d .
The objective is to ﬁnd the control input u t which minimizes the long term cost, minimize
T (cid:88) t
[Equation]
c t
[Equation]
x t , u t ) such that x t +1 =
f t ( x t , u t , w t ) .
Here the u t are allowed to depend on the history of observed states.
In practice, this is often solved by considering the linearized control (sub-)problem where the dynamics are approximated by
[Equation]
and the costs are approximated by a quadratic function in x t and u t , e.g. (Todorov & Li, 2004).
This work considers an important special case: the time homogenous, inﬁnite horizon problem referred to as the linear quadratic regulator (LQR) problem.
The results herein can also be extended to the ﬁnite horizon, time in-homogenous setting, discussed in Section 5.
In the LQR problem, the objective is minimize
[Equation]
such that
[Equation]
where initial state x 0 ∼ D is assumed to be randomly distributed according to distribution D ; the matrices A ∈ R d × d and B ∈ R d × k are referred to as system (or transition) matrices; Q ∈ R d × d and R ∈ R k × k are both positive deﬁnite matrices that parameterize the quadratic costs.
For clarity, this work does not consider a noise disturbance but only a random initial state.
The importance of (some) randomization for analyzing direct methods is discussed in Section 3.
Throughout, assume that A and B are such that the optimal cost is ﬁnite (for example, the controllability of the pair ( A, B ) would ensure this).
Optimal control theory (Anderson & Moore, 1990;
Evans, 2005; Bertsekas, 2011; 2017) shows that the optimal control input can be written as a linear function in the state,
[Equation]
where K ∗ ∈ R k × d .
Planning with a known model.
Planning can be achieved by solving the algebraic Riccati equation,
[Equation]
for a positive deﬁnite matrix P which parameterizes the “cost-to-go” (the optimal cost from a state going forward).
The optimal control gain is then given as:
[Equation]
There are both algebraic solution methods to ﬁnd P and (convex)
SDP formulations to solve for P .
More broadly, even though there are convex formulations for planning, these formulations: 1) do not directly parameterize the policy 2) they are not “end-to-end” approaches in that they are not directly optimizing the cost function of interest and 3) it is not immediately clear how to utilize these approaches in the model-free setting, where the agent only has simulation access.
These formulations are discussed in Section A, where there is a discussion of how the standard SDP formulation is not a direct method that minizes the cost over the set of feasible policies.
1. C ONTRIBUTIONS OF THIS WORK
Even in the most basic case of the standard linear quadratic regulator model, little is understood as to how direct (model-free) policy gradient methods fare.
This work provides rigorous guarantees, showing that, while in fact the approach is a non-convex one, directly using (model free) local search methods leads to ﬁnding the globally optimal policy.
The main contributions are as follows:
• (Exact case) Even with access to exact gradient evaluation, little is understood about whether or not convergence to the optimal policy occurs, even in the limit, due to the non-convexity in the problem.
This work shows that global convergence does indeed occur (and does so efﬁciently) for local search based methods.
• (Model free case)
Without a model, this work shows how one can use simulated trajectories (as opposed to having knowledge of the model) in a stochastic policy gradient method where provable convergence to a globally optimal policy is guaranteed, with (polynomially) efﬁcient computational and sample complexities.
• (The natural policy gradient) Natural policy gradient methods (Kakade, 2001) — and related algorithms such as Trust Region Policy Optimization (Schulman et al., 2015) and the natural actor critic (Peters & Schaal, 2007) — are some of the most widely used and effective policy gradient methods (see Duan et al.
While many results argue in favor of this method based on either information geometry (Kakade, 2001; Bagnell & Schneider, 2003) or based on connections to actor-critic methods (Deisenroth et al., 2013), these results do not provably show an improved convergence rate.
This work is the ﬁrst to provide a guarantee that the natural gradient method enjoys a considerably improved convergence rate over its naive gradient counterpart.
More broadly, the techniques in this work merge ideas from optimal control theory, mathematical (and zeroth order) optimization, and sample based reinforcement learning methods.
These techniques may ultimately help in improving upon the existing set of algorithms, addressing issues such as variance reduction or improving upon the natural policy gradient method (with, say, a GaussNewton method).
The Discussion touches upon some of these issues.
1. R ELATED WORK
In the reinforcement learning setting, the model is unknown, and the agent must learn to act through its interactions with the environment.
Here, solution concepts are typically divided into: modelbased approaches, where the agent attempts to learn a model of the world, and model-free approaches, where the agent directly learns to act and does not explicitly learn a model of the world.
The related work on provably learning LQRs is reviewed from this perspective.
Model-based learning approaches.
In the context of LQRs, the agent attempts to learn the dynamics of “the plant” (i.e. the model) and then plans, using this model, for control synthesis.
Here, the classical approach is to learn the model with subspace identiﬁcation (Ljung, 1999).
Fiechter (1994) provides a provable learning (and non-asymptotic) result, where the quality of the policy obtained is shown to be near optimal (efﬁciency is in terms of the persistence of the training data and the controllability Gramian).
Abbasi-Yadkori & Szepesv ´ ari (2011) also provides provable, nonasymptotic learning results (in a regret context), using a bandit algorithm that achieves lower sample complexity (by balancing exploration-exploitation more effectively); the computational efﬁciency of this approach is less clear.
More recently, Dean et al.
(2017) expands on an explicit system identiﬁcation process, where a robust control synthesis procedure is adopted that relies on a coarse model of the plant matrices ( A and B are estimated up to some accuracy level, naturally leading to a “robust control” setup).
Arguably, this is the most general (and non-asymptotic) result, that is efﬁcient from both a statistical perspective (computationally, the method works with a ﬁnite horizon to approximate the inﬁnite horizon).
This result only needs the plant to be controllable; the work herein needs the stronger assumption that the initial policy in the local search procedure is a stable controller (an assumption which may be inherent to local search procedures, discussed in Section 5).
Model-free learning approaches.
Model-free approaches that do not rely on an explicit system identiﬁcation step typically either: 1) estimate value functions (or state-action values) through Monte
Carlo simulation which are then used in some approximate dynamic programming variant (Bertsekas, 2011) or 2) directly optimize a (parameterized) policy, also through Monte Carlo simulation.
Model-free approaches for learning optimal controllers is not well understood, from a theoretical perspective.
Here, Bradtke et al.
(1994) provides an asymptotic learnability result using a value function approach, namely Q -learning.
P RELIMINARIES AND B ACKGROUND 2. E XACT G RADIENT D ESCENT
This work seeks to characterize the behavior of (direct) policy gradient methods, where the policy is linearly parameterized, as speciﬁed by a matrix K ∈ R k × d which generates the controls:
[Equation]
for t ≥ 0 .
The cost of this K is denoted as:
[Equation]
where { x t , u t } is the trajectory induced by following K , starting with x 0 ∼ D .
The importance of (some) randomization, either in x 0 or noise through having a disturbance, for analyzing gradient methods is discussed in Section 3.
Here, K ∗ is a minimizer of C ( · ) .
Gradient descent on C ( K ) , with a ﬁxed stepsize η , follows the update rule:
[Equation]
It is helpful to explicitly write out the functional form of the gradient.
Deﬁne P K as the solution to:
[Equation]
and, under this deﬁnition, it follows that C ( K ) can be written as:
[Equation]
Also, deﬁne Σ K as the (un-normalized) state correlation matrix, i.e.
[Equation]
Lemma 1.
(Policy Gradient Expression)
The policy gradient is:
[Equation]
Proof.
Observe:
[Equation]
This implies:
[Equation]
using recursion and that x 1 = ( A − BK ) x 0 .
Taking expectations completes the proof.
2. R EVIEW : (M ODEL FREE ) SAMPLE BASED POLICY GRADIENT METHODS
Sample based policy gradient methods introduce some randomization for estimating the gradient.
REINFORCE.
Let π θ ( u | x ) be a parametric stochastic policy, where u ∼ π θ ( ·| x ) .
The policy gradient of the cost, C ( θ ) , is:
[Equation]
(cid:35) , where Q π θ ( x, u ) =
E (cid:34) ∞ (cid:88) t =
c t
[Equation]
= x, u 0 = u (cid:35) , where the expectation is with respect to the trajectory { x t , u t } induced under the policy π θ and where Q π θ ( x, u ) is referred to as the state-action value.
The REINFORCE algorithm uses Monte
Carlo estimates of the gradient obtained by simulating π θ .
The natural policy gradient.
The natural policy gradient (Kakade, 2001) follows the update: θ ← θ − η G − 1 θ ∇ C ( θ ) , where G θ = E
[Equation]
u t
[Equation]
x t
[Equation]
where G θ is the Fisher information matrix.
There are numerous succesful related approaches (Peters & Schaal, 2007; Schulman et al., 2015; Duan et al., 2016).
An important special case is using a linear policy with additive Gaussian noise (Rajeswaran et al., 2017b), i.e.
[Equation]
where K ∈ R k × d and σ 2 is the noise variance.
Here, the natural policy gradient of K (when σ is considered ﬁxed) takes the form:
[Equation]
To see this, one can verify that the Fisher matrix of size kd × kd , which is indexed as [ G K ] ( i,j ) , ( i (cid:48) ,j (cid:48) ) where i, i (cid:48) ∈ { 1 , . . .
k } and j, j (cid:48) ∈ { 1 , . . .
d } , has a block diagonal form where the only non-zeros blocks are [ G K ] ( i, · ) , ( i, · ) =
Σ K (this is the block corresponding to the i -th coordinate of the action, as i ranges from 1 to k ).
This form holds more generally, for any diagonal noise.
Zeroth order optimization.
Zeroth order optimization is a generic procedure (Conn et al., 2009;
Nesterov & Spokoiny, 2015) for optimizing a function f ( x ) , using only query access to the function values of f ( · ) at input points x (and without explicit query access to the gradients of f ).
This is also the approach in using “evolutionary strategies” (Salimans et al., 2017).
The generic approach can be described as follows: deﬁne the perturbed function as
[Equation]
For small σ , the smooth function is a good approximation to the original function.
Due to the
Gaussian smoothing, the gradient has the particularly simple functional form (see Conn et al.
Nesterov & Spokoiny (2015)):
[Equation]
This expression implies a straightforward method to obtain an unbiased estimate of the ∇ f σ 2 ( x ) , through obtaining only the function values f ( x + ε ) for random ε .
T HE ( NON - CONVEX ) O PTIMIZATION L ANDSCAPE
This section provides a brief characterization of the optimization landscape, in order to help provide intuition as to why global convergence is possible and as to where the analysis difﬁculties lie.
Lemma 2.
(Non-convexity)
If d ≥ 3 , there exists an LQR optimization problem, min K C ( K ) , which is not convex, quasi-convex, and star-convex.
Section B provides a speciﬁc example.
In general, for a non-convex optimization problem, gradient descent may not even converge to the global optima in the limit.
For the case of LQRs, the following corollary (of Lemma 8) provides a characterization of the stationary points.
Corollary 3.
(Stationary point characterization)
If ∇ C ( K ) = 0 , then either K is an optimal policy or Σ K is rank deﬁcient.
This lemma is the motivation for using a distribution over x 0 (as opposed to a deterministic starting point): E x 0 ∼D x 0 x (cid:62) 0 being full rank guarantees that Σ K is full rank, which implies all stationary points are a global optima.
An additive disturbance in the dynamics model also sufﬁces.
The concept of gradient domination is important in the non-convex optimization literature (Polyak, 1963; Nesterov & Polyak, 2006; Karimi et al., 2016).
A function f : R d → R is said to be gradient dominated if there exists some constant λ , such that for all x ,
[Equation]
If a function is gradient dominated, this implies that if the magnitude of the gradient is small at some x , then the function value at x will be close to that of the optimal function value.
The following corollary of Lemma 8 shows that C ( K ) is gradient dominated.
Corollary 4.
(Gradient Domination)
Suppose E x 0 ∼D x 0 x (cid:62) 0 is full rank.
Then C ( K ) is gradient dominated, i.e.
[Equation]
where λ is a problem dependent constant (and (cid:104)· , ·(cid:105) denotes the trace inner product).
With gradient domination and no (spurious) local optima, one may hope that recent results on escaping saddle points (Nesterov & Polyak, 2006; Ge et al., 2015; Jin et al., 2017) immediately imply that gradient descent converges quickly.
This is not the case due to that it is not straightforward to characterize the (local) smoothness properties of C ( K ) ; this is a difﬁculty well studied in the optimal control theory literature, related to robustness and stability.
In fact, if it were the case that C ( K ) is a smooth function 1 (in addition to being gradient dominated), then classical mathematical optimization results (Polyak, 1963) would not only immediately imply global convergence, these results would also imply convergence at a linear rate.
M AIN R ESULTS
First, results on exact gradient methods are provided.
From an analysis perspective, this is the natural starting point; once global convergence is established for exact methods, the question of using simulation-based, model-free methods can be approached with zeroth-order optimization methods.
Notation.
(cid:107) Z (cid:107) denotes the spectral norm of a matrix Z ; Tr( Z ) denotes the trace of a square matrix; σ min ( Z ) denotes the minimal singular value of a square matrix Z .
Also, it is helpful to deﬁne
[Equation]
4. M ODEL - BASED OPTIMIZATION : EXACT GRADIENT METHODS
The following three exact update rules are considered:
[Equation]
The natural policy gradient descent direction is deﬁned so that it is consistent with the stochastic case, as per Equation 4.
It is straightforward to verify that the policy iteration algorithm is a special case of the Gauss-Newton method when η = 1 (for the case of policy iteration, convergence in the limit is provided in Todorov & Li (2004); Ng et al.
; Liao & Shoemaker (1991), along with local convergence rates.)
The Gauss-Newton method requires the most complex oracle to implement: it requires access to ∇ C ( K ) , Σ K , and R + B (cid:62) P K B ; it also enjoys the strongest convergence rate guarantee.
At the other extreme, gradient descent requires oracle access to only ∇ C ( K ) and has the slowest convergence rate.
The natural policy gradient sits in between, requiring oracle access to ∇ C ( K ) and Σ K , and having a convergence rate between the other two methods.
Theorem 5.
(Global Convergence of Gradient Methods) Suppose C ( K 0 ) is ﬁnite and and µ > 0 .
• Gauss-Newton case: For a stepsize η = 1 and for
[Equation]
the Gauss-Newton algorithm (Equation 7) enjoys the following performance bound:
[Equation]
• Natural policy gradient case: For a stepsize
[Equation]
and for
[Equation]
natural policy gradient descent (Equation 6) enjoys the following performance bound:
[Equation]
1: Input smoothing
2: for i m do 3:
Sample a policy (cid:98) K i
=
K
+ U
i , where U i is drawn uniformly at random over matrices whose (Frobenius) norm is r .
Simulate (cid:98) K i for (cid:96) steps starting from x 0 ∼ D .
Let (cid:98) C
i and (cid:98) Σ
i be the empirical estimates: (cid:98)
[Equation] [Equation]
(cid:62)
[Equation]
where c t and x t are the costs and states on this trajectory.
5: end for 6: Return the (biased) estimates:
[Equation]
an appropriate (
of the stepsize
η = poly
[Equation]
and for
[Equation]
gradient descent (Equation 5) enjoys the following performance bound:
[Equation]
In comparison to model-based approaches, these results require the (possibly) stronger assumption that the initial policy is a stable controller, i.e. C ( K 0 ) is ﬁnite (an assumption which may be inherent to local search procedures).
The Discussion mentions this as direction of future work.
4. M ODEL FREE OPTIMIZATION : SAMPLE BASED POLICY GRADIENT METHODS
In the model free setting, the controller has only simulation access to the model; the model parameters, A , B , Q and R , are unknown.
The standard optimal control theory approach is to use system identiﬁcation to learn the model, and then plan with this learned model This section proves that model-free, policy gradient methods also lead to globally optimal policies, with both polynomial computational and sample complexities (in the relevant quantities).
Using a zeroth-order optimization approach (see Section 2.2), Algorithm 1 provides a procedure to ﬁnd (controllably biased) estimates, (cid:92) ∇ C ( K ) and (cid:98) Σ K , of both ∇ C ( K ) and Σ K .
These can then be used in the policy gradient and natural policy gradient updates as follows:
[Equation]
where Algorithm 1 is called at every iteration to provide the estimates of ∇ C ( K n ) and Σ K n .
The choice of using zeroth order optimization vs using REINFORCE (with Gaussian additive noise, as in Equation 3) is primarily for technical reasons 2 .
It is plausible that the REINFORCE estimation procedure has lower variance.
One additional minor difference, again for technical reasons, is that
Algorithm 1 uses a perturbation from the surface of a sphere (as opposed to a Gaussian perturbation).
Theorem 6.
(Global Convergence in the Model Free Setting)
Suppose C ( K 0 ) is ﬁnite, µ > 0 , and that x 0 ∼ D has norm bounded by L almost surely.
Also, for both the policy gradient method and the natural policy gradient method, suppose Algorithm 1 is called with parameters:
[Equation]
• Natural policy gradient case: For a stepsize
[Equation]
and for
[Equation]
then, with high probability,
i
[Equation]
e. with probability greater than 1 − exp( − d )
[Equation]
the natural policy gradient descent update (Equation 9) enjoys the following performance bound:
[Equation]
• Gradient descent case: For an appropriate (constant) setting of the stepsize η ,
[Equation]
and if N satisﬁes
[Equation]
(cid:18)
[Equation]
(cid:107) A (cid:107) , (cid:107) B (cid:107) , (cid:107) R (cid:107) , σ min ( R ) (cid:19) , then, with high probability, gradient descent (Equation 8) enjoys the following performance bound:
[Equation]
C ONCLUSIONS AND D ISCUSSION
This work has provided provable guarantees that model-based gradient methods and model-free (sample based) policy gradient methods convergence to the globally optimal solution, with ﬁnite polynomial computational and sample complexities.
Taken together, the results herein place these popular and practical policy gradient approaches on a ﬁrm theoretical footing, making them comparable to other principled approaches (e.g. subspace ID methods and algebraic iterative approaches).
Finite C ( K 0 ) assumption, noisy case, and ﬁnite horizon case.
These methods allow for extensions to the noisy case and the ﬁnite horizon case.
This work also made the assumption that C ( K 0 ) is ﬁnite, which may not be easy to achieve in some inﬁnite horizon problems.
The simplest way to address this is to model the inﬁnite horizon problem with a ﬁnite horizon one; the techniques developed in Section D.1 shows this is possible.
This is an important direction for future work.
Open Problems.
• Variance reduction: This work only proved efﬁciency from a polynomial sample size perspective.
An interesting future direction would be in how to rigorously combine variance reduction methods and model-based methods to further decrease the sample size.
• A sample based Gauss-Newton approach: This work showed how the Gauss-Newton algorithm improves over even the natural policy gradient method, in the exact case.
A practically relevant question for the Gauss-Newton method would be how to both: a) construct a sample based estimator b) extend this scheme to deal with (non-linear) parametric policies.
• Robust control:
In model based approaches, optimal control theory provides efﬁcient procedures to deal with (bounded) model mis-speciﬁcation.
An important question is how to provably understand robustness in a model free setting.
A P LANNING WITH A MODEL
This section brieﬂy reviews some parameterizations and solution methods for the classic LQR and related problems from control theory.
Finite horizon LQR.
First, consider the ﬁnite horizon case.
The basic approach is to view it as a dynamic program with the value function x
Tt P t x t , where
[Equation]
which in turn gives optimal control
[Equation]
recursions run backward in time).
Another approach is to view the LQR problem as a linearly-constrained Quadratic Program in all x t and u t (where the constraints are given by the dynamics, and the problem size equals the horizon).
The QP is clearly a convex problem, but this observation is not useful by itself as the problem size grows with the horizon, and naive use of quadratic programming scales badly.
However, the special structure due to linear dynamics allows for simpliﬁcations and control-theoretic interpretation as follows: the Lagrange multipliers can be interpreted as “co-state” variables, and they follow a recursion that runs backwards in time known as the “adjoint system”.
Using Lagrange duality, one can show that this approach is equivalent to solving the Riccati recursion mentioned above.
Popular use of the LQR in control practice is often in the receding horizon LQR, Camacho & Bordons (2004); Rawlings & Mayne (2009): at time t , an input sequence is found that minimizes the T -step ahead LQR cost starting at the current time, then only the ﬁrst input in the sequence is used.
The resulting static feedback gain converges to the inﬁnite horizon optimal solution as horizon T becomes longer.
Inﬁnite horizon LQR.
Here, the constrained optimization view (QP) is not informative as the problem is inﬁnite dimensional; the dynamic programming viewpoint readily extends.
Suppose the system A, B is controllable (which guarantees optimal cost is ﬁnite).
It turns out that the value function and the optimal controller are static (do not depend on t ) and can be found by solving the
Algebraic Riccati Equation (ARE) given in (1).
The optimal K can then be found from equation (2).
The main computational step is solving the ARE, which is extensively studied (e.g. (Lancaster &
Rodman, 1995)).
One approach due to Kleinman (1968) is to simply run the recursion
P k +1 =
Q +
A T P k A − A T P k B ( R + B T P k B ) − 1 B T P k A with P 1 = Q , which converges to the unique positive semideﬁnite solution of the ARE (since the ﬁxed-point iteration is contractive).
Other approaches are direct and based on linear algebra, which carry out an eigenvalue decomposition on a certain block matrix followed by a matrix inversion (Lancaster & Rodman, 1995).
Direct computation of the control input has also been considered in the optimal control literature, e.g., gradient updates in function spaces (Polak, 1973).
For the linear quadratic setup, direct iterative computation of the feedback gain has been examined in (M˚artensson & Rantzer, 2009), and explored further in (M˚artensson, 2012) with a view towards distributed implementations.
There methods are presented as local search heuristics without provable guarantees of reaching the optimal policy.
SDP formulation.
The LQR problem can also be expressed as a semideﬁnite program (SDP) with variable P , as given in (Balakrishnan & Vandenberghe, 2003) (section 5, equation (34), this is for a continuous-time system but there are similar discrete-time versions).
This SDP can be derived by relaxing the equality in the Riccati equation to an inequality, then using the Schur complement formula to rewrite the resulting Riccati inequality as linear matrix inequality; the objective in the case of LQR is the trace of the positive deﬁnite matrix variable.
This formulation and its dual has been explored in (Balakrishnan & Vandenberghe, 2003).
It is important to note that while the optimal solution of this SDP is the unique positive semideﬁnite solution to the Riccati equation, which in turn gives the optimal policy K ∗ , other feasible P (not equal to P ∗ ) do not necessarily correspond to a feasible, stabilizing policy K .
This means that the feasible set of this SDP is not a convex characterization of all P that correspond to stabilizing
K .
Thus it also implies that if one uses any optimization algorithm that maintains iterates in the feasible set (e.g. interior point methods), no useful policy can be extracted from the iterates before convergence to P ∗ .
For this reason, this convex formulation is not helpful for parametrizing the space of policies K in manner that supports the use of local search methods (those that directly lower the cost function of interest), which is the focus of this work.
B N ON - CONVEXITY OF THE SET OF STABILIZING S TATE F EEDBACK G AINS
Let K ( A, B ) denote the set of state
feedback gains K such that A − BK is stable, i.e., its eigenvalues are inside the unit circle in the complex plane.
This set is generally nonconvex.
A concise counterexample to convexity is provided here.
Let A and B be 3 × 3 identity matrices and
K 1 = (cid:34)0 −− 1    (cid:35) and K 2 = (cid:34)− 10   −  (cid:35) .
Then the spectra of A − BK 1 and A − BK 2 are both concentrated at the origin, yet two of the eigenvalues of A − B (cid:98) K with (cid:98) K
= ( K 1 + K 2 ) / 2 , are outside of the unit circle in the complex plane.
C A NALYSIS : THE EXACT CASE
This section provides the analysis of the convergence rates of the (exact) gradient based methods.
First, some helpful lemmas for the analysis are provided.
Throughout, it is convenient to use the following deﬁnition:
[Equation]
The policy gradient can then be written as:
[Equation]
C. H ELPER LEMMAS
Deﬁne the value V K ( x ) , the state-action value Q K ( x, u ) , and the advantage A K ( x, u ) .
V K ( x, t ) is the cost of the policy starting with x 0 = x and proceeding with K onwards:
[Equation]
Q K ( x, u ) is the cost of the policy starting with x 0 = x , taking action u 0 = u and then proceeding with K onwards:
[Equation]
The advantage A K ( x, u ) is:
[Equation]
The advantage can be viewed as the change in cost starting at state x and taking a one step deviation from the policy K .
The next lemma is identical to that in (Kakade & Langford, 2002; Kakade, 2003) for Markov decision processes.
Lemma 7.
(Cost difference lemma)
Suppose K and K (cid:48) have ﬁnite costs.
Let { x (cid:48) t } and { u (cid:48) t } be state and action sequences generated by K (cid:48) , i.e. starting with x (cid:48)
= x and using u (cid:48) t = − K (cid:48)
x (cid:48) t .
It holds that:
[Equation]
Also, for any x , the advantage is:
[Equation]
[Equation]
Proof.
Let c (cid:48) t be the cost sequence generated by K (cid:48) .
Telescoping the sum appropriately:
[Equation]
which completes the ﬁrst claim
[Equation]
For the second claim, observe that:
[Equation]
And, for u = K (cid:48) x ,
[Equation]
which completes the proof
[Equation]
This lemma is helpful in proving that C ( K ) is gradient dominated.
Lemma 8.
(Gradient domination) Let K ∗ be an optimal policy.
Suppose K has ﬁnite cost and µ > 0 .
It holds that:
[Equation]
For a lower bound
[Equation]
it holds that:
[Equation]
Proof.
From Equation 10 and by completing the square,
[Equation]
(11) with equality when K (cid:48)
=
K − ( R + B (cid:62) P K B ) − 1 E K .
Let x ∗ t and u ∗ t be the sequence generated under K ∗ .
Using this and Lemma 7,
[Equation]
which completes the proof of the upper bound.
For the lower bound, consider K (cid:48)
=
K − ( R + B (cid:62) P K B ) − 1 E K where equality holds in Equation 11.
Let x (cid:48) t and u (cid:48)
t be the sequence generated under K (cid:48) .
Using that C ( K ∗ ) ≤ C ( K (cid:48) ) ,
[Equation]
which completes the proof.
Recall that a function f is said to be smooth (or C 1 -smooth) if it satisﬁes for some ﬁnite β , it satisﬁes:
[Equation]
for all x, y (equivalently, it is smooth if the gradients of f are continuous).
Lemma 9.
(“Almost” smoothness) C ( K ) satisﬁes:
[Equation]
To see why this is related to smoothness (e.g. compare to Equation 12), suppose K (cid:48) is sufﬁciently close to K so that:
[Equation]
(13) and the leading order term 2Tr(Σ K
[Equation]
K
[Equation]
− K ) (cid:62)
[Equation]
K ) would then behave as Tr(( K (cid:48) − K ) (cid:62) ∇ C ( K )) .
The challenge in the proof (for gradient descent) is quantifying the lower order terms in this argument.
Proof.
The claim immediately results from Lemma 7, by using Equation 10 and taking an expectation.
The next lemma spectral norm bounds on P K and Σ K are helpful:
Lemma 10.
It holds that:
[Equation]
Proof.
For the ﬁrst claim, C ( K ) is lower bounded as:
[Equation]
Alternatively, C ( K ) can be lower bounded as:
[Equation]
which proves the second claim
[Equation]
C. G
[Equation]
-N EWTON A NALYSIS
The next lemma bounds the one step progress of Gauss-Newton.
Lemma 11.
Suppose that:
[Equation]
If η ≤
[Equation]
then
[Equation]
Proof.
Observe K (cid:48)
=
K − η ( R + B (cid:62) P K B ) − 1 E K .
Using Lemma 9 and the condition on η ,
[Equation]
where the last step uses Lemma 8.
With this lemma, the proof of the convergence rate of the Gauss Newton algorithm is immediate.
Proof.
(of Theorem 5, Gauss-Newton case)
The theorem is due to that η = 1 leads to a contraction of 1 − ηµ (cid:107) Σ K ∗ (cid:107) at every step.
C. N ATURAL P OLICY G RADIENT D ESCENT A NALYSIS
The next lemma bounds the one step progress of the natural policy gradient.
Lemma 12.
Suppose:
[Equation]
and that η ≤
[Equation]
It holds that:
[Equation]
Proof.
Since K (cid:48)
=
K − ηE K , Lemma 9 implies:
[Equation]
The last term can be bounded as:
[Equation]
Continuing and using the condition on η ,
[Equation]
using Lemma 8.
With this lemma, the proof of the natural policy gradient convergence rate can be completed.
Proof.
(of Theorem 5, natural policy gradient case)
Using Lemma 10,
[Equation]
The proof is completed by induction:
C ( K 1 ) ≤ C ( K 0 ) , since Lemma 12 can be applied.
The proof proceeds by arguing that Lemma 12 can be applied at every step.
If it were the case that
C ( K t ) ≤ C ( K 0 ) , then
[Equation]
and by Lemma 12:
[Equation]
which completes the proof.
C. G RADIENT D ESCENT A NALYSIS
As informally argued by Equation
[Equation]
the proof seeks to quantify how Σ K (cid:48) changes with η .
Then the proof bounds the one step progress of gradient descent.
Σ K PERTURBATION ANALYSIS
This subsections aims to prove the following:
Lemma 13.
( Σ K perturbation)
Suppose K (cid:48) is such that:
[Equation]
It holds that:
[Equation]
The proof proceeds by starting with a few technical lemmas.
First, deﬁne a linear operator on symmetric matrices, T K ( · ) , which can be viewed as a matrix on (cid:0) d +12 (cid:1) dimensions.
Deﬁne this operator on a symmetric matrix X as follows:
[Equation]
Also deﬁne the induced norm of T as follows:
[Equation]
where the supremum is over all symmetric matrices X (whose spectral norm is non-zero).
Also, deﬁne
[Equation]
Lemma 14.
( T K norm bound)
It holds that
[Equation]
Proof.
For a unit norm vector v ∈ R d and unit spectral norm matrix X ,
[Equation]
using that T K (Σ 0 )
=
Σ K .
The proof is completed using the upper bound on (cid:107) Σ K (cid:107) in Lemma 10.
Also, with respect to K , deﬁne another linear operator on symmetric matrices:
[Equation]
Let I to denote the identity operator on the same space.
Deﬁne the induced norm (cid:107) · (cid:107) of these operators as in Equation 14.
Note these operators are related to the operator T K as follows:
Lemma 15.
When ( A − BK ) has spectral radius smaller than 1,
T K =
(I − F K ) − 1 .
Proof.
When ( A − BK ) has spectral radius smaller than 1, T K is well deﬁned and is the solution of T K
= I + T K ◦ F K .
Therefore T K ◦ (I − F K ) =
I and T K
= (I − F K ) − 1 .
Since,
[Equation]
The proof of Lemma 13 seeks to bound:
[Equation]
The following two perturbation bounds are helpful in this.
Lemma 16.
It holds that:
[Equation]
Proof.
Let ∆ =
K − K (cid:48) .
For every matrix X ,
[Equation]
The operator norm of F K −F K (cid:48) is the maximum possible ratio in spectral norm of ( F K −F K (cid:48) )
( X ) and X .
Then the claim follows because (cid:107) AX (cid:107) ≤ (cid:107) A (cid:107)(cid:107) X (cid:107) .
Lemma 17.
If
[Equation]
then
[Equation]
Proof.
Deﬁne A
= I − F K , and B = F K (cid:48)
− F K .
In this case A − 1 = T K and ( A − B ) − 1 =
T K (cid:48) .
Hence, the condition (cid:107)T K (cid:107)(cid:107)F K − F K (cid:48) (cid:107) ≤ 1 / 2 translates to the condition (cid:107)A − 1 (cid:107)(cid:107)B(cid:107) ≤ 1 / 2 .
Observe:
[Equation]
( A − 1 (Σ))
=
(I − (I − A − 1 ◦ B ) − 1 )( T K (Σ)) .
Since (I − A − 1 ◦ B ) −
= I + A − 1 ◦ B ◦
(I − A − 1 ◦ B ) − 1 , (cid:107) (I − A − 1 ◦ B ) − 1 (cid:107) ≤ 1 + (cid:107)A − 1 ◦ B(cid:107)(cid:107)
(I − A − 1 ◦ B ) − 1 (cid:107) ≤ 1 + 1 / 2 (cid:107) (I − A − 1 ◦ B ) − 1 (cid:107) which implies (cid:107) (I − A − 1 ◦ B ) − 1 (cid:107) ≤ 2 .
Hence,
[Equation]
and so
[Equation]
Combining these two,
[Equation]
This proves the main inequality.
The last step of the inequality is just applying deﬁnition of the norm of T K : (cid:107)T K (Σ) (cid:107) ≤ (cid:107)T K (cid:107)(cid:107) Σ (cid:107) .
With these Lemmas, the proof is completed as follows:
Proof.
(of Lemma 13)
First, the proof shows (cid:107)T K (cid:107)(cid:107)F K − F K (cid:48)
(cid:107) ≤ 1 / 2 , which is the desired condition in Lemma 17.
First, observe that under the assumed condition on (cid:107) K − K (cid:48) (cid:107) , implies that
[Equation]
using that σ min ( Q ) µ C ( K ) ≤ 1 due to Lemma 10.
Using Lemma 16,
[Equation]
Using this and Lemma 14,
[Equation]
where the last step uses the condition on (cid:107) K − K (cid:48) (cid:107) .
Thus,
[Equation]
using Lemmas 10 and 16.
G RADIENT D ESCENT P ROGRESS
Equipped with these lemmas, the one step progress of gradient descent can be bounded
[Equation]
Lemma 18.
Suppose that
[Equation]
where
[Equation]
It holds that:
[Equation]
Proof.
By Lemma
[Equation]
where the last step uses Lemma
[Equation]
By Lemma
[Equation]
using the assumed condition on η .
Using this last claim and Lemma 10,
[Equation]
) ≤
[Equation]
Σ K
[Equation]
C
[Equation]
K
[Equation]
σ min ( Q ) and so (cid:107) Σ K (cid:48) (cid:107) ≤ 4 C ( K ) 3 σ min ( Q ) .
Hence, 1 −
[Equation]
K (cid:48)
[Equation]
K (cid:107)
[Equation]
using the condition on η .
In order to prove a gradient descent convergence rate, the following bounds are helpful:
Lemma 19.
It holds that
[Equation]
µ and that:
[Equation]
Proof.
Using Lemma 10,
[Equation]
By Lemma
[Equation]
which proves the ﬁrst claim.
Again using Lemma 8,
[Equation]
which proves the second claim.
With these lemmas, the proof of the gradient descent convergence rate follows:
Proof.
(of Theorem 5, gradient descent case)
First, the following argues that progress is made at t = 1 .
Based on Lemma 10 and Lemma 19, by choosing η to be an appropriate polynomial in
C ( K 0 ) , 1 (cid:107) A (cid:107) , 1 (cid:107) B (cid:107) , 1 (cid:107) R (cid:107) , σ min ( R ) , σ min ( Q ) and µ , the stepsize condition in Equation 16 is satisﬁed.
Hence, by Lemma 18,
[Equation]
which implies that the cost decreases at t = 1 .
Proceeding inductively, now suppose that C ( K t ) ≤
C ( K 0 ) , then the stepsize condition in Equation 16 is still satisﬁed (due to the use of C ( K 0 ) in bounding the quantities in Lemma 19).
Thus, Lemma 18 can again be applied for the update at time t + 1 to obtain:
[Equation]
Provided
[Equation]
then C ( K T ) − C
( K ∗ ) ≤ ε , and the result follows.
D A NALYSIS : THE M ODEL - FREE CASE
This section shows how techniques from zeroth order optimization allow the algorithm to run in the model-free setting with only black-box access to a simulator.
The dependencies on various parameters are not optimized, and the notation h is used to represent different polynomial factors in the relevant factors ( C ( K 0 )
µσ min ( Q ) , (cid:107) A (cid:107) , (cid:107) B (cid:107) , (cid:107) R (cid:107) , 1 /σ min ( R ) ).
When the polynomial also depend on dimension d or accuracy 1 /(cid:15) , this is speciﬁed as parameters ( h ( d, 1 /(cid:15) ) ).
The section starts by showing how the inﬁnite horizon can be approximated with a ﬁnite horizon.
D. A PPROXIMATING C ( K ) AND Σ K WITH FINITE HORIZON
This section shows that as long as there is an upper bound on C ( K ) , it is possible to approximate both C ( K ) and Σ( K ) with any desired accuracy.
Lemma 20.
For any K with ﬁnite C ( K ) , let Σ ( (cid:96) )
K = E [ (cid:80) (cid:96) − 1 i =
x i x (cid:62) i ] and C ( (cid:96) )
( K ) =
E [ (cid:80) (cid:96) − 1 i =
x (cid:62) i
Qx i + u (cid:62) i Ru i ]
= (cid:104) Σ ( t ) K , Q + K (cid:62) RK (cid:105) .
If (cid:96)
[Equation]
(cid:15)µσ
[Equation]
, then (cid:107) Σ ( (cid:96) )
K − Σ K (cid:107) ≤ (cid:15) .
Also, if (cid:96) ≥ d · C 2 ( K )( (cid:107) Q (cid:107) + (cid:107) R (cid:107)(cid:107) K (cid:107) 2 ) (cid:15)µσ 2 min ( Q ) then C ( K ) ≥ C ( (cid:96) )
( K ) ≥ C ( K ) − (cid:15) .
Proof.
First, the bound on Σ K is proved.
Deﬁne the operators T K and F K as in Section C.4, observe Σ K = T K (Σ 0 ) and Σ ( (cid:96) )
K =
Σ K − ( F K ) (cid:96) (Σ K ) .
If X (cid:23) Y , then F K ( X ) (cid:23) F K ( Y ) , this follows immediately from the form of F K ( X ) =
( A +
BK ) X ( A + BK ) (cid:62) .
If X is PSD then WXW (cid:62) is also PSD for any W .
Now, since
[Equation]
(Here the last step is by Lemma 10), and all traces are nonnegative, then there must exists j < (cid:96) such that tr( F jK (Σ
[Equation]
) ≤
[Equation]
min ( Q ) .
Also, since Σ K (cid:22)
[Equation]
Therefore as long as
[Equation]
it follows that:
[Equation]
Here the ﬁrst step is again because of all the terms are PSD, so using more terms is always better.
The last step follows because F j (Σ K ) is also a PSD matrix so the spectral norm is bounded by trace.
In fact, it holds that tr(Σ K − Σ ( (cid:96) ) K ) is smaller than (cid:15) .
Next, observe C ( (cid:96) )
( K ) =
(cid:104) Σ ( (cid:96) )
K , Q + K (cid:62) RK (cid:105) and C ( K )
= (cid:104) Σ K , Q + K (cid:62) RK (cid:105) , therefore
[Equation]
K − Σ ( (cid:96) ) K )( (cid:107) Q (cid:107) + (cid:107) R (cid:107)(cid:107) K (cid:107) 2 ) .
Therefore if (cid:96)
[Equation]
) (cid:15)µσ 2 min ( Q ) , then tr(Σ K − Σ ( (cid:96) ) K ) ≤ (cid:15)/ ( (cid:107) Q (cid:107) + (cid:107) R (cid:107)(cid:107) K (cid:107) 2 ) and hence C ( K ) − C ( (cid:96) )
( K ) ≤ (cid:15) .
D. P ERTURBATION OF C ( K ) AND ∇ C ( K )
The next lemma show that the function value and its gradient are approximate preserved if a small perturbation to the policy K is applied.
Lemma 21.
( C K perturbation)
Suppose K (cid:48) is such that:
[Equation]
then:
[Equation]
Proof.
As in the proof of Lemma
[Equation]
the assumption implies that (cid:107)T K (cid:107)(cid:107)F K − F K
[Equation]
(cid:107) ≤ 1 / 2 , and, from Equation
[Equation]
that
[Equation]
First, observe:
[Equation]
Hence,
[Equation]
(cid:48) )
(cid:107) ≤ 2 (cid:107)T K (cid:107)
[Equation]
(cid:107)F
[Equation]
− F
[Equation]
(cid:48) (cid:107) (cid:0) (cid:107) ( K (cid:48) ) (cid:62) RK (cid:48) )
− K (cid:62) RK (cid:107) + (cid:107) K (cid:62) RK ) (cid:107) (cid:1) + (cid:107)T
[Equation]
(cid:107)(cid:107)
[Equation]
(cid:48)
[Equation]
(cid:107)
[Equation]
=
[Equation]
For the ﬁrst term,
[Equation]
) using the assumption that (cid:107) K (cid:48) − K (cid:107) ≤ (cid:107) K (cid:107) .
For the second term,
[Equation]
R (cid:107) .
Combining the two terms completes the proof.
The next lemma shows the gradient is also stable after perturbation.
Lemma 22.
( ∇ C K perturbation) Suppose K (cid:48) is such that:
[Equation]
then there is a polynomial h grad in C ( K 0 )
µσ min
[Equation]
A
[Equation]
R (cid:107) , 1 σ min ( R ) such that |
[Equation]
Proof.
Recall ∇ C ( K ) =
2 E K Σ K where E K
=
( R + B (cid:62) P K B ) K − B
[Equation]
P K A .
Therefore
[Equation]
Let’s ﬁrst look at the second term.
By Lemma 8,
[Equation]
then by Lemma
[Equation]
Therefore the second term is bounded by
[Equation]
Now the ﬁrst term is bounded.
Since K (cid:48) − K is small enough, (cid:107) Σ K (cid:48) (cid:107) ≤ (cid:107) Σ K (cid:107) + C ( K ) σ min ( Q ) .
For E K (cid:48) − E K , a bound on P K (cid:48) − P K is provided.
By the previous lemma,
[Equation]
P
[Equation]
P
[Equation]
Therefore
[Equation]
Since (cid:107) K (cid:48) (cid:107) ≤ 2 (cid:107) K (cid:107) , and (cid:107) K (cid:107) can be bounded by C ( K ) (Lemma 19), all the terms can be bounded by polynomials of related parameters multiplied by (cid:107) K − K (cid:48) (cid:107) .
D. S MOOTHING AND THE GRADIENT DESCENT ANALYSIS
This section analyzes the smoothing procedure and completes the proof of gradient descent.
Although Gaussian smoothing is more standard, the objective C ( K ) is not ﬁnite for every K , therefore technically E u ∼N (0 ,σ 2 I )
[ C ( K + u )] is not well deﬁned.
This is avoidable by smoothing in a ball.
Let S r represent the uniform distribution over the points with norm r (boundary of a sphere), and B r represent the uniform distribution over all points with norm at most r (the entire sphere).
When applying these sets to matrix a U , the Frobenius norm ball is used.
The algorithm performs gradient descent on the following function
[Equation]
The next lemma uses the standard technique (e.g. in (Flaxman et al., 2005)) to show that the gradient of C r ( K ) can be estimated just with an oracle for function value.
Lemma 23.
[Equation]
This is the same as Lemma 2.1 in Flaxman et al.
(2005), for completeness the proof is provided below.
Proof.
By Stokes formula,
[Equation]
By deﬁnition,
[Equation]
Also,
[Equation]
The Lemma follows from combining these equations, and use the fact that
[Equation]
From the lemma above and standard concentration inequalities, it is immediate that it sufﬁces to use a polynomial number of samples to approximate the gradient.
Lemma 24.
Given an (cid:15) , there are ﬁxed polynomials h r (1 /(cid:15) ) , h sample ( d, 1 /(cid:15) )
such that when r ≤ 1 /h r (1 /(cid:15) ) , with m ≥ h sample ( d, 1 /(cid:15) )
samples of U 1 , ..., U n ∼ S r , with high probability (at least 1 − ( d/(cid:15) ) − d ) the average
[Equation]
is (cid:15) close to ∇ C ( K ) .
Further, if for x ∼ D , (cid:107) x (cid:107) ≤ L almost surely, there are polynomials h
[Equation]
,grad ( d, 1 /(cid:15) ) , h r,trunc (1 /(cid:15) ) , h sample,trunc ( d, 1 /(cid:15), σ, L 2 /µ ) such that when m ≥ h sample,trunc ( d, 1 /(cid:15), L 2 /µ ) , (cid:96) ≥ h (cid:96),grad ( d, 1 /(cid:15) ) , let x ij , u ij (0 ≤ j ≤ (cid:96) ) be a single path sampled using K + U i , then the average ˜ ∇ =
[Equation]
is also (cid:15) close to ∇ C ( K ) with high probability.
Proof.
For the ﬁrst part, the difference is broken into two terms:
[Equation]
For the ﬁrst term, choose h r (1 /(cid:15) )
=
min { 1 /r 0 , 2 h grad /(cid:15) } ( r 0 is chosen later).
By Lemmawhen r is smaller than 1 /h r (1 /(cid:15) )
= (cid:15)/ 2 h grad , every point u on the sphere have (cid:107)∇ C
( K +
U ) − ∇ C ( K ) (cid:107) ≤ (cid:15)/ 4 .
Since ∇ C r ( K ) is the expectation of ∇ C ( K + U ) , by triangle inequality (cid:107)∇
[Equation]
The proof also makes sure that r ≤ r 0 such that for any U ∼ S r , it holds that C ( K + U ) ≤ 2 C ( K ) .
By Lemma 21, 1 /r 0 is a polynomial in the relevant factors.
For the second term, by Lemma 23, E [ ˆ ∇ ] = ∇ C r ( K ) , and each individual sample has norm bounded by 2 dC ( K ) /r , so by Vector Bernstein’s Inequality, know with m ≥ h sample ( d, 1 /(cid:15) )
=
Θ (cid:16) d (cid:16) dC ( K ) (cid:15)r 2 (cid:17) log d/(cid:15) (cid:17) samples, with high probability (at least 1 − ( d/(cid:15) ) − d ) (cid:107)
[Equation]
Adding these two terms and apply triangle inequality gives the result.
For the second part, the proof breaks it into more terms.
Let ∇ (cid:48) be equal to 1 m (cid:80) mi =1 dr 2 C ( (cid:96) )
( K +
U i )
U i (where C ( (cid:96) ) is deﬁned as in Lemma 20), then
[Equation]
The third term is just what was bounded earlier, using h r,trunc (1 /(cid:15) )
=
h r (2 /(cid:15) ) and making sure h sample,trunc ( d, 1 /(cid:15) )
≥ h sample ( d, 2 /(cid:15) ) , this guarantees that it is smaller than (cid:15)/ 2 .
For the second term, choose (cid:96) ≥ 16 d 2 · C 2 ( K )( (cid:107) Q (cid:107) + (cid:107) R (cid:107)(cid:107) K (cid:107) 2 ) (cid:15)rµσmin ( Q )
=:
h (cid:96),grad ( d, 1 /(cid:15) ) .
By Lemma 20, for any K (cid:48) with C ( K (cid:48) ) ≤ 2 C ( K ) , it holds that (cid:107) C ( (cid:96) )
( K (cid:48) )
− C ( K (cid:48) )
(cid:107) ≤ r(cid:15) 4 d .
Therefore by triangle inequality (cid:107)
[Equation]
Finally for the ﬁrst term it is easy to see that E [ ˜ ∇ ] = ∇ (cid:48) where the expectation is taken over the randomness of the initial states x i 0 .
Since (cid:107) x i 0 (cid:107) ≤ L , ( x i 0 )
( x i 0 ) (cid:62) (cid:22) L 2 µ E [ x 0 x (cid:62) 0 ] , as a result the sum [
[Equation]
Therefore, ˜ ∇ − ∇ (cid:48) is again a sum of independent vectors with bounded norm, so by Vector Bernstein’s inequality, when h sample,trunc ( d, 1 /(cid:15), L 2 /µ ) is a large enough polynomial, (cid:107) ˜ ∇−∇ (cid:48) (cid:107) ≤
(cid:15)/with high probability.
Adding all the terms ﬁnishes the proof.
Theorem 25.
There are ﬁxed polynomials h GD,r (1 /(cid:15)
[Equation]
, h GD,
[Equation]
µ
[Equation]
h
[Equation]
( d
[Equation]
/(cid:15) )
such that if every step the gradient is computed as Lemma 24 (truncated at step (cid:96) ), pick step size η and T the same as the gradient descent case of Theorem 5, it holds that C ( K T )
− C
( K (cid:63) ) ≤ (cid:15) with high probability (at least 1 − exp( − d ) ).
Proof.
By Lemma 18, when η ≤ 1 /h GD,η for some ﬁxed polynomial h GD,η (given in Lemma 18), then
[Equation]
Let ˜ ∇ be the approximate gradient computed, and let K (cid:48)(cid:48)
=
K − η ˜ ∇ be the iterate that uses the approximate gradient.
The proof shows given enough samples, the gradient can be estimated with enough accuracy that makes sure
[Equation]
This means as long as C ( K )
− C
( K ∗ ) ≥ (cid:15)
, it holds that
[Equation]
Then the same proof of Theorem 5 gives the convergence guarantee.
Now C ( K (cid:48)(cid:48) )
− C ( K (cid:48) ) is bounded.
By Lemma 21, if (cid:107) K (cid:48)(cid:48) − K
[Equation] [Equation]
R ) µ
[Equation]
Σ K ∗ (cid:107) · (cid:15) · 1 /h func ( h func is the polynomial in Lemma 21), then C ( K (cid:48)(cid:48) )
− C ( K (cid:48) ) is small enough.
To get that, observe
K (cid:48)(cid:48) − K (cid:48) =
η ( ∇ − ˜ ∇ ) , therefore it sufﬁces to make sure
[Equation]
By Lemma 22, it sufﬁces to pick h GD,r (1 /(cid:15) )
=
h r,trunc (2 h func (cid:107) Σ K ∗ (cid:107) / ( µ 2 σ min ( R ) (cid:15) )) , h GD,sample ( d, 1 /(cid:15), L 2 /µ )
=
h sample,trunc ( d, 2 h func (cid:107) Σ K ∗ (cid:107) / ( µ 2 σ min ( R ) (cid:15) ) , L 2 /µ ) , and h GD,(cid:96) ( d, 1 /(cid:15) )
=
h (cid:96),grad ( d, 2 h func (cid:107) Σ K ∗ (cid:107) / ( µ 2 σ min ( R ) (cid:15) )) .
This gives the desired upper-bound on (cid:107)∇ − ˜ ∇(cid:107) with high probability (at least 1 − ( (cid:15)/d ) − d ).
Since the number of steps is a polynomial, by union bound with high probability (at least 1 −
T ( (cid:15)/d ) − d ≥ 1 − exp( − d ) )
the gradient is accurate enough for all the steps, so
[Equation]
The rest of the proof is the same as Theorem 5.
Note that in the smoothing, because the function value is monotonically decreasing and the choice of radius, all the function value encountered is bounded by 2 C ( K 0 ) , so the polynomials are indeed bounded throughout the algorithm.
D. T HE NATURAL GRADIENT ANALYSIS
Before the Theorem for natural gradient is proven, the following lemma shows the variance can be estimated accurately.
Lemma 26.
If for x ∼ D , (cid:107) x (cid:107) ≤ L almost surely, there exists polynomials h r,var (1 /(cid:15) ) , h varsample,trunc ( d, 1 /(cid:15), L 2 /µ ) and h (cid:96),var ( d, 1 /(cid:15) )
such that if ˆΣ K is estimated using at least m ≥ h varsample,trunc ( d, 1 /(cid:15), L 2 /µ ) initial points x 10 , ..., x m 0 , m random perturbations
U i ∼ S r where r ≤ 1 /h r,var (1 /(cid:15) ) , all of these initial points are simulated using ˆ K i
=
K
+ U i to (cid:96) ≥ h (cid:96),var ( d, 1 /(cid:15) ) iterations, then with high probability (at least 1 − ( d/(cid:15) )
− d ) the following estimate
[Equation]
satisﬁes (cid:107) ˜Σ − Σ K (cid:107) ≤ (cid:15) .
Further, when (cid:15) ≤ µ/ 2 , it holds that σ min (ˆΣ K ) ≥ µ/ 2 .
Proof.
This is broken into three terms: let Σ ( (cid:96) )
K be deﬁned as in Lemma 20, let ˆΣ = 1 m (cid:80) mi =1 Σ K
+ U i and ˆΣ ( (cid:96) )
= 1 m (cid:80) mi =1 Σ ( (cid:96) )
K
+ U
i , then it holds that
[Equation]
First, r is chosen small enough so that C ( K + U i ) ≤ 2 C ( K ) .
This only requires an inverse polynomial r by Lemma 21.
For the ﬁrst term, note that E [˜Σ] = ˆΣ ( (cid:96) ) where the expectation is taken over the initial points x i 0 .
Since (cid:107) x i 0 (cid:107) ≤ L , ( x i 0 )
( x i 0 ) (cid:62) (cid:22) L 2 µ E [ x 0 x (cid:62) 0 ] , and as a result the sum (cid:96)
[Equation]
1 (cid:88)
[Equation] [Equation]
Therefore, standard concentration bounds show that when h varsample,trunc is a large enough polynomial, (cid:107) ˜Σ − ˆΣ ( (cid:96) )
(cid:107) ≤ (cid:15)/ 2 holds with high probability.
For the second term, Lemma 20 is applied.
Because C ( K + U i ) ≤ 2 C ( K ) , choosing (cid:96) ≥ h (cid:96),var ( d, 1 /(cid:15) )
= 8 d · C 2 ( K ) (cid:15)µσ 2 min ( Q ) , the error introduced by truncation (cid:107) ˆΣ ( (cid:96) ) − ˆΣ (cid:107) is then bounded by (cid:15)/ 4 .
For the third term, Lemma 13 is applied.
When r ≤ (cid:15) · (cid:16) σ min ( Q ) C ( K ) (cid:17)µ
[Equation]
B
[Equation]
( (cid:107) A − BK
[Equation]
+1)
, (cid:107) Σ K + U
i − Σ K (cid:107) ≤ (cid:15)/ 4 .
Since ˆΣ is the average of Σ K + U i , by the triangle inequality, (cid:107) ˆΣ − Σ K (cid:107) ≤ (cid:15)/ 4 .
Adding these three terms gives the result.
Finally, the bound on σ min (˜Σ K ) follows simply from Weyl’s Theorem.
Theorem 27.
Suppose C ( K 0 ) is ﬁnite and and µ > 0 .
The natural gradient follows the update rule:
[Equation]
Suppose the stepsize is set to be:
[Equation]
If the gradient and variance are estimated as in Lemma 24, Lemma 26 with r = 1 /h NGD,r (1 /(cid:15) ) , with m ≥ h NGD,sample ( d, 1 /(cid:15), L 2 /µ ) samples, both are truncated to h NGD,(cid:96)
( d, 1 /(cid:15) ) iterations, then with high probability (at least 1 − exp( − d ) ) in T iterations where
[Equation]
ε then the natural gradient satisﬁes the following performance bound:
[Equation]
Proof.
By Lemma 12,
[Equation]
Let ˜ ∇ be the estimated gradient, ˜Σ K be the estimated Σ K , and let K (cid:48)(cid:48)
=
K − η ˜ ∇ ˜Σ K − 1 .
The proof shows that when both the gradient and the covariance matrix are estimated accurately enough, then
[Equation]
This implies when
C ( K ) − C
( K (cid:63) ) ≥ (cid:15) ,
[Equation]
which is sufﬁcient for the proof.
By Lemma 21, if (cid:107) K (cid:48)(cid:48) − K
[Equation]
≤
[Equation]
the desired bound on | C
[Equation]
K
[Equation]
− C
[Equation]
K
[Equation]
) | holds.
To achieve this, it sufﬁces to have
[Equation]
This is broken into two terms
[Equation]
≤
[Equation]
For the ﬁrst term, by Lemma 26 we know when the number of samples is large enough (cid:107) ˜Σ − 1 K (cid:107) ≤ 2 /µ .
Therefore it sufﬁces to make sure (cid:107) ˜ ∇ − ∇(cid:107) ≤ (cid:15) 8 h func σ min ( R ) µ 2 (cid:107) Σ K ∗ (cid:107) , this can be done by Lemma 24 by setting h NGD,grad,r (1 /(cid:15) )
=
h
[Equation]
, L/µ
[Equation]
) and h NGD,(cid:96),grad ( d, 1 /(cid:15) )
=
h (cid:96),
[Equation]
( d,
[Equation]
(cid:107) Σ
[Equation]
K
[Equation]
2 σ min
[Equation]
R
[Equation]
For the second term, it sufﬁces to make sure (cid:107) ˜Σ
[Equation]
≤
[Equation]
( R ) µ
[Equation]
K )
[Equation]
By standard matrix perturbation, and σ min (Σ K ) ≥ µ , (cid:107) ˜Σ − 1 K − Σ − 1 K (cid:107) ≤ 2 (cid:107) ˜Σ K − Σ K (cid:107)
/µ(when (cid:107) ˜Σ K − Σ K (cid:107) ≤ µ/ 2 ).
Therefore by Lemma 26 it sufﬁces to choose h NGD,var,r (1 /
[Equation] [Equation]
h
[Equation]
L/
[Equation]
and h NGD,(cid:96),var ( d,
[Equation]
(cid:15) )
[Equation]
This is indeed a polynomial because (cid:107)∇ C ( K ) (cid:107) is bounded by Lemma 19.
Finally, choose
[Equation]
max { h NGD,gradsample , h NGD,varsample } , and h NGD,(cid:96)
[Equation]
This ensures all the bounds mentioned above hold and that
[Equation]
The rest of the proof is the same as Theorem 5.
Note again that in the smoothing, because the function value is monotonically decreasing and the choice of radius, all the function values encountered are bounded by 2 C ( K 0 ) , so the polynomials are indeed bounded throughout the algorithm.