Global Convergence of Direct Policy Search for State-Feedback HRobust Control: A Revisit of Nonsmooth Synthesis with Goldstein Subdifferential
Abstract
Direct policy search has been widely applied in modern reinforcement learningand continuous control.
However, the performance of direct policy search onnonsmooth robust control synthesis has not been well understood.
The optimal
H 1 control framework aims at designing a policy to minimize the closed-loop
H 1 norm, and is arguably the most important robust control paradigm.
In thiswork, we show that direct policy search is guaranteed to ﬁnd the global solution ofthe robust H 1 state-feedback control design problem.
Notice that policy searchfor optimal H 1 control leads to a constrained nonconvex nonsmooth optimizationproblem where the nonconvex feasible set consists of all the policies stabilizingthe closed-loop dynamics.
We show that for this nonsmooth optimization problem,all Clarke stationary points are global minimum.
Next, we identify the coercivityof the closed-loop H 1 objective function, and prove that the sublevel sets of theresultant policy search problem are compact.
Based on these properties, we showthat the Goldstein subdifferential method and its various implementable variantscan be guaranteed to stay in the nonconvex feasible set and eventually ﬁnd theglobal optimal solution for the H 1 state-feedback synthesis problem.
Our workbuilds a new connection between nonconvex nonsmooth optimization theory androbust control, leading to the ﬁrst global convergence result for direct policy searchon optimal H 1 synthesis.
Introduction
Reinforcement learning (RL) has achieved impressive performance on many continuous controltasks [49, 34], and policy optimization is one of the main workhorses for such applications [15,52, 48, 50].
Recently, there have been extensive research efforts studying the global convergenceproperties of policy optimization methods on benchmark control problems including linear quadraticregulator (LQR) [18, 7, 35, 57, 37, 19, 33, 25], stabilization [44], linear robust/risk-sensitive control[60, 59, 23, 61, 62], Markov jump linear quadratic control [28, 27, 46], output feedback control[17, 63], and dynamic ﬁltering [55].
For all these benchmark problems, the objective function in thepolicy optimization formulation is always differentiable over the entire feasible set, and the existingconvergence theory heavily relies on this fact.
Consequently, an important open question remainswhether direct policy search can enjoy similar global convergence properties when applied to thefamous H 1 control problem whose objective function is not always differentiable in the policy space[1, 2, 3, 24, 9, 12, 41].
Different from LQR which considers stochastic disturbance, H 1 controldirectly addresses the worst-case disturbance, and provides arguably the most fundamental robustcontrol paradigm [64, 16, 51, 4, 14, 20].
Regarding the connection with RL, it has also been shownthat H 1 control can be applied to stabilize the training of adversarial RL schemes in the linear quadratic setup [59, Section 5].
Given the fundamental importance of H 1 control, we view it asan important benchmark for understanding the theoretical properties of direct policy search in thecontext of robust control and adversarial RL.
The main contribution of our paper is the ﬁrst globalconvergence guarantee of direct policy search for the H 1 state-feedback control synthesis problem.
The objective of the H 1 state-feedback synthesis is to design a linear state-feedback policy thatstabilizes the closed-loop system and minimizes the H 1 norm from the disturbance to a performancesignal at the same time.
The design goal is also equivalent to synthesizing a state-feedbackpolicy that minimizes a quadratic cost subject to the worst-case disturbance.
We will present theproblem formulation for the H 1 state-feedback synthesis and discuss such connections in Section 2.
Essentially, H 1 state-feedback synthesis can be formulated as a constrained policy optimizationproblem min K 2 K J ( K ) , where the decision variable K is a matrix parameterizing the linear statefeedback policy, the objective function J ( K ) is the closed-loop H 1 -norm for given K , and thefeasible set K consists of all the linear state-feedback policies stabilizing the closed-loop dynamics.
Notice that the feasible set for the H 1 state-feedback control problem is the same as the nonconvexfeasible set for the LQR policy search problem [18, 7].
However, the objective function J ( K ) for the
H 1 control problem can be non-differential over certain feasible points, introducing new difﬁcultyto direct policy search.
There has been a large family of nonsmooth H 1 policy search algorithmsdeveloped based on the concept of Clarke subdifferential [1, 2, 3, 24, 9, 12].
However, no globalconvergence guarantee has been reported in the literature.
Our paper bridges this gap by making thefollowing two contributions.
We show that all Clarke stationary points for the H 1 state-feedback control problem arealso global minimum.
We identify the coercivity property of the H 1 cost function and use this property to showthat the Goldstein subdifferential method [22] and its implementable variants [
58, 13, 9,10, 31, 32] can be guaranteed to stay in the nonconvex feasible set of stabilizing policiesduring the optimization process and eventually ﬁnd the global optimal solution of the H 1state-feedback control problem
Finite-time complexity bounds for ﬁnding (  , ✏ ) -stationary points are also provided
To the best of our knowledge, our work offers the ﬁrst global convergence guarantee of direct policysearch on H 1 control problems.
We believe that our work also serves as a meaningful initial steptowards a general convergence theory of direct policy search on nonsmooth robust control synthesis.
Finally, it is worth clarifying the differences between H 1 control and mixed H 2 / H 1 design.
67 For mixed
H 2 / H 1 control, the objective is to design a stabilizing policy that minimizes an H 2 68 performance bound and satisﬁes an H 1 constraint at the same time [21, 30, 29, 40].
In other words,mixed H 2 / H 1 control aims at improving the average H 2 performance while “maintaining" a certain 70 level of robustness by keeping the closed-loop H 1 norm to be smaller than a pre-speciﬁed number.
In contrast, H 1 control aims at “improving" the system robustness and the worst-case performancevia achieving the smallest closed-loop H 1 norm.
In [60], it has been shown that the natural policygradient method initialized from a policy satisfying the H 1 constraint can be guaranteed to maintainthe H 1 requirement during the optimization process and eventually converge to the optimal solutionof the mixed design problem.
However, notice that the objective function for the mixed H 2 / H 1 76 control problem is still differentiable over all the feasible points, and hence the analysis technique in[60] cannot be applied to our H 1 control setting.
More discussion on the connections and differencesbetween these two problems will be presented in the supplementary material..
Problem Formulation and Preliminaries2.1 Notation
The set of p -dimensional real vectors is denoted as R p .
For a matrix A , we use the notation A T , k A k ,tr A ,  min ( A ) ,
k A k 2 , and ⇢ ( A ) to denote its transpose, largest singular value, trace, smallest singularvalue, Frobenius norm, and spectral radius, respectively.
When a matrix P is negative semideﬁnite(deﬁnite), we will use the notation P  (  )0 .
When P is positive semideﬁnite (deﬁnite), we use thenotation P ⌫ (  )0 .
Consider a (real) sequence u := { u 0 , u 1 , · · · } where u t 2 R n u for all t .
This 87 sequence is said to be in ` n u 2 if[-(cid:80) ∞ t =
-]
{+P 1 t =
+}
k u t k 2 < 1 where k u t k denotes the standard (vector) 2-normof u t .
In addition, the 2 -norm for u 2 ` n u 2 is deﬁned as k u k 2 :=[-(cid:80) ∞ t =
-]
P
[Equation]
2.2 Problem statement: H 1 state-feedback synthesis and a policy optimization formulation
We consider the following linear time-invariant (LTI) system
[Equation]
1) where x t 2 R n x is the state, u t 2 R n u is the control action, and w t 2 R n w is the disturbance.
Wehave A 2 R n x ⇥ n x , B 2 R n x ⇥ n u , and n w = n x .
We denote x := { x 0 , x 1 , · · · } , u := { u 0 , u 1 , · · · } , 93 and w := { w 0 , w 1 , · · · } .
The initial condition is ﬁxed as x 0 = 0 .
The objective of H 1 controlis to choose { u t } to minimize the quadratic cost [-(cid:80) ∞ t =
-]P [Equation]
[Equation] u [Equation] Ru t ) in the presence of theworst-case ` 2 disturbance satisfying k w k  1 .
In this paper, the following assumption is adopted.
Assumption 1.
The matrices Q and R are positive deﬁnite.
The matrix pair ( A, B ) is stabilizable.
In H 1 control, { w t } is considered to be the worst-case disturbance satisfying the ` 2 norm boundk w k  1 , and can be chosen in an adversarial manner.
This is different from LQR which makesstochastic assumptions on { w t } .
Without loss of generality, we have chosen the ` 2 upper bound on wto be 1 .
In principle, we can formulate the H 1 control problem with any arbitrary ` 2 upper boundon w , and there is no technical difference.
We will provide more explanations on this fact in thesupplementary material.
Therefore, H 1 control can be formulated as the following minimax problem
[Equation]
Under Assumption 1, it is well known that the optimal solution for (2) can be achieved using a linearstate-feedback policy u t =  Kx t (see [4]).
Given any K , the LTI system (1) can be rewritten as
[Equation]
Now we deﬁne z t =
( Q + K T RK ) 12 x t .
We have k z t {+k 
+} = x T t ( Q + K T RK ) x t = x T t Qx t + u T t Ru t .
We denote z := { z 0 , z 1 , · · · } .
If x 2 ` n x 2 , then we have k z k 2 = P 1 t {+=
+} ( x T t Qx t + u T t Ru t ) < + 1 .
Therefore, the closed-loop LTI system (3) can be viewed as a linear operator mapping any disturbancesequence { w t } to another sequence { z t } .
We denote this operator as G K , where the subscript 109 highlights the dependence of this operator on K .
If K is stabilizing, i.e. ⇢ ( A  BK ) < 1 , then 110 G
K is bounded in the sense that it maps any ` 2 sequence w to another sequence z in ` n x 2 .
For anystabilizing K , the ` 2 ! ` 2 induced norm of G K can be deﬁned as:
[Equation]
Since G K is a linear operator, it is straightforward to show
[Equation]
Therefore, the minimax optimization problem (2) can be rewritten as the policy optimization problem:min K 2 K k G K k 22 ! 2 , where K is the set of all linear state-feedback stabilizing policies, i.e. K = { K {+
+} R n x ⇥ n u : ⇢ ( A  BK ) < 1 } .
In the robust control literature [1, 2, 3, 24, 9, 12], it is standard to dropthe square in the cost function and just reformulate (2) as min K
2 K k G K k 2 ! 2 .
This is exactly the 117 policy optimization formulation for H 1 state-feedback control.
The main reason why this problem is 118 termed as H 1 state-feedback control is that in the frequency domain, G K can be viewed as a transferfunction which lives in the Hardy H 1 space and has an H 1 norm being exactly equal to k G K k 2 ! 2 .
Applying the frequency-domain formula for the H 1 norm, we can calculate k G K k 2 ! 2 as
[Equation]
where I is the identity matrix, and  max denotes the largest eigenvalue of a given symmetric matrix.
Therefore, eventually the H 1 state-feedback control problem can be formulated as
[Equation]
where J ( K ) is equal to the H 1 norm speciﬁed by (5).
Classical H 1 control theory typically 124 solves (6) via introducing extra Lyapunov variables and reparameterizing the problem into a higherdimensional convex domain over which convex optimization algorithms can be applied [64, 16, 6].
In this paper, we revisit (6) as a benchmark for direct policy search, and discuss how to search theoptimal solution of (6) in the policy space directly.
Applying direct policy search for (6) leads toa nonconvex nonsmooth optimization problem.
A main technical challenge is that the objectivefunction (5) can be non-differentiable over some important feasible points [1, 2, 3, 24, 9, 12].
2.3 Direct policy search: A nonsmooth optimization perspective
Now we brieﬂy review several key facts known for the H 1 policy optimization problem (6).
Proposition 1.
The set K = { K : ⇢ ( A  BK ) < 1 } is open.
In general, it can be unbounded and 133 nonconvex.
The cost function (5) is continuous and nonconvex in K .
See [18, 8] for some related proofs.
We have also included more explanations in the supplementarymaterial.
An immediate consequence is that (6) becomes a nonconvex optimization problem.
Anotherimportant fact is that the objective function (5) is also nonsmooth.
As a matter of fact, (5) is subjectto two sources of nonsmoothness.
Based on (5), we can see that the largest eigenvalue for a ﬁxedfrequency ! is nonsmooth, and the optimization step over ! 2 [0 , 2 ⇡ ] is also nonsmooth.
As a matterof fact, the H 1 objective function (5) can be non-differentiable over important feasible points, e.g.optimal points.
Fortunately, it is well known 1 that the H 1 objective function (5) has the followingdesired property so it is Clarke subdifferentiable.
Proposition 2.
The H 1 objective function (5) is locally Lipschitz and subdifferentially regular overthe stabilizing feasible set K .
Recall that J : K !
R is locally Lipschitz if for any bounded S ⇢ K , there exists a constant L > 0 145 such that | [Equation]
[Equation] |  L k K  K 0 k 2 for all K, K 0 2 S .
Based on Rademacher’s theorem, a 146 locally Lipschitz function is differentiable almost everywhere, and the Clarke subdifferential is welldeﬁned for all feasible points.
Formally, the Clarke subdifferential is deﬁned as
[Equation]
where conv denotes the convex hull.
Then we know that the Clarke subdifferential for the H 1objective function (5) is well deﬁned for all K 2 K .
We say that K is a Clarke stationary point if0 2 @
C J ( K ) .
The following fact is also well known.
Proposition 3.
If K is a local min of J , then 0 2 @ C J ( K ) and K is a Clarke stationary point.
Under Assumption 1, it is well known that there exists K ⇤ 2 K achieving the minimum of (6).
Since
K is an open set, K ⇤ has to be an interior point of K and hence K ⇤ has to be a Clarke stationary point.
In Section 3, we will prove that any Clarke stationary points for (6) are actually global minimum.
Now we brieﬂy elaborate on the subdifferentially regular property stated in Proposition 2.
For anygiven direction d (which has the same dimension as K ), the generalized Clarke directional derivativeof J is deﬁned as
[Equation]
In contrast, the (ordinary) directional derivative is deﬁned as follows (when existing)
[Equation]
In general, the Clarke directional derivative can be different from the directional derivative, andthe directional derivative does not always exist.
The objective function J ( K ) is subdifferentiallyregular if for every K 2 K , the ordinary directional derivative always exists and coincides with thegeneralized one for every direction, i.e. J 0 ( K, d ) =
J  ( K, d ) .
The most important consequence ofthe subdifferentially regular property is given as follows.
Corollary 1.
Suppose K † 2 K is a Clarke stationary point for J .
If J is subdifferentially regular,then the directional derivatives J {+
+}
( K † , d ) are non-negative for all d .
on nonsmooth H 1 synthesis for long time, this fact is well known and hence we do not claim any credits in deriving this result.
As a matter of fact, although not explicitly stated, the proof of Proposition 2 is hinted in the last paragraph of [2, Section III] given the facts that the H 1 norm is a convex function over the Hardy H 1 space (which is a Banach space) and the mapping from K 2 K to the (inﬁnite-dimensional) Hardy H 1 space is smooth.
For completeness, a simple proof of Proposition 2 based on Clarke’s chain rule is included in the supplementary material.
See [11] for related proofs.
Notice that having non-negative directional derivatives does not meanthat the point K † is a local minimum.
Nevertheless, the above fact will be used in our maintheoretical developments.
Now we brieﬂy summarize two key difﬁculties in establishing a globalconvergence theory for direct policy search on the H 1 state-feedback control problem (6).
First, it isunclear whether the direct policy search method will get stuck at some local minimum.
Second, itis challenging to guarantee the direct policy search method to stay in the nonconvex feasible set Kduring the optimization process.
Since K is nonconvex, we cannot use a projection step to maintainfeasibility.
Our main results will address these two issues.
2.4 Goldstein subdifferential
Generating a descent direction for nonsmooth optimization is not trivial.
Many nonsmoothoptimization algorithms are based on the concept of Goldstein subdifferential [22].
Before proceedingto our main result, we brieﬂy review this concept here.
Deﬁnition 1 (Goldstein subdifferential) .
Suppose J is locally Lipschitz.
Given a point K 2 K and aparameter  > 0 , the Goldstein subdifferential of J at K is deﬁned to be the following set
[Equation]
where B  ( K ) denotes the  -ball around K .
The above deﬁnition implicitly requires B  ( K ) ⇢ K .
A point K is (  , ✏ ) -stationary if dist(0 , @  J ( K ))  ✏ .
It is well-known that the minimal norm 183element of the Goldstein subdifferential generates a descent direction.
This fact is stated as follows.
Proposition 4 ([22]) .
Let F be the minimal norm element in @  J ( K ) .
Suppose K  ↵ F/ k F k 2 2 K 184 for any ↵   .
Then we have
[Equation]
The idea of Goldstein subdifferential has been used in designing algorithms for nonsmooth
H 1control [3, 24, 9, 12].
We will show that the Goldstein subdifferential method can be guaranteed toﬁnd the global minimum of (6).
Optimization Landscape for H 1 State-Feedback Control
In this section, we investigate the optimization landscape of H 1 state-feedback control, and showthat any Clarke stationary points of (6) are also global minimum.
We start by showing the coercivityof the H 1 objective function (5).
Lemma 1.
The H 1 objective function J ( K ) deﬁned by (5) is coercive over the set K in the sense thatfor any sequence { K l } 1 l =1 ⇢ K we have J ( K l ) !
+ 1 , if either k K l k 2 ! + 1 , or K l converges 194 to an element in the boundary @ K .
Proof.
We will only provide a proof sketch here.
A detailed proof is presented in the supplementarymaterial.
Suppose we have a sequence { K l } satisfying k K l k 2 ! + 1 .
We can choose w ={ w 0 , 0 , 0 , · · · } with k w 0 k = 1 and show J ( K l )  w T 0 ( Q + ( K l ) T RK l ) w 0   min ( R )
k K l w 0 k 2 .
Clearly, we have used the positive deﬁniteness of R in the above derivation.
Then by carefullychoosing w 0 , we can ensure J ( K l ) ! + 1 as k K l k 2 ! + 1 .
Next, we assume K l ! K 2 @ K .
We 200 have ⇢ ( A  BK )
= 1 , and hence there exists some {+! 
+} such that ( e j !
I  A + BK ) becomes singular.
Then we can use the positive deﬁniteness of Q to show J ( K l )   1 / 2 min ( Q ) [-∥ ( e jω 
-]k
{+( e j ! 
+}
I  A + BK l )  1 k .
Notice  min ( e j !
I  A + BK l ) {+! 
+} as l ! 1 , which implies k ( e j !
I  A + BK l )  1 k !
+ 1as l ! 1 .
Therefore, we have J ( K l ) ! + 1 as K l ! K 2 @ K .
More details for the proof can be 204 found in the supplementary material.
We want to emphasize that the positive deﬁniteness of ( Q, R ) are crucial for proving the coercivityof the cost function (5).
Built upon Lemma 1, we can obtain the following nice properties of thesublevel sets of (6).
Lemma 2. Consider the H 1 state-feedback control problem (6) with the objective function J ( K )deﬁned in (5) .
Under Assumption 1, the sublevel set deﬁned as K  = { K 2 K : J ( K )   } is 210 compact and path-connected for every   J ( K ⇤ ) where K ⇤ is the global minimum of (6) .
Proof.
The compactness of K  directly follows from the continuity and coercivity of J ( K ) , and isactually a consequence of [5, Proposition 11.12].
The path-connectedness of the strict sublevel setsfor the continuous-time H 1 control problem has been proved in [26].
We can slightly modify theproof in [26] to show that the strict sublevel set { K 2 K : J ( K ) <
} is path-connected.
Based onthe fact that every non-strict sublevel sets are compact, now we can apply [36, Theorem 5.2] to show
K  is also path-connected.
An independent proof based on the non-strict version of the bounded reallemma is provided in the supplementary material.
The path-connectedness of K  for every  actually implies the uniqueness of the minimizing setin a certain sense [36, Sections 2&3].
Due to the space limit, we will defer the discussion on theuniqueness of the minimizing set to the supplementary material.
Here, we present a stronger resultwhich is one of the main contributions of our paper.
Theorem 1.
Consider the H 1 state-feedback control problem (6) .
Under Assumption 1, any Clarkestationary point of J ( K ) is a global minimum.
A detailed proof is presented in the supplementary material.
Here we provide a proof sketch.
Since
Q and R are positive deﬁnite, the non-strict version of the bounded real lemma [6, Section 2.7.3]states that J ( K )   if and only if there exists a positive deﬁnite matrix P such that the followingmatrix inequality holds
[Equation]
The above matrix inequality is linear in P but not linear in K .
A standard trick from the control theorycan be combined with the Schur complement lemma to convert the above matrix inequality conditionto another condition which is linear in all the decision variables [6].
Speciﬁcally, there exists a matrixfunction LMI( Y, L,  ) which is linear in ( Y, L,  ) such that LMI( Y, L,  )
if and only if (12)is feasible with K = LY  1 and P =  Y  1 .
The matrix function LMI( Y, L,  ) involves a largermatrix.
Hence we present the analytical formula of LMI( Y, L,  ) in the supplementary material andskip it here.
Since LMI( Y, L,  ) is linear in ( Y, L,  ) , we know LMI( Y, L,  )  0 is just a convexsemideﬁnite programming condition.
Based on this convex necessary and sufﬁcient condition for
J ( K )   , we can prove the following important lemma.
Lemma 3.
For any K 2 K satisfying J ( K ) > J ⇤ , there exists d 6 = 0 such that J 0 ( K, d ) < 0 , where
J ⇤ =
J ( K ⇤ ) and K ⇤ is the global minimum of (6) .
Proof.
Suppose we have K = LY  1 where ( Y, L, J ( K )) is a feasible point for the convex regime
LMI( Y, L, J ( K ))  0 .
In addition, we have K ⇤ =
L ⇤ ( Y ⇤ )  1 where ( Y ⇤ , L ⇤ , J ( K ⇤ )) is a pointsatisfying LMI( Y ⇤ , L ⇤ , J ( K ⇤ ))  0 .
Since the LMI condition is convex, the line segment between( Y, L, J ( K )) and ( Y ⇤ , Q ⇤ , J ( K ⇤ )) is also in this convex set.
For any t > 0 , we know ( Y + t  Y, L +t  L, J ( K ) + t ( J ( K ⇤ )  J ( K ))) also satisﬁes LMI( Y + t  Y, L + t  L, J ( K ) + t ( J ( K ⇤ )
J ( K )))  0 , where  L = L ⇤  L , and  Y
=
Y ⇤  Y .
Therefore, based on the boundedreal lemma, we know J (( L + t  L )( Y + t  Y )  1 )  J ( K ) + t ( J ( K ⇤ )  J ( K )) .
Let’s choosed =  LY  1 + LY  1  Y Y  1 .
Then we have
[Equation]
A detailed veriﬁcation of the above inequality is provided in the supplementary material.
Noticed 6 = 0 .
If  LY  1 + LY  1  Y Y  1 = 0 , the above argument still works and we reach to theconclusion J {+
+}
( K, 0) < 0 .
But this is impossible since we always have J {+
+} ( K, 0)
= 0 .
Hence we haved 6 = 0 .
This completes the proof for this lemma.
Now we are ready to provide the proof for Theorem 1.
Based on Lemma 3 and the fact that J ( · ) issubdifferentially regular, the proof can be done by contradiction.
Suppose K ⇤ is the global minimum,and K † is a Clarke stationary point.
If K † is not a global minimum.
Then by Lemma 3, there existsd 6 = 0 such that J 0 ( K † , d ) < 0 , this contradicts the fact that J {+
+} ( K † , d )  0 for all d by Corollary 1.
Therefore, K † has to be the global minimum of (6).
The above proof relies on Lemma 3 and the fact that J is subdifferentially regular.
Without usingthe subdifferential regular property, Lemma 3 itself is not sufﬁcient for proving Theorem 1.
It isalso worth mentioning that Lemma 3 can be viewed as an extension of the lifting technique in [55,
Theorem 4] to non-differentiable points.
In this section, we ﬁrst show that the Goldstein subdifferential method can be guaranteed to stay inthe nonconvex feasible regime K during the optimization process and eventually converge to theglobal minimum of (6).
The complexity of ﬁnding (  , ✏ ) -stationary points of (6) is also presented.
Then we further discuss the convergence guarantees for various more implementable forms of the
Goldstein subdifferential method.
4.1 Global convergence and complexity of the Goldstein subdifferential method
We will investigate the global convergence of the following Goldstein subdifferential method fordirect policy search of the H 1 optimal state-feedback policy:
[Equation]
(13) where F n is the minimum norm element of the Goldstein subdifferential @  n J ( K n ) .
We assume thatan initial stabilizing policy is available, i.e. K 0 2 K .
The same initial policy assumption has also beenmade in the global convergence theory for direct policy search on LQR [18].
More recently, someprovable guarantees have been obtained for ﬁnding such stabilizing policies via direct policy search[44].
Hence such an assumption on the initial policy K 0 is reasonable.
Our global convergence resultrelies on the fact that there is a strict separation between any sublevel set of (6) and the boundary of
K .
This fact is formalized as follows.
Lemma 4. Consider the H 1 state-feedback control problem (6) with the cost function J ( K ) deﬁnedin (5) .
Suppose   J ⇤ .
Under Assumption 1, there is a strict separation between the sublevel set K and K c which is the complement of the feasible set K .
In other words, we have dist ( K  , K c ) > 0 .
Proof.
Obviously, the set K  \ K c is empty (since we know K  ⇢ K ).
Based on Lemma 2, we 280 know K  is compact.
Since K is open, we know K c is closed.
Therefore, there is a strict separationbetween K  and K c , and we have dist ( K  , K c ) > 0 .
Now we are ready to present our main convergence result.
Theorem 2.
Consider the H 1 state-feedback control problem (6) with the cost function J ( K ) deﬁnedin (5) .
Suppose an initial stabilizing policy is given, i.e. K 0 2 K .
Denote  0 :=
dist( K J ( K 0 ) , K c ) >0 .
Choose  n = c  0 n +1 for all n with c being a ﬁxed number in (0 , 1) .
Then the Goldstein subdifferentialalgorithm (13) is guaranteed to stay in K for all n .
In addition, we have J ( K n ) ! J ⇤ as n ! 1 .
Proof.
We have  n  c  0 <  0 for all n .
Now we use an induction proof to show K n 2 K J ( K 0 )for all n = 0 , 1 , · · · .
For n = 0 , we know K 0  c  0 F 0 / k F 0 k 2 has to be within the  0 ball around 289 K 0 since we know the norm of F 0 / k F 0 k 2 is exactly equal to 1 .
Since  0 := dist( K J ( K 0 ) , K c ) > 0 ,we know K 0   0 F 0 / k F 0 k 2 2 K .
As a matter of fact, we know B [-δ 
-] 0 ( K 0 ) has to be a subset of K .
Hence we can apply (11) to show that K 1 exists and is also in K J ( K 0 ) .
Similarly, we can repeat thisargument to show K n 2 K J ( K 0 ) for all n .
Next, we can apply (11) to every step and then sum theinequalities over all n .
Then the following inequality holds for all N :
[Equation]
Since we have P 1 n =0  n = 1 , we know lim inf n !1 k F n k 2 = 0 .
There exists one subsequence{ i n } such that k F i n k 2 ! 0 .
For this subsequence, the resultant policy sequence { K i n } is also 296 bounded (notice that the policy parameter sequence stays in the compact set K J ( K 0 ) for all n ) and has 297 a convergent subsequence.
We can show that the limit of this subsequence is a Clarke stationary point.
Hence the function value { J ( K i n ) } associated with this subsequence converges to J ⇤ .
Notice that
J ( K n ) is monotonically decreasing for the entire sequence { n } .
Hence we have J ( K n ) ! J ⇤ .
We have tried to be brief in giving the above proof.
We present a more detailed proof in thesupplementary material.
We believe that this is the ﬁrst global convergence result for direct policysearch on the H 1 state-feedback control problem.
The above result only provides an asymptoticconvergence guarantee to ensure J ( K n ) ! J ⇤ .
One can use a similar argument to establish a ﬁnitetime complexity bound for ﬁnding the (  , ✏ ) -stationary points of (6).
Such a result is given as follows.
Theorem 3.
Consider the H 1 problem (6) with the cost function (5) .
Suppose K 0 2 K .
Denote 306  0 :=
dist( K J ( K 0 ) , K c ) > 0 .
For any  <  0 , we can choose  n =  for all n to ensure that
the
Goldstein subdifferential algorithm (13) to stay in K and satisﬁes the following ﬁnite-time bound:
[Equation]
In other words, we have min 0  n  N k F n k 2  ✏ after N = O   ✏  where  :=
J ( K 0 )  J ⇤ .
Forany  <  0 and ✏ > 0 , the complexity of ﬁnding a (  , ✏ ) -stationary point is O   ✏  .
The above result can be proved using a similar argument from Theorem 2.
We can use the sameinduction argument to show K n 2 K J ( K 0 ) for all n , and (14) holds with  n =  .
Then the desiredconclusion directly follows.
The complexity for nonsmooth optimization of Lipschitz functions isquite subtle.
While the above result gives a reasonable characterization of the ﬁnite time performanceof the Goldstein subdifferential method on the H 1 state-feedback control problem, it does notquantify how fast J ( K n ) converges to J ⇤ .
Recall that (  , ✏ ) -stationarity means dist(0 , @  J ( K ))  ✏ ,while ✏ -stationarity means dist(0 , @
C J ( K ))  ✏ .
As commented in [58, 13], (  , ✏ ) -stationaritydoes not imply being  -close to an ✏ -stationary point of J .
Consequently, the function value ofa (  , ✏ ) -stationary point can be very far from J ⇤ even for very small  and ✏ .
Theorem 5 in [58]shows that there is no ﬁnite time algorithm that can ﬁnd ✏ -stationary points provably for all Lipschitzfunctions.
It is still possible that one can develop some ﬁnite time bounds for ( J ( K n )  J ⇤ ) viaexploiting other advanced properties of the H 1 cost function (5).
This is an important future task.
4.2 Implementable variants and related convergence results
In practice, it can be difﬁcult to evaluate the minimum norm element of the Goldstein subdifferential.
Now we discuss implementable variants of the Goldstein subdifferential algorithm and the guarantees.
Gradient sampling [9, 10, 31].
The gradient sampling (GS) method is the main optimizationalgorithm used in the robust control package HIFOO [3, 24].
Suppose we can access a ﬁrst-orderoracle which can evaluate r J for any differentiable points in K 2 .
Based on Rademacher’s theorem, alocally Lipschitz function is differentiable almost everywhere.
Therefore, for any K n 2 K , we canrandomly sample policy parameters over B  n ( K n ) and obtain differentiable points with probabilityone.
For all these sampled differentiable points, the Clarke subdifferential at each point is just thegradient.
Then the convex hull of these sampled gradients can be used as an approximation of @  n K n .
The minimum norm element from the convex hull of the sampled gradients can be solved via a simpleconvex quadratic program, and is sufﬁcient for generating a reasonably good descent direction forupdating K n +1 as long as we sample at least ( n x n u + 1) differentiable points for each n [9].
In theunconstrained setup, the cluster points of the GS algorithm can be guaranteed to be Clarke stationary[9, Theorem 3.1].
This result can be combined with Theorem 1 and Lemma 4 to show the globalconvergence of the GS method.
The following theorem will be treated formally in the supplementary.
Theorem 4 (Informal statement) .
Consider the policy optimization problem (6) with the H 1 costfunction deﬁned in (5) .
Suppose K 0 2 K .
The iterations generated from the GS algorithm isguaranteed to stay in K for all iterations and achieve J ( K n ) ! J ⇤ with probability one.
Non-derivative sampling (NS) [32].
The NS method can be viewed as the derivative-free version ofthe GS algorithm.
Suppose we only have the zeroth-order oracle which can evaluate the functionvalue J ( K ) for K 2 K .
The main difference between NS and GS is that the NS algorithm relies onestimating the gradient from function values via Gupal’s estimation method.
In the unconstrainedsetting, the cluster points of the NS method can be guaranteed to be Clarke stationary with probabilityone [32, Theorem 3.8].
We can combine [32, Theorem 3.8] with our results (Theorem 1 and Lemma4) to prove the global convergence of NS in our setting.
The details are given in the supplementary.
Model-free implementation of NS.
When the model is unknown, there are various methods availablefor estimating the H 1 -norm from data [38, 39, 47, 45, 56, 43, 54, 53].
Based on our experience,the multi-input multi-output (MIMO) power iteration method [42] works quite well as a stochasticzeroth-order oracle for the purpose of implementing NS in the model-free setting.
While the samplecomplexity for model-free NS is unknown, we will provide some numerical justiﬁcations to showthat such a model-free implementation closely tracks the convergence of its model-based counterpart.
[Figure]
Interpolated normalized gradient descent (INGD) with ﬁnite-time complexity.
No ﬁnite-timeguarantees for ﬁnding (  , ✏ ) -stationary points have been reported for the GS/NS methods.
In [58, 13],the INGD method has been developed as another implementable approximation of the Goldsteinsubdifferential method, and is proved to satisfy high-probability ﬁnite-time complexity bounds forﬁnding (  , ✏ ) -stationary points of Lipschitz functions.
INGD uses an iterative sampling strategy togenerate a descent direction which serves a role similar to the minimal norm element of the Goldsteinsubdifferential.
A ﬁrst-order oracle for differentiable points is needed for implementing INGD.
It has been show [58, Theorem 8] that for unconstrained nonsmooth optimization of L -Lipschitzfunctions, the above INGD algorithm can be guaranteed to ﬁnd the (  , ✏ ) -stationary point withthe high-probability iteration complexity O ⇣
[Equation] log [Equation] ⌘ , where  :=
J ( K 0 )  J ⇤ is the initialfunction value gap, and p is the failure probability (i.e. the optimization succeeds with the probability(1  p ) ).
We can combine the proofs for Theorem 3 and [58, Theorem 8] to obtain the followingcomplexity result for our H 1 setting.
A formal treatment is given in the supplementary material.
Theorem 5 (Informal statement) .
Consider the policy optimization problem (6) with the H 1 costfunction deﬁned in (5) .
Suppose K 0 2 K .
Denote  0 :=
dist( K J ( K 0 ) , K c ) > 0 , and let L 0 be the
Lipschitz constant of J ( K ) over the set K J ( K 0 ) .
For any  <  0 , we can choose  n =  for all n toensure that the iterations of the INGD algorithm stay in K almost surely, and ﬁnd a (  , ✏ ) -stationarypoint with the high-probability iteration complexity O [Equation] log
[Equation] 1 p [Equation]  Numerical Simulations
We provide some numerical simulations in this section as well as the supplementary material.
The leftplot in Figure 1 shows that GS, NS, INGD, and model-free NS work well for the following example:
[Equation]
For this example, we have J ⇤ = 7 . 3475 .
We initialize from K 0 =
4931  0 .
1368  2 . 2654] ,which satisﬁes ⇢ ( A  BK 0 ) = 0 .
5756 < 1 .
The hyperparameter choices are detailed in thesupplementary material.
We can see that model-free NS closely tracks the trajectory of NS and workswell.
In the middle plot of Figure 1, we test the NS method on randomly generated cases.
We set
A 2 R 3 ⇥ 3 to be I + ⇠ , where each element of ⇠ 2 R 3 ⇥ 3 is sampled uniformly from [0 , 1] .
For
B 2 R 3 ⇥ 1 , each element is uniformly sampled from [0 , 1] .
We have Q =
I + ⇣
I 2 R 3 ⇥ 3 with ⇣uniformly sampled from [0 , 0 . 1] , and R 2 R uniformly sampled from [1 , 1 . 5] .
For each experiment,the initial condition K 0 2 R 1 ⇥ 3 is also randomly sampled such that ⇢ ( A  BK 0 ) < 1 .
The NSmethod converges globally for all the cases.
In the right plot, we focus on the model-free setting for(16).
We decrease the samples used in the H 1 estimation and show how this increases the noisein the zeroth-order oracle and worsens the convergence behaviors of the model-free NS method.
Nevertheless, the model-free NS method tracks its model-based counterpart with enough samples.