Global Convergence of Direct Policy Search for State-Feedback H ∞ Robust Control: A Revisit of Nonsmooth Synthesis with Goldstein Subdifferential
Abstract
Direct policy search has been widely applied in modern reinforcement learning and continuous control.
However, the theoretical properties of direct policy search on nonsmooth robust control synthesis have not been fully understood.
The optimal
H ∞ control framework aims at designing a policy to minimize the closed-loop
H ∞ norm, and is arguably the most fundamental robust control paradigm.
In this work, we show that direct policy search is guaranteed to find the global solution of the robust H ∞ state-feedback control design problem.
Notice that policy search for optimal H ∞ control leads to a constrained nonconvex nonsmooth optimization problem, where the nonconvex feasible set consists of all the policies stabilizing the closed-loop dynamics.
We show that for this nonsmooth optimization problem, all
Clarke stationary points are global minimum.
Next, we identify the coerciveness of the closed-loop H ∞ objective function, and prove that all the sublevel sets of the resultant policy search problem are compact.
Based on these properties, we show that Goldstein’s subgradient method and its implementable variants can be guaranteed to stay in the nonconvex feasible set and eventually find the global optimal solution of the H ∞ state-feedback synthesis problem.
Our work builds a new connection between nonconvex nonsmooth optimization theory and robust control, leading to an interesting global convergence result for direct policy search on optimal H ∞ synthesis.
Introduction
Reinforcement learning (RL) has achieved impressive performance on many continuous control tasks [57, 39] and policy optimization is one of the main workhorses for such applications [17, 63, 56, 58].
Recently, there have been extensive research efforts studying the global convergence properties of policy optimization methods on benchmark control problems including linear quadratic regulator (LQR) [20, 7, 40, 68, 43, 21, 28], stabilization [51, 50], linear robust/risk-sensitive control [71, 70, 25, 72, 73], Markov jump linear quadratic control [31, 30, 32, 53], output feedback control [19, 75, 38, 16, 15, 42, 74], and dynamic filtering [66].
For all these benchmark problems, the objective function in the policy optimization formulation is always differentiable over the entire feasible set, and the existing convergence theory heavily relies on this fact.
Consequently, an important open question remains whether direct policy search can enjoy similar global convergence properties when applied to the famous H ∞ control problem whose objective function can be non-differentiable over certain points in the policy space [1–3, 27, 9, 12, 47].
Different from LQR which considers stochastic disturbance sequences, H ∞ control directly addresses the worst-case disturbance, and provides arguably the most fundamental robust control paradigm [76, 18, 60, 4, 14, 22].
Regarding the connection with
RL, it has also been shown that H ∞ control can be applied to stabilize the training of adversarial
RL schemes in the linear quadratic setup [70, Section 5].
Given the fundamental importance of H ∞ control, we view it as an important benchmark for understanding the theoretical properties of direct policy search in the context of robust control and adversarial RL.
In this work, we study and prove the global convergence properties of direct policy search on the H ∞ state-feedback synthesis problem.
The objective of the H ∞ state-feedback synthesis is to design a linear state-feedback policy that stabilizes the closed-loop system and minimizes the H ∞ norm from the disturbance to a performance signal at the same time.
The design goal is also equivalent to synthesizing a state-feedback policy that minimizes a quadratic cost subject to the worst-case disturbance.
We will present the problem formulation for the H ∞ state-feedback synthesis and discuss such connections in Section 2.
Essentially, H ∞ state-feedback synthesis can be formulated as a constrained policy optimization problem
min K ∈K J ( K ) , where the decision variable K is a matrix parameterizing the linear statefeedback policy, the objective function J ( K ) is the closed-loop H ∞ -norm for given K , and the feasible set K consists of all the linear state-feedback policies stabilizing the closed-loop dynamics.
Notice that the feasible set for the H ∞ state-feedback control problem is the same as the nonconvex feasible set for the LQR policy search problem [20, 7].
However, the objective function J ( K ) for the
H ∞ control problem can be non-differential over certain feasible points, introducing new difficulty to direct policy search.
There has been a large family of nonsmooth H ∞ policy search algorithms developed based on the concept of Clarke subdifferential [1–3, 27, 9, 12].
However, a satisfying global convergence theory is still missing from the literature.
Our paper bridges this gap by making the following two contributions.
We show that all Clarke stationary points for the H ∞ state-feedback policy search problem are also global minimum. 2.
We identify the coerciveness of the H ∞ cost function and use this property to show that Goldstein’s subgradient method [24] and its implementable variants [69, 13, 9, 10, 36, 37] can be guaranteed to stay in the nonconvex feasible set of stabilizing policies during the optimization process and eventually find the global optimal solution of the H ∞ statefeedback control problem.
Finite-time complexity bounds for finding ( δ, ϵ )
-stationary
points are also provided.
Our work sheds new light on the theoretical properties of policy optimization methods on H ∞ control problems, and serves as a meaningful initial step towards a general global convergence theory of direct policy search on nonsmooth robust control synthesis.
Finally, it is worth clarifying the differences between H ∞ control and mixed H 2 / H ∞ design.
For mixed H 2 / H ∞ control, the objective is to design a stabilizing policy that minimizes an Hperformance bound and satisfies an H ∞ constraint at the same time [23, 35, 33, 46].
In other words, mixed H 2 / H ∞ control aims at improving the average H 2 performance while “maintaining" a certain level of robustness by keeping the closed-loop H ∞ norm to be smaller than a pre-specified number.
In contrast, H ∞ control aims at “improving" the system robustness and the worst-case performance via achieving the smallest closed-loop H ∞ norm.
In [71], it has been shown that the natural policy gradient method initialized from a policy satisfying the H ∞ constraint can be guaranteed to maintain the H ∞ requirement during the optimization process and eventually converge to the optimal solution of the mixed design problem.
However, notice that the objective function for the mixed H 2 / H ∞ control problem is still differentiable over all the feasible points, and hence the analysis technique in [71] cannot be applied to our H ∞ control setting.
More discussions on the connections and differences between these two problems will be given in the supplementary material.
Problem Formulation and Preliminaries 2.1 Notation
The set of p -dimensional real vectors is denoted as R p .
For a matrix A , we use the notation A T , ∥
A ∥ , tr A , σ min ( A ) , ∥
A ∥ 2 , and ρ ( A ) to denote its transpose, largest singular value, trace, smallest singular value, Frobenius norm, and spectral radius, respectively.
When a matrix P is negative semidefinite (definite), we will use the notation P ⪯ ( ≺ )0 .
When P is positive semidefinite (definite), we use the notation P ⪰ ( ≻ )0 .
Consider a (real) sequence u := { u 0 , u 1 , · · · } where u t ∈ R n u for all t .
This sequence is said to be in ℓ n u 2 if (cid:80) ∞ t =
∥
u t ∥ 2 < ∞ where ∥ u t ∥ denotes the standard (vector) 2-norm of u t .
In addition, the 2 -norm for u ∈ ℓ n u 2 is defined as ∥ u ∥ 2 := (cid:80) ∞ t =
∥
u t ∥ 2 .
2.2 Problem statement: H ∞ state-feedback synthesis and a policy optimization formulation
We consider the following linear time-invariant (LTI) system
[Equation]
where x t ∈ R n x is the state, u t ∈ R n u is the control action, and w t ∈ R n w is the disturbance.
We have A ∈ R n x × n x , B ∈ R n x × n u , and n w = n x .
We denote x := { x 0 , x 1 , · · · } , u := { u 0 , u 1 , · · · } , and w :=
{ w 0 , w 1 , · · · } .
The initial condition is fixed as x 0 = 0 .
The objective of H ∞ control is to choose { u t } to minimize the quadratic cost (cid:80) ∞ t =
( x T t Qx t + u T t Ru t ) in the presence of the worst-case ℓ 2 disturbance satisfying ∥ w ∥ ≤ 1 .
In this paper, the following assumption is adopted.
Assumption 1.
The matrices Q and R are positive definite.
The matrix pair ( A, B ) is stabilizable.
In H ∞ control, { w t } is considered to be the worst-case disturbance satisfying the ℓ 2 norm bound ∥ w ∥ ≤ 1 , and can be chosen in an adversarial manner.
This is different from LQR which makes stochastic assumptions on { w t } .
Without loss of generality, we have chosen the ℓ 2 upper bound on w to be 1 .
In principle, we can formulate the H ∞ control problem with any arbitrary ℓ 2 upper bound on w , and there is no technical difference.
We will provide more explanations on this fact in the supplementary material.
Therefore, H ∞ control can be formulated as the following minimax problem
[Equation]
Under Assumption 1, it is well known that the optimal solution for (2) can be achieved using a linear state-feedback policy u t = − Kx t (see [4]).
Given any K , the LTI system (1) can be rewritten as
[Equation]
Now we define z t =
( Q + K T RK ) 12 x t .
We have ∥ z t ∥ 2 = x T t ( Q + K T RK ) x t = x T t
Qx t + u T t Ru t .
We denote z := { z 0 , z 1 , · · · } .
If x ∈ ℓ n x 2 , then we have ∥ z ∥ 2 = (cid:80) ∞ t =0 ( x T t Qx t + u T t Ru t ) < + ∞ .
Therefore, the closed-loop LTI system (3) can be viewed as a linear operator mapping any disturbance sequence { w t } to another sequence { z t } .
We denote this operator as G K , where the subscript highlights the dependence of this operator on K .
If K is stabilizing, i.e. ρ ( A − BK ) < 1 , then
G K is bounded in the sense that it maps any ℓ 2 sequence w to another sequence z in ℓ n x 2 .
For any stabilizing K , the ℓ 2 → ℓ 2 induced norm of G K can be defined as:
[Equation]
Since G K is a linear operator, it is straightforward to show
[Equation]
Therefore, the minimax optimization problem (2) can be rewritten as the policy optimization problem: min K ∈K ∥
G K ∥ 22 → 2 , where K is the set of all linear state-feedback stabilizing policies, i.e. K = { K ∈ R n x × n u : ρ ( A − BK ) < 1 } .
In the robust control literature [1–3, 27, 9, 12], it is standard to drop the square in the cost function and just reformulate (2) as min K ∈K ∥
G K ∥ 2 → 2 .
This is exactly the policy optimization formulation for H ∞ state-feedback control.
The main reason why this problem is termed as H ∞ state-feedback control is that in the frequency domain, G K can be viewed as a transfer function which lives in the Hardy H ∞ space and has an H ∞ norm being exactly equal to ∥ G K ∥ 2 → 2 .
Applying the frequency-domain formula for the H ∞ norm, we can calculate ∥ G K ∥ 2 → 2 as
[Equation]
where I is the identity matrix, and λ max denotes the largest eigenvalue of a given symmetric matrix.
Therefore, eventually the H ∞ state-feedback control problem can be formulated as
[Equation]
where J ( K ) is equal to the H ∞ norm specified by (5).
Classical H ∞ control theory typically solves (6) via introducing extra Lyapunov variables and reparameterizing the problem into a higherdimensional convex domain over which convex optimization algorithms can be applied [76, 18, 6].
In this paper, we revisit (6) as a benchmark for direct policy search, and discuss how to search the optimal solution of (6) in the policy space directly.
Applying direct policy search to address (6) leads to a nonconvex nonsmooth optimization problem.
A main technical challenge is that the objective function (5) can be non-differentiable over some important feasible points [1–3, 27, 9, 12].
2.3 Direct policy search: A nonsmooth optimization perspective
Now we briefly review several key facts known for the H ∞ policy optimization problem (6).
Proposition 1.
The set K =
{ K : ρ ( A − BK ) < 1 } is open.
In general, it can be unbounded and nonconvex.
The cost function (5) is continuous and nonconvex in K .
See [20, 8] for some related proofs.
We have also included more explanations in the supplementary material.
An immediate consequence is that (6) becomes a nonconvex optimization problem.
Another important fact is that the objective function (5) is also nonsmooth.
As a matter of fact, (5) is subject to two sources of nonsmoothness.
Based on (5), we can see that the largest eigenvalue for a fixed frequency ω is nonsmooth, and the optimization step over ω ∈ [0 , 2 π ] is also nonsmooth.
As a matter of fact, the H ∞ objective function (5) can be non-differentiable over important feasible points, e.g. optimal points.
Fortunately, it is well known 1 that the H ∞ objective function (5) has the following desired property so it is Clarke subdifferentiable.
Proposition 2.
The H ∞ objective function (5) is locally Lipschitz and subdifferentially regular over the stabilizing feasible set K .
Recall that J : K
→ R is locally Lipschitz if for any bounded S ⊂ K , there exists a constant L >
such that | J ( K ) − J ( K ′ )
| ≤ L ∥ K − K ′ ∥ 2 for all K, K ′ ∈ S .
Based on Rademacher’s theorem, a locally Lipschitz function is differentiable almost everywhere, and the Clarke subdifferential is well defined for all feasible points.
Formally, the Clarke subdifferential is defined as
[Equation]
where conv denotes the convex hull.
Then we know that the Clarke subdifferential for the H ∞ objective function (5) is well defined for all K ∈ K .
We say that K is a Clarke stationary point if 0 ∈ ∂ C J ( K ) .
The following fact is also well known.
Proposition 3.
If K is a local min of J , then 0 ∈ ∂ C J ( K ) and K is a Clarke stationary point.
Under Assumption 1, it is well known that there exists K ∗ ∈ K achieving the minimum of (6).
Since
K is an open set, K ∗ has to be an interior point of K and hence K ∗ has to be a Clarke stationary point.
In Section 3, we will prove that any Clarke stationary points for (6) are actually global minimum.
Now we briefly elaborate on the subdifferentially regular property stated in Proposition 2.
For any given direction d (which has the same dimension as K ), the generalized Clarke directional derivative of J is defined as
[Equation]
In contrast, the (ordinary) directional derivative is defined as follows (when existing)
[Equation]
the proof of Proposition 2 is hinted in the last paragraph of [2, Section III] given the facts that the H ∞ norm is a convex function over the Hardy H ∞ space (which is a Banach space) and the mapping from K ∈ K to the (infinite-dimensional) Hardy H ∞ space is strictly differentiable.
For completeness, a simple proof of Proposition 2 based on Clarke’s chain rule [11] is included in the supplementary material.
In general, the Clarke directional derivative can be different from the (ordinary) directional derivative.
Sometimes the ordinary directional derivative may not even exist.
The objective function J ( K ) is subdifferentially regular if for every K ∈ K , the ordinary directional derivative always exists and coincides with the generalized one for every direction, i.e. J ′ ( K, d )
=
J ◦
( K, d ) .
The most important consequence of the subdifferentially regular property is given as follows.
Corollary 1.
Suppose K † ∈ K is a Clarke stationary point for J .
If J is subdifferentially regular, then the directional derivatives J ′
( K † , d ) are non-negative for all d .
See [54, Theorem 10.1] for related proofs and more discussions.
Notice that having non-negative directional derivatives does not mean that the point K † is a local minimum.
Nevertheless, the above fact will be used in our main theoretical developments.
Now we briefly summarize two key difficulties in establishing a global convergence theory for direct policy search on the H ∞ state-feedback control problem (6).
First, it is unclear whether the direct policy search method will get stuck at some local minimum.
Second, it is challenging to guarantee the direct policy search method to stay in the nonconvex feasible set K during the optimization process.
Since K is nonconvex, we cannot use a projection step to maintain feasibility.
Our main results will address these two issues.
2.4 Goldstein subdifferential
Generating a good descent direction for nonsmooth optimization is not trivial.
Many nonsmooth optimization algorithms are based on the concept of Goldstein subdifferential [24].
Before proceeding to our main result, we briefly review this concept here.
Definition 1 (Goldstein subdifferential) .
Suppose J is locally Lipschitz.
Given a point K ∈ K and a parameter δ > 0 , the Goldstein subdifferential of J at K is defined to be the following set
[Equation]
(10) where B δ ( K ) denotes the δ -ball around K .
The above definition implicitly requires B δ ( K ) ⊂ K .
Based on the above definition, one can further define the notion of ( δ, ϵ ) -stationarity.
A point K is said to be ( δ, ϵ ) -stationary if dist(0 , ∂ δ J ( K ))
≤ ϵ .
It is well-known that the minimal norm element of the Goldstein subdifferential generates a good descent direction.
This fact is stated as follows.
Proposition 4 ([24]) .
Let F be the minimal norm element in ∂ δ J ( K ) .
Suppose K − αF/ ∥ F ∥ 2 ∈ K for any 0 ≤ α ≤ δ .
Then we have
[Equation]
The idea of Goldstein subdifferential has been used in designing algorithms for nonsmooth H ∞ control [3, 27, 9, 12].
We will show that such policy search algorithms can be guaranteed to find the global minimum of (6).
It is worth mentioning that there are other notions of enlarged subdifferential [2] which can lead to good descent directions for nonsmooth H ∞ synthesis.
In this paper, we focus on the notion of Goldstein subdifferential and related policy search algorithms.
Optimization Landscape for H ∞ State-Feedback Control
In this section, we investigate the optimization landscape of the H ∞ state-feedback policy search problem, and show that any Clarke stationary points of (6) are also global minimum.
We start by showing the coerciveness of the H ∞ objective function (5).
Lemma 1.
The H ∞ objective function J ( K ) defined by (5) is coercive over the set K in the sense that for any sequence { K l } ∞ l =1 ⊂ K we have J ( K l ) →
+ ∞ , if either ∥ K l ∥ 2 → + ∞ , or K l converges to an element in the boundary ∂ K .
Proof.
We will only provide a proof sketch here.
A detailed proof is presented in the supplementary material.
Suppose we have a sequence { K l } satisfying ∥ K l ∥ 2 → + ∞ .
We can choose w = { w 0 , 0 , 0 , · · · } with ∥ w 0 ∥ = 1 and show J ( K l ) ≥ w T 0 ( Q + ( K l ) T RK
[Equation]
Clearly, we have used the positive definiteness of R in the above derivation.
Then by carefully choosing w 0 , we can ensure J ( K l ) → + ∞ as ∥ K l ∥ 2 → + ∞ .
Next, we assume K l → K ∈ ∂ K .
We have ρ ( A − BK )
= 1 , and hence there exists some ω 0 such that ( e jω
I − A + BK ) becomes singular.
Then we can use the positive definiteness of Q to show J ( K l ) ≥ λ 1 / 2 min ( Q ) ∥ ( e jω
I − A
+ BK l ) − 1 ∥ .
Notice σ min ( e jω
I − A
+ BK l ) → 0 as l → ∞ , which implies ∥ ( e jω
I − A
+ BK l ) − 1 ∥ → + ∞ as l → ∞ .
Therefore, we have J ( K l ) → + ∞ as K l → K ∈ ∂ K .
More details for the proof can be found in the supplementary material.
We want to emphasize that the positive definiteness of ( Q, R ) are crucial for proving the coerciveness of the cost function (5).
Built upon Lemma 1, we can obtain the following nice properties of the sublevel sets of (6).
Lemma 2. Consider the H ∞ state-feedback policy search problem (6) with the objective function
J ( K ) defined in (5) .
Under Assumption 1, the sublevel set defined as K γ :=
{ K ∈ K : J ( K ) ≤ γ } is compact and path-connected for every γ ≥ J ( K ∗ ) where K ∗ is the global minimum of (6) .
Proof.
The compactness of K γ directly follows from the continuity and coerciveness of J ( K ) , and is actually a consequence of [5, Proposition 11.12].
The path-connectedness of the strict sublevel sets for the continuous-time H ∞ control problem has been proved in [29].
We can slightly modify the proof in [29] to show that the strict sublevel set { K ∈ K : J ( K ) < γ } is path-connected.
Based on the fact that every non-strict sublevel sets are compact, now we can apply [41, Theorem 5.2] to show
K γ is also path-connected.
An independent proof based on the non-strict version of the bounded real lemma is also provided in the supplementary material.
The path-connectedness of K γ for every γ actually implies the uniqueness of the minimizing set in a certain strong sense [41, Sections 2&3].
Due to the space limit, we will defer the discussion on the uniqueness of the minimizing set to the supplementary material.
Here, we present a stronger result which is one of the main contributions of our paper.
Theorem 1.
Consider the H ∞ state-feedback policy search problem (6) .
Under Assumption 1, any
Clarke stationary point of J ( K ) is a global minimum.
A detailed proof is presented in the supplementary material.
Here we provide a proof sketch.
Since Q and R are positive definite, the non-strict version of the bounded real lemma 2 states that J ( K ) ≤ γ if and only if there exists a positive definite matrix P such that the following matrix inequality holds
[Equation]
The above matrix inequality is linear in P but not linear in K .
A standard trick from the control theory can be combined with the Schur complement lemma to convert the above matrix inequality condition to another condition which is linear in all the decision variables [6].
Specifically, there exists a matrix function LMI( Y, L, γ ) which is linear in ( Y, L, γ ) such that LMI( Y, L, γ ) ⪯ 0 and Y ≻
if and only if (12) is feasible with K = LY − 1 and P = γY − 1 ≻ 0 .
The matrix function LMI( Y, L, γ ) involves a larger matrix.
Hence we present the analytical formula of LMI( Y, L, γ ) in the supplementary material and skip it here.
Since LMI( Y, L, γ ) is linear in ( Y, L, γ ) , we know LMI( Y, L, γ ) ⪯ 0 is just a convex semidefinite programming condition.
Based on this convex necessary and sufficient condition for J ( K ) ≤ γ , we can prove the following important lemma.
Lemma 3.
For any K ∈ K satisfying J ( K ) > J ∗ , there exists a matrix direction d ̸ = 0 such that
[Equation]
d
[Equation]
J
[Equation]
− J ( K ) < 0 , where J ∗ =
J ( K ∗ ) and K ∗ is the global minimum of (6) .
Proof.
Suppose we have K = LY − 1 where ( Y, L, J ( K )) is a feasible point for the convex regime
LMI( Y, L, J ( K )) ⪯ 0 .
In addition, we have K ∗ =
L ∗ ( Y ∗ ) − 1 where ( Y ∗ , L ∗ , J ( K ∗ )) is a point satisfying LMI( Y ∗ , L ∗ , J ( K ∗ )) ⪯ 0 .
Since the LMI condition is convex, the line segment between ( Y, L, J ( K )) and ( Y ∗ , Q ∗ , J ( K ∗ )) is also in this convex set.
For any t > 0 , we know ( Y + t ∆ Y, L + t ∆ L, J ( K ) + t ( J ( K ∗ ) − J ( K ))) also satisfies LMI( Y + t ∆ Y, L + t ∆ L, J ( K ) + t ( J ( K ∗ ) −
J ( K ))) ⪯ 0 , where ∆ L = L ∗ − L , and ∆ Y
=
Y ∗ − Y .
Therefore, based on the bounded real lemma, we know J (( L + t ∆ L )( Y + t ∆ Y ) − 1 )
≤ J ( K ) + t ( J ( K ∗ ) − J ( K )) .
Let’s choose d = ∆ LY − 1 − LY − 1 ∆ Y Y − 1 .
Then we have
[Equation]
A detailed verification of the above inequality is provided in the supplementary material.
Notice d ̸ = 0 .
If ∆ LY − 1 − LY − 1 ∆ Y Y − 1 = 0 , the above argument still works and we reach to the conclusion J ′ ( K, 0) < 0 .
But this is impossible since we always have J ′ ( K, 0)
= 0 .
Hence we have d ̸ = 0 .
This completes the proof for this lemma.
Now we are ready to provide the proof for Theorem 1.
Based on Lemma 3 and the fact that J ( · ) is subdifferentially regular, the proof can be done by contradiction.
Suppose K ∗ is the global minimum, and K † is a Clarke stationary point.
If K † is not a global minimum.
Then by Lemma 3, there exists d ̸ = 0 such that J ′
( K † , d ) < 0 , this contradicts the fact that J ′ ( K † , d ) ≥ 0 for all d by Corollary 1.
Therefore, K † has to be the global minimum of (6).
The above proof relies on Lemma 3 and the fact that J is subdifferentially regular.
Without using the subdifferentially regular property, Lemma 3 itself is not sufficient for proving Theorem 1.
It is also worth mentioning that Lemma 3 can be viewed as a modification of the convex parameterization/lifting results in [62, 66] for non-differentiable points.
Global Convergence of Direct Policy Search on H ∞ State-Feedback Control
In this section, we first show that Goldstein’s subgradient method [24] can be guaranteed to stay in the nonconvex feasible regime K during the optimization process and eventually converge to the global minimum of (6).
The complexity of finding ( δ, ϵ ) -stationary points of (6) is also presented.
Then we further discuss the convergence guarantees for various implementable forms of Goldstein’s subgradient method.
4.1 Global convergence and complexity of Goldstein’s subgradient method
We will investigate the global convergence of Goldstein’s subgradient method for direct policy search of the optimal H ∞ state-feedback policy.
Goldstein’s subgradient method iterates as follows
[Equation]
(13) where F n is the minimum norm element of the Goldstein subdifferential ∂ δ n J ( K n ) .
We assume that an initial stabilizing policy is available, i.e. K 0 ∈ K .
The same initial policy assumption has also been made in the global convergence theory for direct policy search on LQR [20].
More recently, some provable guarantees have been obtained for finding such stabilizing policies via direct policy search [51, 50].
Hence such an assumption on the initial policy K 0 is reasonable.
Our global convergence result relies on the fact that there is a strict separation between any sublevel set of (6) and the boundary of K .
This fact is formalized as follows.
Lemma 4. Consider the H ∞ state-feedback policy search problem (6) with the cost function J ( K ) defined in (5) .
Denote the complement of the feasible set K as K c .
Suppose Assumption 1 holds and γ ≥ J ∗ .
Then there is a strict separation between the sublevel set K γ and K c .
In other words, we have dist ( K γ , K c ) > 0 .
Proof.
Obviously, the set K γ ∩ K c is empty (since we know K γ ⊂ K ).
Based on Lemma 2, we know K γ is compact.
Since K is open, we know K c is closed.
Therefore, there is a strict separation between K γ and K c , and we have dist ( K γ , K c ) > 0 .
Now we are ready to present our main convergence result.
Theorem 2.
Consider the H ∞ state-feedback policy search problem (6) with the cost function J ( K ) defined in (5) .
Suppose Assumption 1 holds, and an initial stabilizing policy is given, i.e. K 0 ∈ K .
Denote ∆ 0 :
=
dist( K J ( K 0 ) , K c ) > 0 .
Choose δ n = c ∆ 0 n +1 for all n with c being a fixed number in (0 , 1) .
Then Goldstein’s subgradient method (13) is guaranteed to stay in K for all n .
In addition, we have J ( K n ) → J ∗ as n → ∞ .
Proof.
We have δ n ≤ c ∆ 0 < ∆ 0 for all n .
Now we use an induction proof to show K n ∈ K J ( K 0 ) for all n .
For n = 0 , we know K 0 − c ∆ 0 F 0 / ∥ F 0 ∥ 2 has to be within the ∆ 0 ball around K 0 since we know the norm of F 0 / ∥ F 0 ∥ 2 is exactly equal to 1 .
Since ∆ 0 := dist( K J ( K 0 ) , K c ) > 0 , we know
K 0 − δ 0 F 0 / ∥
F 0 ∥ 2 ∈ K .
As a matter of fact, we know B δ
( K 0 ) has to be a subset of K .
Hence we can apply (11) to show that K 1 exists and is also in K J ( K 0 ) .
Similarly, we can repeat this argument to show K n ∈ K J ( K 0 ) for all n .
Next, we can apply (11) to every step and then sum the inequalities over all n .
Then the following inequality holds for all N :
[Equation]
Since we have (cid:80) ∞ n =0 δ n = + ∞ , we know lim inf n →∞ ∥
F n ∥ 2 = 0 .
There exists one subsequence { i n } such that ∥ F i n ∥ 2 → 0 .
For this subsequence, the resultant policy sequence { K i n } is also bounded (notice that the policy parameter sequence stays in the compact set K J ( K 0 ) for all n ) and has a convergent subsequence.
We can show that the limit of this subsequence is a Clarke stationary point.
Hence the function value associated with this subsequence converges to J ∗ .
Notice that J ( K n ) is monotonically decreasing for the entire sequence { n } .
Hence we have J ( K n ) → J ∗ .
We have tried to be brief in giving the above proof.
We will present a more detailed proof in the supplementary material.
We believe that this is the first result showing that direct policy search can be guaranteed to converge to the global optimal solution of the H ∞ state-feedback control problem.
The above result only provides an asymptotic convergence guarantee to ensure J ( K n ) → J ∗ .
One can use a similar argument to establish a finite-time complexity bound for finding the ( δ, ϵ ) -stationary
points of (6).
Such a result is given as follows.
Theorem 3.
Consider the H ∞ problem (6) with the cost function (5) .
Suppose Assumption 1 holds, and K 0 ∈ K .
Denote ∆ 0 :
= dist( K J ( K 0 ) , K c ) > 0 .
For any δ < ∆ 0 , we can choose δ n = δ for all n to ensure that Goldstein’s subgradient method (13) stays in K and satisfies the following finite-time complexity bound:
[Equation]
In other words, we have min 0 ≤ n ≤ N ∥ F n ∥ 2 ≤ ϵ after N = O (cid:0) ∆ δϵ (cid:1) where ∆ :=
J ( K 0 ) − J ∗ .
For any δ < ∆ 0 and ϵ > 0 , the complexity of finding a ( δ, ϵ ) -stationary point is O (cid:0) ∆ δϵ (cid:1) .
Proof.
The above result can be proved using a similar argument from Theorem 2.
We can use the same induction argument to show K n ∈ K J ( K 0 ) for all n , and (14) holds with δ n = δ .
Then the desired conclusion directly follows.
The complexity for nonsmooth optimization of Lipschitz functions is quite subtle.
While the above result gives a reasonable characterization of the finite-time performance of Goldstein’s subgradient method on the H ∞ state-feedback control problem, it does not quantify how fast J ( K n ) converges to J ∗ .
Recall that ( δ, ϵ ) -stationarity means dist(0 , ∂ δ J ( K ))
≤ ϵ , while ϵ -stationarity means dist(0 , ∂ C J ( K ))
≤ ϵ .
As commented in [59, 69, 13], ( δ, ϵ ) -stationarity does not imply being δ -close to an ϵ -stationary point of J .
Importantly, the function value of a ( δ, ϵ ) -stationary point can be far from J ∗ even for small δ and ϵ .
Theorem 5 in [69] shows that there is no finite time algorithm that can find ϵ -stationary points provably for all Lipschitz functions.
It is still possible that one can develop some finite time bounds for ( J ( K n ) − J ∗ ) via exploiting other advanced properties of the
H ∞ cost function (5).
This is an important future task.
4.2 Implementable variants and related convergence results
In practice, it can be difficult to evaluate the minimum norm element of the Goldstein subdifferential.
Now we discuss implementable variants of Goldstein’s subgradient method and related guarantees.
Gradient sampling [9, 10, 36].
The gradient sampling (GS) method is the main optimization algorithm used in the robust control package HIFOO [3, 27].
Suppose we can access a first-order oracle which can evaluate ∇ J for any differentiable points in the feasible set 3 .
Based on Rademacher’s theorem, a locally Lipschitz function is differentiable almost everywhere.
Therefore, for any K n ∈ K , we can randomly sample policy parameters over B δ n ( K n ) and obtain differentiable points with probability one.
For all these sampled differentiable points, the Clarke subdifferential at each point is just the gradient.
Then the convex hull of these sampled gradients can be used as an approximation for the Goldstein subdifferential ∂ δ n K n .
The minimum norm element from the convex hull of the sampled gradients can be solved via a simple convex quadratic program, and is sufficient for generating a reasonably good descent direction for updating K n +1 as long as we sample at least ( n x n u + 1) differentiable points for each n [9].
In the unconstrained setup, the cluster points of the
GS algorithm can be guaranteed to be Clarke stationary [36, 9].
Such a result can be combined with
Theorem 1 and Lemma 4 to show the global convergence of the GS method on the H ∞ state-feedback synthesis problem.
The following theorem will be treated formally in the supplementary material.
Theorem 4 (Informal statement) .
Consider the policy optimization problem (6) with the H ∞ cost function defined in (5) .
Suppose Assumption 1 holds, and K 0 ∈ K .
The iterations generated from the trust-region version of the GS method (described in [36, Section 4.2] and restated in the supplementary material) can be guaranteed to stay in K for all iterations and achieve J ( K n ) → J ∗ with probability one.
Non-derivative sampling (NS) [37].
The NS method can be viewed as the derivative-free version of the GS algorithm.
Suppose we only have the zeroth-order oracle which can evaluate the function value J ( K ) for K ∈ K .
The main difference between NS and GS is that the NS algorithm relies on estimating the gradient from function values via Gupal’s estimation method.
In the unconstrained setting, the cluster points of the NS method can be guaranteed to be Clarke stationary with probability one [37, Theorem 3.8].
We can combine [37, Theorem 3.8] with our results (Theorem 1 and
Lemma 4) to prove the global convergence of NS in our setting.
A detailed discussion is given in the supplementary material.
Model-free implementation of NS.
When the system model is unknown, there are various methods available for estimating the H ∞ -norm from data [44, 45, 55, 52, 67, 49, 65, 64].
Based on our own experiences/tests, the multi-input multi-output (MIMO) power iteration method [48] works quite well as a stochastic zeroth-order oracle for the purpose of implementing NS in the model-free setting.
While the sample complexity for model-free NS is unknown, we will provide some numerical justifications to show that such a model-free implementation closely tracks the convergence behaviors of its model-based counterpart.
Interpolated normalized gradient descent (INGD) with finite-time complexity.
No finite-time guarantees for finding ( δ, ϵ ) -stationary
points have been reported for the GS/NS methods.
In [69, 13], the INGD method has been developed as another implementable variant of Goldstein’s subgradient method, and is proved to satisfy high-probability finite-time complexity bounds for finding ( δ, ϵ ) - stationary points of Lipschitz functions.
INGD uses an iterative sampling strategy to generate a descent direction which serves a role similar to the minimal norm element of the Goldstein subdifferential.
A first-order oracle for differentiable points is needed for implementing INGD 4 .
It has been show [69, 13] that for unconstrained nonsmooth optimization of L -Lipschitz functions, the INGD algorithm can be guaranteed to find the ( δ, ϵ ) -stationary point with the high-probability iteration complexity O (cid:16)
∆ L 2 ϵ 3 δ log( 1 p ) (cid:17) , where ∆ :=
J ( K 0 ) − J ∗ is the initial function value gap, and p is the failure probability (i.e. the optimization succeeds with the probability (1 − p ) ).
We can combine the proofs for [13, Theorem 2.6] and Theorem 3 to obtain the following complexity result for our H ∞ setting.
A formal treatment is given in the supplementary material.
Theorem 5 (Informal statement) .
Consider the policy optimization problem (6) with the H ∞ cost function defined in (5) .
Suppose Assumption 1 holds, and the initial policy is stabilizing, i.e. K 0 ∈ K .
Denote ∆ 0 :
=
dist( K J ( K 0 ) , K c ) > 0 , and let L 0 be the Lipschitz constant of J ( K ) over the set
K J ( K 0 ) .
For any δ < ∆ 0 , we can choose δ n = δ for all n to ensure that the iterations of the INGD algorithm stay in K almost surely, and find a ( δ, ϵ ) -stationary point with the high-probability iteration complexity O (cid:16)
∆ L 20 ϵ 3 δ log( 1 p ) (cid:17) , where p is the failure probability.
[Figure]
Numerical Simulations
To support our theory, we provide some numerical simulations in this section.
The left plot in Figureshows that GS, NS, INGD, and model-free NS work well for the following example:
[Equation] [Equation]
For this example, we have J ∗ = 7 . 3475 .
We initialize from K 0 =
4931 − 0 .
1368 − 2 . 2654] , which satisfies ρ ( A − BK 0 ) = 0 .
5756 < 1 .
The hyperparameter choices are detailed in the supplementary material.
We can see that model-free NS closely tracks the trajectory of NS and works well.
In the middle plot of Figure 1, we test the NS method on randomly generated cases.
We set
A ∈ R 3 × 3 to be I + ξ , where each element of ξ ∈ R 3 × 3 is sampled uniformly from [0 , 1] .
For
B ∈ R 3 × 1 , each element is uniformly sampled from [0 , 1] .
We have Q =
I + ζI ∈ R 3 × 3 with ζ uniformly sampled from [0 , 0 . 1] , and R ∈ R uniformly sampled from [1 , 1 . 5] .
For each experiment, the initial condition K 0 ∈ R 1 × 3 is also randomly sampled such that ρ ( A − BK 0 ) < 1 .
The NS method converges globally for all the cases.
In the right plot, we focus on the model-free setting for (16).
We decrease the number of samples used in the H ∞ estimation and show how this increases the noise in the zeroth-order H ∞ oracle and worsens the convergence behaviors of the model-free NS method.
Nevertheless, the model-free NS method tracks its model-based counterpart with enough samples.
More numerical results can be found in the supplementary material.
Conclusions and Future Work
In this paper, we developed the global convergence theory for direct policy search on the H ∞ statefeedback synthesis problem.
Although the resultant policy optimization formulation is nonconvex and nonsmooth, we managed to show that any Clarke stationary points for this problem are actually global minimum, and the concept of Golstein subdifferential can be used to build direct policy search algorithms which are guaranteed to converge to the global optimal solutions.
The finite-time guarantees in this paper are developed only for finding ( δ, ϵ ) -stationary points.
An important future task is to investigate the finite-time bounds for the optimality gap (i.e. J ( K n ) − J ∗ ) as well as the sample complexity of direct policy search on model-free H ∞ control.
It is also of great interests to investigate the convergence properties of direct policy search in nonlinear/output-feedback settings 5 .
Acknowledgments and Disclosure of Funding
This work is generously supported by the NSF award CAREER-2048168 and the 2020 Amazon research award.
The authors would like to thank Michael L. Overton, Maryam Fazel, Yang Zheng,
Peter Seiler, Geir Dullerud, Aaron Havens, Darioush Kevian, Kaiqing Zhang, Na Li, Mehran Mesbahi,
Tamer Ba¸sar, Mihailo Jovanovic, and Javad Lavaei for the valuable discussions, as well as the helpful suggestions from the anonymous reviewers of NeurIPS.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Our works only addresses the H ∞ state-feedback control problem.
More study is needed for the partial observation case.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We do not believe that our current work will create any negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix in the supplementary material.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Our code has been included in the supplementary material.
We emphasize that the nature of our work is theoretical, and our code is only provided to demonstrate our theory.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Supplementary Material A More Details on Problem Formulation and Background A.1 Difference between H ∞ control and mixed H 2 / H ∞ Design
In this paper, we are interested in H ∞ control whose objective is to design a state-feedback policy K that stabilizes the closed-loop system and minimizes the closed-loop H ∞ norm (or equivalently the ℓ 2 → ℓinduced norm) of G K at the same time.
As discussed in the main paper, H ∞ control can be formulated as min
K ∈K ∥
G K ∥ 2 → 2 .
We have mentioned that such a formulation corresponds to a worst-case assumption on the disturbance w .
Since ∥ G K ∥ 2 → 2 is a nonsmooth function in K , the H ∞ control leads to a nonsmooth optimization problem.
The global optimal H ∞ norm for this problem is denoted as γ ∗ .
In contrast, the design objective of the mixed H 2 / H ∞ control is to synthesize a a linear state-feedback policy which minimizes an upper bound on the H 2 cost and satisfies an additional H ∞ -robustness requirement in the form of ∥ G K ∥ 2 → 2 < γ with some pre-specified value of γ .
Recall that we denote the non-strict γ -admissible sublevel set of the H ∞ objective function as K γ =
{ K ∈ K : ∥ G K ∥ 2 → 2 ≤ γ } .
Similarly, we definie the strict γ -admissible sublevel set of the H ∞ objective function as ˜ K γ
=
{ K ∈ K : ∥ G K ∥ 2 → 2 < γ } .
Then for any γ > γ ∗ , the H 2 / H ∞ mixed design problem can be formulated as
[Equation]
where P K is the minimal positive definite solution to the following Riccati equation
[Equation]
Notice that as γ → ∞ , the above mixed design problem reduces to the LQR problem.
For finite γ , tr( P K ) provides an upper bound for the LQR cost.
For the mixed H 2 / H ∞ control problem, the closed-loop H ∞ norm ∥ G K ∥ 2 → 2 only appears in the constraint.
One can show that cost function tr( P K ) is still differentiable in the policy space for any K ∈ ˜ K .
The main result in [71] states that given an initial policy K 0 ∈ ˜ K γ for γ > γ ∗ , then the natural policy gradient method with well-chosen stepsize is guaranteed to stay in ˜ K γ and fine the optimal policy minimizes tr( P K ) .
This means that the natural policy gradient method applied to this mixed design problem can improve the average performance (via minimizing tr( P K ) ) while maintaining the robustness level γ .
A missing step is how to use direct policy search to obtain such an initial policy satisfying ∥ G
K 0 ∥ 2 → 2 < γ in the first place.
One by-product of the results in this paper is a direct policy search method for obtaining such initial policy provably.
It is worth mentioning that it is also possible to to tackle the H ∞ state-feedback synthesis via solving a sequence of mixed H 2 / H ∞ design problems, and the approximate central path algorithm in [34] is developed based on such an idea.
However, there is no global convergence theory reported for such successive minimization approaches [34].
A.2 Formulating H ∞ control with arbitrary ℓ 2 bound on the disturbance
To address the worst-case disturbance w , one may be interested in the following problem formulation
[Equation]
where Λ is an arbitrary positive number.
An interesting fact is that (A.1) still leads to the same policy optimization problem min
K
[Equation]
∥ G K ∥ 2 → 2 regardless of the value of Λ .
Now we briefly explain this fact.
Again, it is well- know that it suffices to consider linear state-feedback policies for solving (A.1).
Notice that G K is a linear operator for any fixed K .
Therefore, it is straightforward to verify
[Equation]
Consequently, (A.1) can be equivalently formulated as min K
[Equation]
The constant Λ 2 can be removed without changing the optimization problem.
Therefore, no matter what positive value we use for Λ , (A.1) is always equivalent to min K ∈K ∥
G ∥ 2 → 2 .
A.3 More explanations for Proposition
Now we provide some explanations for Proposition 1 which states several facts observed in the existing literature.
The continuity of the cost function (5) can be seen from its analytic form.
The openness of K can be seen from the fact that the spectral radius ρ ( A − BK ) is a continuous function of K .
The unboundedness of K has been proved in [8].
To see that K can be nonconvex, we provide a simple example here.
Suppose
A = B
= I , and we choose two policies as follows
[Equation]
Then one can easily check that K, K ′ ∈ K , while their convex combination 12 ( K 1 + K 2 ) / ∈ K .
Finally, we provide an example to show J ( K ) is nonconvex in K .
Suppose A = B = Q = R
=
I and choose two policies as follows:
[Equation]
Let K ′′ =
12 ( K + K ′ ) , then we have J ( K ′′ ) = 35 .
99 and 12 ( J ( K ) + J ( K ′ ))
= 13 .
Hence J ( K ′′ ) > 0 .
5 J ( K ) + 0 .
5 J ( K ′ ) , which shows J ( K ) is nonconvex in K .
A.4 Proof of Proposition
Proposition 2 is a consequence of the chain rule stated in [11, Theorem 2.3.10].
As mentioned previously, Proposition 2 should be considered as a well-known fact, and we do not claim any credits in proving it.
Although not explicitly stated, the proof of Proposition 2 has been hinted in the discussion of [2, Section III].
We only present the proof for completeness.
To apply [11, Theorem 2.3.10], we can rewrite the closed-loop H ∞ norm ∥ G K ∥ 2 → 2 as a composition g 2 ◦ g 1 ( K ) where g 1 maps the policy parameter K to a stable transfer function ( Q + K T RK ) 12 ( zI − A + BK ) − 1 which lives in the infinite-dimensional H ∞ Hardy space, and g 2 maps any stable transfer function in that Hardy space to its H ∞ norm.
It is well known that the infinite-dimensional H ∞ Hardy space consisting of all stable LTI systems is a Banach space [76].
We also know that g 2 is convex on the Hardy space (by triangle inequality of the H ∞ norm).
The convexity of g 2 implies that g 2 is also subdifferentially regular.
One can also show that g 1 is a strictly differentiable mapping from K to the Hardy space given ρ ( A − BK ) <
1 and Q being positive definite.
Now we can immediately apply [11, Theorem 2.3.10] to obtain the desired conclusion in Proposition 2.
Remark 1.
For readability, we also provide some brief explanations for the fact that the mapping g 1 is strictly differentiable.
For each K ∈ K , g 1 maps K to an infinite-dimensional bounded operator which maps from ℓ n wto ℓ n x 2 and has a Toeplitz structure.
The Toeplitz structure can be combined with Fubini’s theorem to show simple upper bounds for the operator norm of g 1 .
One such upper bound is provided by ∥ ( Q + K T RK ) 12 ∥ (cid:80) ∞ k =0 ∥
( A − BK ) k ∥ , which is obviously finite for K ∈ K .
Such a bound allows us to study the operator g 1 via treating it as an infinite-dimensional Toeplitz matrix whose properties are completely determined by its first block column.
By definition, g 1 admits a strict derivative at K if the following operator norm convergence result holds for any V :
[Equation]
where ∥·∥ denotes the operator norm, and D s g 1 is the candidate for the strict derivative defined via taking derivatives of the infinite-dimensional matrix g 1 in a block-by-block manner.
Since we know ρ ( A − BK ) < 1 , the Toeplitz structure of g 1 can be combined with Fubini’s theorem again to prove the existence of D s g 1 and provide simple upper bounds for the operator norm on the left side of (A.2) , leading to the desired operator norm convergence result in (A.2) .
A.5 A non-strict version of the bounded real lemma
The bounded real lemma (also referred to as the Kalman–Yakubovich–Popov (KYP) lemma) is an important tool for characterizing the sublevel sets of the closed-loop H ∞ norm.
There is some subtlety in the assumptions needed for different versions of the bounded real lemma.
We briefly clarify this subtlety here.
Consider the following closed-loop linear system:
[Equation]
where ρ ( A cl ) < 1 .
Recall that the closed-loop H ∞ norm of the above system (from { w t } to { z t } ) is defined as
[Equation]
The strict version of the bounded real lemma [76, Theorem 21.12] states that H cl < γ if and only if there exists a positive definite matrix P such that
[Equation]
The above strict-version version of the bounded real lemma does not require any assumptions on ( A cl , B cl , C cl ) .
There is also a non-strict version of the bounded real lemma which requires the extra assumption that ( A cl , B cl , C cl ) is minimal (i.e. ( A cl , B cl ) is controllable and ( A cl , C cl ) is observable).
Consider (A.3) with ( A cl , B cl , C cl ) being a minimal realization.
Then the non-strict version of the bounded real lemma 6 states that H cl ≤ γ if and only if there exists a positive definite matrix P such that
[Equation]
For the main result in this paper, the proof require the above non-strict version of the bounded real lemma.
Specifically, in our setting, we have ∥ G K ∥ 2 → 2 = H cl if we choose A cl =
A − BK , B cl =
I , and C cl =
( Q + K T RK ) 12 .
Since Q is positive definite, such choice of ( A cl , B cl , C cl ) leads to a minimal realization, and hence it is valid to apply the non-strict version of the bounded real lemma to characterize the non-strict sublevel sets of ∥ G
K ∥ 2 → 2 .
B Detailed Proofs of Our Main Results in Sections 3 &
In the main paper, we have only provided the proof sketches for Lemma 1, Lemma 2, Theorem 1, Lemma 3, and Theorem 2.
In this section, we provide detailed proofs for these results.
B.1 Proof for Lemma
Suppose we have a sequence { K l } satisfying ∥ K l ∥ 2 → + ∞ .
We can choose w = { w 0 , 0 , 0 , · · · } with ∥ w 0 ∥ = 1 .
Then we have:
[Equation]
where the first inequality holds since we plugged into a specific w over the max operation and the matrix Q
+ ( K l ) T RK l is positive definite.
The second inequality uses the fact that R ≥ λ min ( R ) I .
Then by carefully choosing w 0 , we can ensure J ( K l ) → + ∞ as ∥ K l ∥ → + ∞ .
Next, we assume K l → K where K is on the boundary ∂ K .
Clearly we have ρ ( A − BK )
= 1 .
We will use a frequency-domain argument to prove J ( K l ) → + ∞ .
Since ρ ( A − BK ) = 1 , there exists some ω 0 such that the matrix ( e jω
I − A + BK ) becomes singular.
Obviously, for the same ω 0 , the matrix ( e − jω
I − A + BK ) is also singular.
Therefore, we have:
[Equation]
Clearly, the above argument relies on the fact that Q is positive definite.
Notice that we have ρ ( A − BK l ) < 1 for each l , and hence we have σ min ( e ± jω
I − A
+ BK l ) > 0 , i.e. the smallest singular values of ( e ± jω
I − A
+ BK l ) are strictly positive for all l .
As l → ∞ , one can show that the matrices ( e ± jω
I − A
+ BK l ) converge to the singular matrices ( e ± jω
I − A + BK ) with K =
lim
l →∞ K l ∈ ∂ K .
Hence we have σ min ( e ± jω
I − A
+ BK l ) → 0 as l → ∞ , which implies ∥ ( e ± jω
I − A
+ BK l ) − 1 ∥ → + ∞ as l → ∞ .
Therefore, we have J ( K l ) → + ∞ as K l → K ∈ ∂ K .
This completes the proof.
B.2 Proof for Lemma
We first show the compactness of K γ .
Since J is continuous, we know K γ =
{ K ∈ K : J ( K ) ≤ γ } is a closed set.
It remains to show K γ is bounded.
Suppose there exist γ > 0 such that K γ is unbounded.
Then 6 See [6, Section 2.7.3] for the continuous-time counterpart.
there exists a sequence { K l } ∞ l =1 ⊂ K γ such that ∥ K l ∥ 2 → + ∞ as l → ∞ .
But by coerciveness of J ( K ) , we must have J ( K l ) → + ∞ as well, which contradicts that J ( K l ) ≤ γ for all l .
Hence K γ is bounded.
Therefore, K γ is compact.
The path-connectedness of the strict sublevel sets for the continuous-time H ∞ control problem has been proved in [29].
We can slightly modify the proof in [29] to show that the strict sublevel set { K ∈ K : J ( K ) < γ } is path-connected.
Based on the fact that every non-strict sublevel sets are compact, we can apply [41, Theorem 5.2] to show K γ is also path-connected.
We can also prove the path-connectedness of K γ by using the non-strict version of the bounded real lemma reviewed in Section A.5.
This proof is more self-contained, and hence also included here.
Since Q and R are positive definite, the non-strict version of the bounded real lemma [6, Section 2.7.3] states that J ( K ) ≤ γ if and only if there exists a positive definite matrix P such that the following non-strict matrix inequality holds
[Equation]
The above matrix inequality is linear in P but not linear in K .
A standard trick from the control theory can be combined with the Schur complement lemma to convert the above matrix inequality condition to another condition which is linear in all the decision variable [6].
Specifically, there exists a matrix function LMI( Y, L, γ ) which is linear in ( Y, L, γ ) such that Y ≻ 0 and LMI( Y, L, γ ) ⪯ 0 if and only if (B.1) is feasible with K = LY − 1 , and P = γY − 1 .
For completeness, we provide the detailed derivation of LMI( Y, L, γ ) as follows.
Step 1 : Let ˜ P = 1 γ P , dividing both sides of (B.1) by γ .
Then by Schur complement lemma, (B.1) can be rewritten as:
[Equation]
To see this, noticing that − γQ − 1 is negative definite and (B.1) can be rewritten as:
[Equation]
Step 2 : Now we can apply Schur complement lemma to (B.2) again to show its equivalence to:
[Equation]
To see this, noticing that − ˜ P is negative definite and the LHS of (B.2) can be rewritten as:
[Equation]
Step 3 : Then we left and right multiply diag( (cid:2)
˜ P −
I
I ˜ P − 1 (cid:3) ) on both sides of (B.4), this will not change the definiteness of (B.4) and leads the LHS of (B.4) to:   ˜ P
[Equation] [Equation]
Step 4 : Substituting Y =
˜
P − 1 and KY =
L into the above matrix leads (B.4) to:
[Equation]
Step 5 :
Furthermore, (B.8) is equivalent to:
[Equation]

[Equation]
(B.9)
Applying Schur complement lemma to (B.9) again leads to:
[Equation]
Y
[Equation]
L
[Equation]
γ ) :
[Equation]
This proves the equivalence between (B.10) and (B.1).
Therefore, we have: {
[Equation]
∈ K
[Equation]
( K
[Equation]
γ
[Equation]
⇐⇒ { K : (B.1) is feasible , P ≻ 0 } ⇐⇒ { K = LY − 1 : (B.10) is feasible , Y ≻ 0 } .
(B.11)
Noticing that the set of ( Y, L ) satisfying (B.10) and Y ≻ 0 is convex and hence path-connected.
In addition, the map K = LY − 1 is continuous for positive definite Y .
We can conclude that K γ =
{ K ∈ K : J ( K ) ≤ γ } is path-connected.
Such a proof is actually quite similar to the proof presented in [29].
The main difference is that the assumptions in this paper allow us to directly use the non-strict version of the bounded real lemma.
B.3 Proof for Lemma
The proof of this lemma depends on the above convex parameterization ( Y, L ) .
Since LMI( Y, L, γ ) is linear in ( Y, L, γ ) , the condition (B.10) is convex.
Suppose we have K = LY − 1 where ( Y, L, J ( K )) is a feasible point for the convex regime LMI( Y, L, J ( K )) ⪯ 0 and Y ≻ 0 .
In addition, by the non-strict version of the bounded real lemma, we know there exists a pair
( Y ∗ , L ∗ ) such that K ∗ =
L ∗ ( Y ∗ ) − 1 , LMI( Y ∗ , L ∗ , J ( K ∗ )) ⪯ 0 and
Y ∗ ≻ 0 .
By convexity of the LMI condition LMI( Y, L, γ ) ⪯ 0 and Y ≻ 0 , we know that the line segment between ( Y, L, J ( K )) and ( Y ∗ , L ∗ , J ( K ∗ )) is also in this convex set.
Therefore, for any 0 ≤ t ≤ 1 , we know the point ( Y + t ∆ Y, L + t ∆ L, J ( K ) + t ( J ( K ∗ )
[Equation]
J ( K ))) also satisfies
[Equation]
where ∆ L = L ∗ − L , and ∆ Y =
Y ∗ − Y .
Since Y ≻ 0 and Y ∗ ≻ 0 , we automatically have Y + t ∆ Y ≻ 0 .
Based on (B.11), we can construct a policy ( L + t ∆ L )( Y + t ∆ Y ) − 1 and the resultant closed-loop H ∞ norm must be smaller than or equal to J ( K ) +
t ( J ( K ∗ ) − J ( K )) .
Formally, we have
[Equation]
∗
[Equation]
Based on the fact J ( K ∗ ) <
J ( K ) , we can use the above inequality to construct a direction d such that J ′ ( K, d ) < 0 .
Specifically, let’s choose d = ∆ LY − 1 − LY − 1 ∆ Y Y − 1 .
Then we have
[Equation]
where the step ( a ) relies on the fact that ( Y + t ∆ Y ) −
= Y − 1 − tY − 1 ∆ Y Y − 1 + O ( t 2 ) and the step ( b ) uses the fact that J ( · ) is locally Lipschitz (Proposition 2).
Finally, the last inequality holds since we know J ( K ) >
J ( K ∗ ) .
Notice d ̸ = 0 .
Otherwise the above argument still works and we have J ′ ( K, 0) < 0 .
This is impossible since we know J ′ ( K, 0)
= 0 .
This leads to a contradiction, and we must have d ̸ = 0 .
This completes the proof for this lemma.
B.4 Proof for Theorem
We first show that K n ∈ K J ( K 0 ) for all n by induction.
By choice, we have δ n = c ∆ 0 n +1 ≤ c ∆ 0 for all n with some c ∈ (0 , 1) .
For n = 1 , we have K 1 =
K 0 − c ∆ 0 F 0 / ∥ F 0 ∥ 2 .
Since the norm of F 0 / ∥ F 0 ∥is equal to one and δ
= c ∆ 0 > 0 , we have K 1 ∈ B δ
( K 0 ) , where B δ 0 ( K 0 ) is the δ 0 -ball centered at
K 0 .
Based on the definition of ∆ 0 , we know B δ 0 ( K 0 ) ⊆
K , and hence K 1 ∈ K .
In addition, (13) implies J ( K 1 ) ≤ J ( K 0 ) − δ
∥ F 0 ∥ 2 .
Hence we have K J ( K 1 ) ⊆
K J ( K 0 ) , which implies K 1 ∈ K J ( K 0 ) .
Similarly, we can repeat this argument to show K n ∈ K J ( K 0 ) for all n .
Next, we can apply (11) to every step and then sum the inequalities over all n .
Then the following inequality holds for all N :
[Equation]
where the second inequality holds since J ( K N +1 ) ≥ J ∗ .
Since we have (cid:80) ∞ n =0 δ n = c ∆ 0 (cid:80) ∞ n =1 1 n = + ∞ , we know lim inf n →∞ ∥
F n ∥ 2 = 0 .
Therefore, there exists one subsequence { n i } such that ∥ F n i ∥ 2 → 0 .
For this subsequence, the policy parameter sequence { K n i } stays in the compact set K J ( K 0 ) .
Hence we know that the resultant policy sequence { K n i } is also bounded and has a convergent subsequence { K n i ( l ) } , which converges to some limit point K ∞ ∈ K J ( K 0 ) .
In addition, we have ∥ F n i ( l ) ∥ 2 → 0 as well.
When δ > 0 is sufficiently small, we know B δ ( K ∞ )
⊂ K .
Then there exists N δ such that B δ n i ( l ) ( K n i ( l ) )
⊂ B δ ( K ∞ ) for all n i ( l ) ≥ N δ .
Hence we have F n i ( l ) ∈ ∂
δ ni ( l ) J ( K n i ( l ) )
⊂
∂ δ J ( K ∞ ) for all n i ( l ) ≥ N δ .
Noticing that F n i ( l ) → 0 and ∂ · J ( · ) is closed, we have 0 ∈ ∂
δ J ( K ∞ ) for any δ > 0 .
It is also well-know that one has (cid:84) n i ( l ) ∂ δ ni ( l ) J ( K ∞ ) =
∂ C J
K ∞ ) (see Remark 2 for extra explanations).
Hence we have 0 ∈ ∂ C ( K ∞ ) , and K ∞ has to be a Clarke stationary point.
Based on Theorem 1, we have J ( K ∞ ) =
J ∗ , and hence the function-value subsequence { J ( K n i ( l ) ) } converges to J ∗ .
Notice that { J ( K n ) } is monotonically decreasing for the entire sequence { n } .
Hence the sequence { J ( K n ) } has a limit, and this limit has to be J ( K ∞ )
=
J ∗ .
This completes the proof.
Remark 2.
For completeness, we also explain the well-known fact regarding (cid:84) n ∂ δ n J ( K ) =
∂ C J ( K ) as δ n
→ 0 .
Here { δ n } is allowed to be any monotonically-decreasing sequence satisfying ∂
δ
J ( K ) ⊂
K .
By definition, we have ∂ C J ( K ) ⊂
∂
δ n J ( K ) for all n .
Since { δ n } is monotonically decreasing, we also have ∂ δ n +1 J ( K ) ⊂
∂ δ n J ( K ) for all n .
Therefore, we have
[Equation]
To show (cid:84) n ∂ δ n J ( K ) ⊂ ∂ C J ( K ) , we can use the following contradiction argument, which is standard (e.g. see [26, Remark 3.7]).
Let us assume that there exists V ∈ (cid:84) n ∂ δ n J ( K ) \ ∂ C J ( K ) .
Denote S n := {∪
K ′ ∈
B δn ( K ) ∂
C J ( K ′ ) } .
Obviously, S n depends on K and δ n .
In [24], S n has been shown to be compact and nested.
By [24, Lemma 2.6], we have
V
[Equation]
Therefore, we can express V as V = (cid:80) j t j V j with V j ∈ (cid:84) n S n , t j ≥ 0 , and (cid:80)
j t j = 1 .
Notice V j ∈ S n for all n .
Based on the definition of Clarke subdifferential, we know that for each n , there exists a sequence of differentiable points { K n,r j } such that K n,r j → K n j , ∇ J ( K n,r j ) → V j as r → ∞ , and ∥ K n j − K
∥
≤ δ n .
Then there exists a large enough r ( n, j ) such that ∥ V j − ∇ J ( K n,r ( n,j ) j ) ∥ ≤ δ n , and ∥ K n,r ( n,j ) j − K ∥ ≤ 2 δ n .
Since δ n → 0 as n → ∞ , we have found a sequence { K n,r ( n,j ) j } ∞ n =1 , such that K n,r ( n,j ) j → K and ∇ J ( K n,r ( n,j ) j ) → V j as n → ∞ .
Therefore, we have V j ∈ ∂ C J
( K ) .
By convexity of ∂ C J ( K ) [24,
Proposition 2.3], we know V = (cid:80) j t j V j ∈ ∂
C J
( K ) .
This contradicts the assumption that V is not in ∂ C J ( K ) .
Therefore, we must have (cid:84) n ∂ δ n J ( K ) ⊂
∂
C J ( K ) .
Consequently, we know (cid:84) n ∂ δ n J ( K ) =
∂ C J ( K ) .
C Discussions on Implementable Variants and Related Convergence Results
In this section, we give more detailed discussions on implementable variants of Goldstein’s subgradient method and related convergence guarantees.
Formal treatments to the informal theorems presented in Section 4.2 are also presented.
C.1 Gradient Sampling
The gradient sampling (GS) method can be viewed as an approximated version of Goldstein’s subgradient method.
The idea of GS has been explained in Section 4.2.
In the unconstrained optimization setting, it has been
Require : K 0 ∈ K at which J is continuously differentiable, initial sampling radius δ 0 ∈ (0 , ∆ 0 2 ) , initial stationarity target ϵ 0 ∈ [0 , ∞ ) , reduction factors µ δ , µ ϵ ∈ (0 , 1] , sample size m ≥ n x n u + 1 , line search parameters ( β, θ ) ∈ (0 , 1) × (0 , 1) for n = 0 , 1 , 2 , · · · do
Independently sample { K n, 1 , · · · K n,m } uniformly from B δ n ( K n )
Compute F n as the solution of the following convex quadratic program:
[Equation]
subject to
[Equation]
} if ∥ F n ∥ 2 ≤ ϵ n , set δ n +1 ← µ δ δ n , ϵ n +1 ← µ ϵ ϵ n , and t n ←else set δ n +1 ← δ n , ϵ n +1 ← ϵ n , define:
[Equation]
and compute t n such that
[Equation]
if J is continuously differentiable at K n − t n ˆ F n then set K n +1 ← K n − t n ˆ F n else set K n +1 randomly as any point where J is continuously differentiable such that
[Equation]
end for shown
[Table]
GS
[Table]
probability
[Table] [Table]
[Table]
For our problem, we need to ensure that the iterates do not travel outside the feasible set K , and hence we use the trust-region version of GS, which was originally developed in [36, Section 4.2].
For clarity, the trust-region version of GS in [36, Section 4.2] is restated as Algorithm 1.
For our purpose, we need to ensure δ 0 < ∆ 0 2 .
See [36, Section 4.2] for more discussions on the convergence theory for Algorithm 1.
As explained in [36, Section 4.2], Theorem 3.3 in [36] also holds for the trust-region version of GS, and hence we know that every cluster point of Algorithm 1 is Clarke stationary (with probability one).
From the discussion in [36, Section 4.2], we also know that the trust-region version of GS can guarantee ∥ K n +1 − K n ∥ ≤ 2 t n δ n ≤ 2 δ 0 < ∆ 0 for any
K n ∈ K .
Recall that ∆ 0 is the distance between K J ( K 0 ) and K c .
Therefore, by induction, we can show that the iterations generated by Algorithm 1 will be guaranteed to stay in the sublevel set K J ( K 0 ) with probability one.
Then it is straightforward to combine the above facts and Theorem 1 to show the global convergence of Algorithm 1 for the H ∞ state-feedback synthesis problem.
We formalize Theorem 4 as the following global convergence result.
Theorem C.1.
Consider the policy optimization problem (6) with the H ∞ cost function defined in (5) .
Suppose Assumption 1 holds, and K 0 ∈ K .
Let { K n } be a sequence generated by Algorithm 1 with µ δ , µ ϵ < 1 .
With probability one, we have K n ∈ K for all n , and the algorithm does not stop such that δ n ↓ 0 and ϵ n ↓ 0 .
In addition, the function-value sequence { J ( K n ) } is monotonically decreasing almost surely, and we have
J ( K n ) → J ∗ as n → ∞ with probability one.
Proof.
As commented previously, we can use an induction argument to show that the iterates { K n } generated by Algorithm 1 stay inside the feasible set K almost surely for all n .
Now we present some details for this argument.
For n = 0 , we know B δ
( K 0 )
⊂ K , and hence { K 0 , 1 , · · · , K 0 ,m } ⊂ K .
With probability one,
J is differentiable on K 0 ,j for all j ∈ { 1 , 2 , · · · , m } .
Therefore, F 0 is well defined with probability one.
There are two possibilities.
If ∥ F 0 ∥ 2 ≤ ϵ 0 , then we set K 1 = K 0 and shrink ( ϵ 0 , δ 0 ) .
Obviously, we still have K 1 ∈ K J ( K 0 ) .
If ∥ F 0 ∥ 2 > ϵ 0 , the algorithm is guaranteed to find a good descent condition satisfying (C.2).
Notice that we have K 0 − t
ˆ F 0 ∈ K for any t ∈ { 1 , θ, θ 2 , · · · } .
If J is continuously differentiable at K 0 − t 0 ˆ F 0 , then we have K 1 = K 0 − t
ˆ F 0 ∈ B
δ 0 ( K 0 ) ⊆
K as ∥ ˆ F 0 ∥ 2 = δ 0 and t 0 ≤ 1 .
If J is not continuously differentiable at K 0 − t 0 ˆ F 0 , then K 1 is randomly generated in a way that (C.3) and (C.4) hold.
From (C.4), we can combine δ 0 ∈ (0 , ∆ 0 / 2) and t 0 ≤ 1 to show ∥ K 1 − K 0 ∥ ≤ 2 t 0 δ 0 < ∆ 0 .
Then we have K 1 ∈ K as well.
Since the line search guarantees J ( K 1 ) ≤ J ( K 0 ) , we further have K 1 ∈ K J ( K 0 ) .
To summarize, no matter whether ∥ F 0 ∥ 2 is larger than ϵ 0 or not, we always have K 1 ∈ K J ( K 0 )
⊂ K .
Then we can repeat this argument to show K n ∈ K
J ( K 0 ) ⊂
K for all n (with probability one).
From [36, Section 4.2] (and Theorem 1), every cluster point of { K n } has to be a Clark stationary point (and hence a global optimal point) of J in the almost sure sense 7 .
From Lemma 2, K J ( K 0 ) is compact.
Then with probability one, { K n } stays in the compact set K J ( K 0 ) for all n , and there must exist a subsequence of { K n } which converge to one of its cluster points.
Therefore, this subsequence must converge to the global optimal point of J , and the function-value sequence associated with this policy subsequence has to converge to J ∗ almost surely.
Notice that { J ( K n ) } is monotonically decreasing for the entire sequence { n } in an almost sure sense.
Therefore, we have J ( K n ) → J ∗ almost surely.
This completes the proof.
The implementation of (C.3) and (C.4) can be done via performing the sampling strategy described in the end of [36, Section 2].
In addition, ∆ 0 is typically unknown in practice, and hence one can just tune the parameter δuntil it is sufficiently small.
C.2 Non-derivative Sampling
In [37], it has been shown that the gradient oracle in the gradient sampling method can be replaced with the function value oracle and similar convergence guarantees can still be obtained.
This leads to the non-derivative sampling (NS) method.
To avoid the gradient computation, one can use the Gupal estimation to approximate the derivatives.
For completeness, we briefly review such approximation below, more details can be found in [37].
For α > 0 , let n = n x n u , the Steklov averaged function J α is defined as follows:
[Equation]
where ψ α : R n → R + is the Steklov mollifier defined by
[Equation]
Then the partial derivatives of J α are given by
[Equation]
for i = 1 , · · · , n , where K i refers to the i -th element of vec( K ) , B ∞ :=
[ − 1 / 2 , 1 / 2] n is the unit cube centered at 0 and
[Equation]
Therefore, for the unit cube Z := (cid:81) ni
=1 B ∞ in R n × n , we can use the Gupal estimate of vec( ∇ J α ( K )) : χ ( K, α, z ) := ( χ 1 ( K, α,
[Equation]
1 ) , · · · , χ n ( K, α, ζ n )) for z := ( ζ 1 , · · · , ζ n ) ∈ Z.
It is worth to mention that such approximation needs 2 n evaluations of J compared to the n evaluations for gradient sampling Algorithm 1.
The pseudo code for the NS method is presented in Algorithm 2 [37].
Similar to the GS algorithm, the diminishing δ n guarantees the iterations of NS Algorithm stay inside the feasible set K .
Furthermore, [37, Theorem 3.8] shows that every cluster point of { K n } is a Clark stationary for J ( K ) .
Therefore, we have the global convergence based on Theorem 1.
Theorem C.2.
Consider the policy optimization problem (6) with the H ∞ cost function defined in (5) .
Suppose Assumption 1 holds, and K 0 ∈ K .
Let { K n } be a sequence generated by Algorithm 2 with µ δ , µ ϵ < 1 .
With probability one, we have K n ∈ K , and the algorithm does not stop such that δ n ↓ 0 and ϵ n ↓ 0 .
In addition, the function-value sequence { J ( K n ) } is monotonically decreasing almost surely, and we have J ( K n ) → J ∗ as n → ∞ with probability one.
Proof.
We can use an induction argument to show that the iterates { K n } generated by Algorithm 2 stay inside the feasible set K almost surely for all n .
For n = 0 , we {+know B δ 
+}
( K 0 )
⊂ K , and hence { K 0 , 1 , · · · , K 0 ,m } ⊂ K .
Since { ζ 0 , 1 , · · · , ζ 0 ,m } are sampled from Z , F 0 is well defined. If ∥ F 0 ∥ 2 ≤ ϵ 0 , then we set K 1 = Kand shrink ( ϵ 0 , δ 0 ) .
Obviously, we have K 1 ∈ K J ( K 0 ) .
If ∥ F 0 ∥ 2 > ϵ 0 , we need to compute the step
Algorithm 2:
Require : K 0 ∈ K , initial sampling radius δ 0 ∈ (0 , ∆ 0 ) , a stationary target ϵ 0 ∈ [0 , ∞ ) , reduction factors µ δ , µ ϵ ∈ (0 , 1] , sample size m ≥ n x n u + 1 , line search parameters ( β, θ, t, κ ) ∈ (0 , 1) 2 .
A sequence of mollifier parameters α n ↓ 0 in (0 , 1] .
for n ∈ N ∪ { 0 } do
Independently sample { K n, 1 , · · · K n,m } uniformly from B δ n ( K n )
Independently sample { ζ n, 1 , · · · ζ n,m } uniformly from Z
Compute F n as the solution of min2 ∥
F ∥subject to F ∈ F n = conv { χ ( K n, 1 , α n , ζ n, 1 ) , · · · , χ ( K n,m , α n , ζ n,m ) } if ∥ F n ∥ ≤ ϵ n , set ϵ n +1 ← µ ϵ ϵ n , δ n +1 ← µ δ δ n , and t n ← 0 .
else set δ n +1 ← δ n , ϵ n +1 ← ϵ n , ˆ F n = F n / ∥ F n ∥ 2 , compute t n as follows:
Update K n +1 =
K n − t n ˆ F n end for size t 0 as described in Algorithm 2.
There are two possibilities: if t 0 = 0 , then we still have K 1 =
Kand hence K 1 ∈ K J ( K 0 ) .
If t 0 ̸ = 0 , then the algorithm has found a good descent condition such that
J ( K 0 − t 0 ˆ F {+0 ) ≤ J ( K 0 ) − βt 
 ∥ F 0 ∥ holds. In this case, we set K 1 = K 0 − t 
 ˆ F 0 .+}
Notice that we must have
K 0 − t 0 ˆ F 0 ∈ K as t 0 ≤ δ 0 and ∥ ˆ F 0 ∥ 2 = 1 .
Since the descent condition guarantees J ( K 1 ) ≤ J ( K 0 ) , we further have K 1 ∈ K J ( K 0 ) .
To summarize, no matter whether ∥ F 0 ∥ 2 is larger than ϵ 0 or not, we always have
K 1 ∈ K J ( K 0 )
⊂ K .
Then we can repeat this argument to show K n ∈ K J ( K 0 )
⊂ K for all n (with probability one).
From [37, Theorem 3.8] (and Theorem 1), every cluster point of { K n } has to be a Clark stationary point (and hence a global optimal point) of J in the almost sure sense.
From Lemma 2, K J ( K 0 ) is compact.
Then with probability one, { K n } stays in the compact set K J ( K 0 ) for all n , and there must exist a subsequence of { K n } which converge to one of its cluster points.
Therefore, this subsequence must converge to the global optimal point of J , and the function-value sequence associated with this policy subsequence has to converge to J ∗ almost surely.
Notice that { J ( K n ) } is monotonically decreasing for the entire sequence { n } in an almost sure sense.
Therefore, we have J ( K n ) → J ∗ almost surely.
This completes the proof.
C.3 Model-Free NS
When the model is unknown, there are various methods available for estimating the H ∞ -norm from data [44, 45, 55, 52, 67, 49, 65, 64].
The model-free NS method just replaces the exact oracle for the H ∞ cost (5) with some noisy estimates.
Based on our experience, the multi-input multi-output (MIMO) power iteration method [48] works quite well as a stochastic zeroth-order oracle for the purpose of implementing NS in the model-free setting.
The main idea of the MIMO power iteration method is that the H ∞ norm of an LTI MIMO system can be approximated from its finite-time approximated representation.
Given a black-box simulator for a stable system G K , the power iteration method provides an efficient oracle for estimating the H ∞ of G K from trajectories of G
K .
The trajectory length is denoted as N , which quantifies the sample size.
The larger N is, the better the H ∞ -norm estimation is.
We refer the readers to [48] for implementation details of the power iteration method.
The sample complexity for model-free NS is unknown.
However, our numerical results show that such such a model-free method tracks the convergence of its model-based counterpart given reasonable amount of data.
C.4 Interpolated Normalized Gradient Descent (INGD) with Finite-Time Complexity
The INGD method was originally proposed in [69], and can be viewed as an implementable approximation of Goldstein’s subgradient method.
The main advantage of INGD is that it yields finite-time iteration complexity for finding ( δ, ϵ ) -stationary points.
The original method in [69] works for any directionally differentiable functions.
Specially, at an iterate K n , [69] proposes Algorithm 3 to compute the update direction F n .
In the unconstrained setup, it has been shown that Algorithm 3 terminates with high probability [69, Lemma 13].
In addition, one can combine Algorithm 3 and (13) with a fixed δ yields the INGD method (See Algorithm 4).
A non-asymptotic analysis for Algorithm 4 has been provided.
In particular, it has been show that Algorithm 4 terminates with high probability as well [69, Theorem 8].
Algorithm 3: MiniNorm
Input : K , δ > 0 , and ϵ >
Let n = 0 , call the generalized gradient oracle F
=
O ( K, 0) while ∥ F n ∥ > ϵ and δ 4 ∥ F n ∥ ≥ J ( K ) − J ( K − δ F n ∥ F n ∥ ) do
Sample Y n uniformly from [ K n , K − δ F n ∥ F n ∥ ]
Call the generalized gradient oracle H n =
O ( Y n , − F n )
Update F n = β n F n + (1 − β n ) H n with β n = 4 L 2 −∥ F n ∥ 2 4 L 2 +2 ∥
F n ∥n ← n +end while
Return F n
Algorithm 4:
Interpolated Normalized Gradient Descent (
Initial : K 0 , L
Input : δ , ϵ for l = 0 , · · · , L do
F l = MiniNorm1 ( K l , δ, ϵ )
K l +1 =
K l − δF l / ∥
F l ∥end
Return K L
However, Algorithm 3 (the original algorithm in [69]) requires a special generalized gradient oracle O defined in [69, Assumption 1].
To relax this requirement, a variant of Algorithm 3 has been proposed in [13].
The version of INGD in [13] relies on a variant of Algorithm 3 which only requires standard gradient oracle for any differentiable points.
This variant of the MiniNorm1 sub-routine is documented in Algorithm 5.
Therefore, the generalized gradient oracle O is no longer required. Algorithm Input : K , δ > 0 , and ϵ > Let n = 0 , F 0 = ∇ J ( Y 0 )
where Y 0 ∼ B δ
( K )
while ∥ F n ∥ 2 > ϵ and δ 4 ∥ F n ∥ 2 ≥ J ( K ) − J ( K − δF n / ∥ F n ∥ 2 ) do
Choose any r satisfying 0 < r <
∥ F n ∥ 2 (cid:113)
[Equation]
(1 −
[Equation]
F n
[Equation]
22 128 L 2 )
Sample H n uniformly from B r ( F n )
Sample Y n uniformly from [ K, K − δ H n ∥ H n ∥ 2 ]
[Equation]
end while
Return F n
Similar to the Algorithm 3, MiniNorm2 also terminates with high probability.
Now we discuss how to modify the analysis for our H ∞ control problem.
It has been show in [69, Theorem 8] that for unconstrained nonsmooth optimization of L -Lipschitz functions, the above INGD algorithm can be guaranteed to find the ( δ, ϵ ) -stationary point with the high-probability iteration complexity O (cid:16)
∆ L 2 ϵ 3 δ log( 1 p ) (cid:17) , where ∆ is the initial function value gap, and p is the failure probability (i.e. the optimization succeeds with the probability (1 − p ) ).
We can choose δ < ∆ 0 to show that the resultant algorithm can find the ( δ, ϵ ) -stationary point for H ∞ state-feedback control problem with the high-probability iteration complexity O (cid:16)
∆ L 2 ϵ 3 δ log( 1 p ) (cid:17) .
Now we formalize Theorem 5 as follows.
Theorem C.3.
Consider the H ∞ state-feedback control problem (6) with the objective function J ( K ) defined in (5) .
Suppose K 0 ∈ K and denote ∆ 0 := dist( K J ( K 0 ) , K c ) .
Let ∆ =
J ( K 0 ) − J ∗ .
Denote the Lipschitz [Table] [Table] constant of J ( K ) over the sublevel set K J ( K 0 ) as L 0 .
For any δ < ∆ 0 , the iterations generated by the INGD algorithm will stay in K almost surely.
In addition, the INGD method finds a ( δ, ϵ ) -stationary point with a high-probability iteration complexity O (cid:16)
∆ L 20 ϵ 3 δ log( 4∆ pδϵ ) (cid:17) .
Proof.
We first show that the iterates of INGD stay inside the feasible set K when δ < ∆ 0 .
At step n , the Algorithm 3 returns an F n either ∥ F n ∥ ≤ ϵ or J ( K n − δF n / ∥ F n ∥ 2 ) ≤ J ( K n ) − δ
∥ F n ∥ 2 .
If ∥ F n ∥ ≤ ϵ , the
INGD algorithm will terminate as we have found a ( δ, ϵ ) -stationary point.
For the latter case, we update K n + as K n +1 =
K n − δF n / ∥ F n ∥ 2 .
Therefore, we only update K when a descent condition is satisfied.
Since
K 0 ∈ K and δ < ∆ 0 , we have K 1 =
K 0 − δF 0 / ∥ F 0 ∥ 2 ∈ K .
In addition, since J ( K 1 ) ≤ J ( K 0 ) − δ 4 ∥ F 0 ∥ 2 , we further have K 1 ∈ K J ( K 0 ) as K J ( K 1 ) ⊆
K J ( K 0 ) .
Repeating such argument shows that K n ∈ K J ( K 0 ) and hence K n ∈ K for all n .
The high probability complexity bound is a direct result of [69, Theorem 8].
We omit the details here.
D Numerical experiments
In this section, we provide simulation results to show the effeteness of the aforementioned implantable algorithms.
Computations are executed on a desktop computer with a 3.3 GHz Intel Xeon W-1350 processor with MATLAB
R2021b.
We implemented the Interpolated Normalized Gradient Descent (INGD) algorithm with with decreasing δ , Gradient Sampling (GS), and Nonderivative Sampling (NS).
In addition, model-free NS has also been implemented.
Subsequently, we provide the simulation result for one-time INGD method, which aims to illustrate that ( δ, ϵ ) -stationarity does not imply being δ -close to an ϵ -stationary point of J .
Finally, we implement the NS and GS methods in higher dimensional cases.
D.1 More details on the simulation results in Figure
Implementation of INGD, GS, NS, and model-free NS methods: For the problem matrices given in (16), we have J ∗ = 7 . 3475 .
In order to obtain the Clark stationary point by using INGD method.
We need to perform the repeated INGD algorithm with δ decreases to 0.
To this end, we set ϵ = 1 × 10 − 5 and δ n +1 = 0 .
7 δ n with δ 0 = 0 .
01 for n = 0 , 1 , · · · .
In other words, whenever the INGD method successfully finds a ( δ n , ϵ ) -stationary point, we reduce δ n by a constant factor (i.e., δ n +1 = 0 .
7 δ n ), then the INGD method tries to find a ( δ n +1 , ϵ ) - stationary point.
As δ n tends to 0, one should expect that the INGD algorithm returns a Clark stationary point.
In addition, we also implement the Gradient Sampling (GS) and Non-derivative sampling (NS) methods, where the algorithm parameters are set as in Table 1.
Finally, for the model-free NS method for the given problem matrices (16), we choose the sample size in the H ∞ -norm estimation as N = 100 .
Implementation of randomly generated cases : The middle plot of Figure 1 shows the performance of NS algorithm on the randomly generated cases.
In particular, we set A ∈ R 3 × 3 to be I + ξ , where each element of ξ ∈ R 3 × 3 is sampled uniformly from [0 , 1] ; B ∈ R 3 × 1 with each element uniformly sampled from [0 , 1] ;
Q = I + ζI ∈ R 3 × 3 with ζ uniformly sampled from [0 , 0 . 1] ; R ∈ R uniformly sampled from [1 , 1 . 5] .
For each experiment, the initial condition K 0 ∈ R 1 × 3 is also randomly sampled such that ρ ( A − BK 0 ) < 1 .
Implementation of Model-free NS method with decreasing sample size N : The right plot of Figure 1 provides the simulation result of model-free NS method with decreasing sample size N used in the H ∞ -norm estimation.
The hyperparameters are the same as the ones used in NS.
The only difference is that only we need to specify a sample size in the MIMO power iteration method for evaluating J ( K ) from sampled trajectories.
Specifically, we choose N = [100 50 20 10] .
As we can see from the right plot in Figure 1, as we decrease the sample size, the H ∞ -norm oracle become more noisy, and the algorithm performance has been degraded with more oscillations.
Two gradient oracles for differentiable points.
To implement INGD and GS methods, we need to compute the gradients of J ( K ) at differentiable points.
We provide two different oracles for computing the gradients of
J ( K ) at differentiable points.
To this end, let us rewrite J ( K ) as follows:
[Equation]
D.
[Equation]
[Figure]
where H ( K, ω ) =
( Q + K T RK ) 12 ( e jω
I − A + BK ) − 1 and σ max denotes the maximum singular value of a given matrix.
It is obvious that (D.1) is equivalent to the cost function as in (5).
It is well known that J ( K ) is differentiable at K if the H ∞ norm of H ( K, ω ) is achieved at one frequency ω 0 and the the largest singular value of H ( K, ω 0 ) has multiplicity one [61].
For the first oracle O 1 ( K ) , given a differentiable point K ∈ R n u × n x , O 1 ( K ) returns a finite time difference estimation of the gradient J ( K ) at K defined as follows:
[Equation]
where E ij has the same dimension as K whose ( i, j ) -th entry is 1 and other entries are 0, and h is a small positive scalar ( h = 0 .
0001 for example).
While the second oracle O 2 ( K ) returns ∇ J ( K ) via explicit analytic formula.
For completeness, we restate the explicit formula of the gradients of J ( K ) at differentiable points.
Let H ( K, ω 0 ) =
U Σ V H be a singular value decomposition, where U, V ∈ C n x × n x are complex unitary matrices consists of the left singular vectors and right singular vectors of H ( K, ω 0 ) , respectively.
Let u 1 be the first column of U and let v 1 be the first column of V .
In addition, denote H 1 ( K ) =
( Q + K T RK ) 12 ,
H 2 ( K ) =
( e jω
I − A + BK ) − 1 , and define I = (cid:82) ∞
e − τH 1 ( K ) H 2 ( K ) v 1 u ∗ 1 e − τH 1 ( K ) dτ .
Then the gradient of J ( K ) at differentiable point K can be calculated as:
[Equation]
The derivation of the above formula uses the chain rule of the total derivative and dJ k
=
Tr( ∇ J ( K ) dK T ) .
We omit the details here.
Figure 2 shows the simulation results for the problem matrices given in (16) by using the above two different gradient oracles for INGD and GS, respectively.
It can be seen that both oracles work well and the trajectories of two different oracles are consistent to each other.
D.2 Numerical experiment on INGD method with constants ( δ, ϵ )
The simulation results for INGD with Mininorm2 and constants ( δ, ϵ ) are shown in Figure 3, where we set δ = 0 .
01 and ϵ = 1 × 10 − 8 .
From the left figure of Figure 3, it can be seen that it takes 10 steps for INGD to find a ( δ, ϵ ) -stationary point.
At step 10, we have ∥ F ∥ 2 < ϵ .
However, J ( K ) does not converge to the optimal value as shown in the right plot of Figure 3.
This result confirms that ( δ, ϵ ) -stationarity does not imply being δ -close to an ϵ -stationary point of J as commented in [69, 13].
D.3 Numerical experiments on higher-dimension cases
In this section, we test the effeteness of NS and GS methods on higher-dimensional cases.
In particular, we first consider the following set of problem matrices:
[Equation]
[Table]
[Equation]
[Table]
[Figure]
[Figure]
For the above example, we have J ∗ = 43 .
and we select the initialization K 0 to be:
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure] [Equation] [Figure] [Equation]
[Figure]
[Equation]
[Figure]
[Equation]
which satisfies ρ ( A − BK 0 )
= 0 .
9567 < 1 .
In addition, we also perform the NS and GS algorithm on the randomly generated cases with higher dimensions.
We set A ∈ R 4 × 4 to be I + ξ , where each element of ξ ∈ R 4 × 4 is sampled uniformly from [0 , 1] ; B ∈ R 4 ×with each element uniformly sampled from [0 , 1] ; Q = I + ζI ∈ R 4 × 4 with ζ uniformly sampled from [0 , 0 . 1] ;
R =
I + υI ∈ R 2 × 2 with υ uniformly sampled from [0 , 0 . 5] .
For each experiment, the initial condition
K 0 ∈ R 2 × 4 is also chosen such that ρ ( A − BK 0 ) < 1 .
Figure 4 shows the simulation results on the higher dimensional cases
E Further Discussions E.1 Uniqueness of the Minimizing set
Our theory does not answer whether the global minimum for the H ∞ state-feedback synthesis problem is unique or not.
As commented in Section 3, the path-connectedness of K γ for every γ can be used to show that there exists a unique global minimizing set in a certain sense [41, Sections 2&3].
However, we are not able to rule out the possibility that the uniuqe global minimizing set actually consists of multiple points.
Whether the global minimum of the H ∞ state-feedback control problem is unique or not is an interesting open question.
E.2 Possible generalizations for nonlinear systems
For nonlinear systems, it is possible to generalize Theorems 3 and 5.
The proofs for the finite-time complexity of finding ( δ, ϵ ) -stationary
points can be generalized to the constrained policy optimization setting, as long as the cost function J is coercive over the nonconvex feasible set K .
In contrast, the convergence to global minimum may be too much to ask for general nonlinear robust control problems.
If the cost function J is not coercive, one may just add regularization to induce coerciveness such that one can still find some approximated ( δ, ϵ ) -stationary points provably.
Such developments are worth more investigations in the future.