Globally Convergent Policy Search for Output Estimation
Abstract
We introduce the ﬁrst direct policy search algorithm which provably converges to the globally optimal dynamic ﬁlter for the classical problem of predicting the outputs of a linear dynamical system, given noisy, partial observations.
Despite the ubiquity of partial observability in practice, theoretical guarantees for direct policy search algorithms, one of the backbones of modern reinforcement learning, have proven difﬁcult to achieve.
This is primarily due to the degeneracies which arise when optimizing over ﬁlters that maintain an internal state.
In this paper, we provide a new perspective on this challenging problem based on the notion of informativity , which intuitively requires that all components of a ﬁlter’s internal state are representative of the true state of the underlying dynamical system.
We show that informativity overcomes the aforementioned degeneracy.
Speciﬁcally, we propose a regularizer which explicitly enforces informativity, and establish that gradient descent on this regularized objective – combined with a “reconditioning step” – converges to the globally optimal cost at a O (1 /T ) rate.
Introduction
Data used for prediction and control of real world dynamical systems is almost always noisy and incomplete (partially observed).
Sensors and other measurement procedures inevitably introduce errors into the datasets, so designing reliable learning algorithms for these noisy or partially observed domains requires confronting fundamental questions of disturbance ﬁltering and state estimation.
Despite the ubiquity of partial observation in practice, these concerns are often underexplored in modern analyses of learning for control that assume perfect observations of the underlying dynamics.
In this work we study the output estimation ( OE ) problem or learning to predict in partially observed linear dynamical systems.
The output estimation problem is one of the most fundamental problems in theoretical statistics and learning theory.
Both in theory and in practice, advances in predicting partially observed linear systems have led to successes in a variety of areas from controls to biology and economics, (c.f. e.g. Athans [1974], Lillacci and Khammash [2010], Gautier and Poignet [2001]).
We revisit this classical problem from a modern optimization perspective, and study the possibility of learning the optimal predictor via model-free procedures and direct policy search.
Relative to model-based procedures, which ﬁrst estimate the underlying dynamics and then return a policy by solving an optimization problem using the estimated model, model-free methods offer several potential advantages.
For instance, direct policy search allows one to easily specify the complexity of the policy class over which one searches.
In addition, model-free policy search optimizes for performance directly on the true system, rather than an approximate model.
As such, there is no gap between the model used for synthesis and the system on which the policy is deployed.
In light of these advantages, there has recently been signiﬁcant interest from both theoreticians and practitioners in understanding the foundations of model-free decision making.
So far, this attention has focused mostly on problems with full-state observation such as the linear quadratic regulator ( LQR ) or fully-observed Markov Decision Processes (MDPs), for which optimal policies are static (without memory).
Progress on partially observed problems has been complicated by the difﬁculties associated with optimizing over dynamic policies (with memory) that maintain internal state to summarize past observations.
In this paper, we provide the ﬁrst policy search algorithm which provably converges to the globally optimal ﬁlter for the OE problem, and shed new light on the intricacies of the underlying optimization landscape.
The Output Estimation problem.
We study one of the simplest and most basic problems with partial observability: the output estimation ( OE ) problem.
In brief, the goal is to search for a predictor of the output z ( t ) of a linear dynamical system given partial measurements y ( t ) .
For the true system with states x ( t ) and dynamics that evolve according to,
[Equation]
the goal is to ﬁnd the parameters
K =
( A K , B K , C K ) of the ﬁlter (interchangably, policy ),
[Equation]
that minimizes the steady-state prediction error,
[Equation]
In this paper, we study solving the OE problem via model-free methods, where the goal is to search for the optimal ﬁlter parameters K
=
( A K , B K , C K ) using direct policy search without knowledge or estimation of the true system parameters A , C , G , W 1 , W 2 ; see Section 2 for further details.
The OE problem is not a control problem per se, as decisions ˆ z do not inﬂuence the state evolution of the true system (1.1).
Nonetheless, it is an attractive stepping stone for studying model-free control.
Like LQG , optimal policies for OE are dynamic , i.e. have memory.
OE therefore serves as a valuable testbed to develop the theoretical foundations of direct policy search for this important class of policies which has thus far remained poorly understood.
Related work
Recent years have witnessed a resurgence of interest in direct policy search for control and reinforcement learning.
Fazel et al.
[2018] established global convergence of policy gradient methods on discrete-time LQR .
Subsequent work has sharpened rates [Malik et al., 2019], analyzed convergence under more general frameworks [Bu et al., 2019], and extended the analysis to continuous-time [Mohammadi et al., 2021].
Beyond LQR , Zhang et al.
[2020] analyzed policy search for mixed H 2 / H 1 and risk-sensitive control.
Sun and Fazel [2021] also considered analysis via convex reformulations, for state feedback problems.
For discrete state-action MDP s, Agarwal et al.
[2021] established convergence rates for a variety of policy gradient methods, cf.
also Bhandari and Russo [2019].
All of these works considered static state-feedback policies, with perfect state information.
For problems with partial observation – which necessitates the use of dynamic policies - the work most relevant to our own is Tang et al.
[2021], which studied policy search for LQG .
They established that all stationary points corresponding to minimal controllers are globally optimal, and demonstrated that gradient descent may fail to converge to optimal policies.
1.1 Contributions
We propose a novel policy search method which provably converges to a globally optimal L OE cost.
Despite extensive prior work on static policy search (see above), our result constitutes the ﬁrst rigorous guarantee for policy search over dynamic policies.
While the OE problem admits a convex reformulation [Scherer et al., 1997], it is also well known that its landscape contains suboptimal critical points.
This may seem like a contradiction at ﬁrst, but we demonstrate that such spurious critical points occur precisely when the policy loses what we term “informativity”: that the steady covariance between the controller state and system state, ⌃ 12 := lim T !
1 1 T [
R
T 0 x ( t )ˆ x ( t ) >
d t ] , becomes rank-deﬁcient.
With this insight, we propose a regularizer R that ensures the internal state of the learned policy remains “uniformly informative” about the state of the true system, and prove that gradient descent on the regularized objective L OE ( · ) +  R ( · ) converges at a O (1 /T ) rate to an optimal policy.
The efﬁcacy of the approach is illustrated via numerical experiments, cf.
Appendix F.6.
Code is provided in the supplementary material.
Our techniques.
Searching over dynamic policies introduces two key challenges: (1) spurious critical points can arise when one or more factors become “degenerate” in a certain way; (2) changes of basis produce a continuum of “equivalent realizations” of the ﬁlter K , some of which are poorly conditioned.
Similar challenges have been observed in problems with rotational symmetries, e.g. nonconvex matrix factorization.
Neither challenge arises when searching over static policies [Fazel et al., 2018].
Our approach is centered around the idea of convex reformulations of control synthesis problems, and the following fact regarding functions which admit these reformulations, cf.
Appendix H.1 for proof.
Fact 1.1.
Let f : R n x !
R be a differentiable, possibly nonconvex function such that min x f ( x ) is ﬁnite.
Suppose there exists a differentiable function : R n ⌫ !
R n x satisfying the following two properties: (i) the mapping is surjective, i.e. for all x 2 R n x there exists ⌫ 2 R n ⌫ such that x = ( ⌫ ) , (ii) under the change of variables the function f cvx ( ⌫ ) :
= f
( ( ⌫ )) is differentiable and convex .
Then all ﬁrst-order stationary points, x s.t r f ( x ) = 0 , are globally optimal.
The OE problem, LQG, and many other related control tasks admit convex reformulations [Scherer et al., 1997, Masubuchi et al., 1998].
Given that gradient descent (under mild regularity assumptions) converges to stationary points, we might hope that Fact 1.1 guarantees that direct policy search on the OE ﬁlter will succeed at ﬁnding an optimal policy, when applied to loss functions admitting such convex reformulations.
Somewhat surprisingly, we ﬁnd that this is emphatically not the case: gradient descent on the L OE objective fails to reliably converge to optimal solutions (see Section 3.1).
To resolve this paradox, we show that the surjectivity condition of Fact 1.1 may fail for the convex reparametrization of OE : there are ﬁlters K with ﬁnite cost L OE ( K ) , which are not in the image of the reformulation map ( · ) .
We ﬁnd that degeneracy occurs precisely when informativity , deﬁned in Section 1.1 as ⌃ 12 , K having full rank, fails to hold.
Conversely, when ⌃ 12 , K is full-rank, the conditions of Fact 1.1 are met and the parametrization behaves as needed.
Thus, we identify noninformativity - rank deﬁciency of ⌃ 12 , K - as the fundamental notion of degeneracy corresponding to challenge (1).
Motivated by this observation, we introduce a novel “informativity regularizer”
R info ( · ) which enforces that ⌃ 12 , K is full rank.
Our proposed algorithm, IR-PG alternates between gradient updates on the regularized loss L  ( · ) :
= L OE ( · ) +  R info ( · ) , and “reconditioning” steps to ensure well-conditioned realizations of the ﬁlters K , thereby addressing challenge (2) above.
We stress that our notion of informativity differs from the minimality criterion emphasized in Tang et al. [2021], whose limitations we discuss in Section 3.1.
In order to achieve our quantitative converge guarantees, we establish numerous results which may be of independent interest, including: (i) a quantitative analysis of the OE convex reformulation due to Scherer [1995], (ii) novel bounds on the magnitude of solution to Lyapunov equations under the closed-loop OE ﬁlter dynamics.
Both arguments appeal to a (quantitative measure of) informativity.
Finally, we develop a quantitative analogue of Fact 1.1 via a paradigm we call differentiable convex liftings , which allows us to establish a form of gradient dominance that we term weak-PL in reference to the well-known Polyak- Ł ojasiewicz inequality; cf.
Section 4.1 for details.
Preliminaries
Before presenting our main results in Section 3, we ﬁrst introduce some relevant deﬁnitions, and provide the reader with some relevant background on prediction in partially-observed dynamical systems.
We adopt standard notation wherever possible, and for brevity, defer details to Appendix A.
As outlined in the introduction, we consider the problem of predicting the outputs of a partially observed linear dynamical system.
We refer to the dynamical system deﬁned in Eq.
(1.1) as the true system , with states x ( t ) 2 R n , observations y ( t ) 2 R m , and performance outputs z ( t ) 2 R p .
To ensure the dynamical system has a well-deﬁned steady-state, we assume that A is stable.
Assumption 2.1.
The matrix A is Hurwitz stable .
That is, the real components of all its eigenvalues are strictly negative: <
[  i ( A )] < 0 for i 2 [ n ] .
Because these policies only access the system outputs, and are only evaluated in relation to system outputs, we assume that the true system state is observable .
Further, we assume that dynamics are subject to sufﬁciently rich noise excitations.
Assumption 2.2.
The pair ( A , C ) in Eq.
(1.1) is observable.
That is, the observability Gramian deﬁned as
[Equation]
Assumption 2.3.
We assume that the noise matrices W 1 and W 2 are strictly positive deﬁnite.
As stated previously, we restrict our attention to ﬁnding the best dynamic ﬁlter within the parametric family described in Eq.
Note that this family contains the Bayes optimal predictor for the L OE objective, as we will later describe in more detail.
We review a number of basic facts:
A. Steady state distributions.
We deﬁne K stab :=
{ K : A K is Hurwitz-stable } to be the set of ﬁlters such that
A
K is stable.
Under Assumption 2.1, Appendix E.1 shows that this is equivalent to the stability of the closed-loop matrix A cl , K :
[Equation]
Stability of A cl , K is a sufﬁcient condition for L OE ( K ) to be ﬁnite, and for the following limiting covariance to be well deﬁned: ⌃ K
[Equation]
( t ) ˆ x K
[Equation]
t ) i h x ( t ) ˆ x K ( t ) i > 2 S 2 n + .
This steady-state covariance is given by the solution to the continuous-time Lyapunov equation,
[Equation]
where W cl , K :
= h
W0 0 B K W 2 B > K i . Notice that ⌃ K depends only on ( A K , B K ) , but not on C K , and that the ﬁrst n ⇥ n block of ⌃ K does not depend on K at all.
To highlight these distinctions, we partition
[Equation]
and deﬁne K ctrb := { K 2 K stab : ⌃ 22 , K  0 } as the set of ﬁlters whose internal state covariance is full rank.
We refer to these as the controllable policies, as these are precisely the policies for which the pair ( A K , B K ) is controllable.
B. Equivalent realizations.
There are many different ways of parametrizing a given dynamic feedback policy, all of which have exactly the same input-output behavior.
Let GL ( n ) denote the set of invertible n ⇥ n matrices. In particular, given an invertible matrix S 2 GL ( n ) the OE loss of a ﬁlter K is invariant under the following class of similarity transforms:
[Equation]
Formally, for any K 2 K stab and any S 2 GL ( n ) , L OE ( K ) =
L OE ( Sim S ( K )) .
We say that K and K 0 are equivalent realizations if they are related by a similarity transformation Sim S ( K )
=
K 0 for some S 2 GL ( n ) .
Note that the set K ctrb is also preserved under similarity transformation.
C. Optimal policies.
The landmark result by Kalman shows that for the system deﬁned by ( A , C , W 1 , W 2 ) the Kalman ﬁlter K ?
=
( A  L ?
C , L ? , G ) achieves minimal L OE loss.
Here,
L ? is the Kalman gain which is deﬁned in terms of the solution of the following Riccati equation:
[Equation]
We deﬁne the set of optimal ﬁlters K opt to be those which are equivalent to the Kalman ﬁlter:
[Equation]
D. Restricted problem setting.
The problem description outlined above, including Assumptions 2.1 to 2.3, constitutes the standard OE problem, well-known in control theory, cf.
[Doyle et al., 1989, §IV.D].
In this paper, we will make the following additional assumption that restricts the class of OE problems we consider.
Further discussion on the utility and necessity of this assumption (for our analysis) is provided in Section 3.2 and Appendix D; the latter also shows that
Assumption 2.4 holds for “generic” problem instances.
Assumption 2.4.
The (cannonical) optimal policy ( A  L ?
C , L ? ) is controllable, i.e.
R 1 0 exp( s ( A  L ? C )) L ? L > ?
exp( s ( A  L ? C )) >
d s is strictly positive deﬁnite.
E. Interaction protocol.
In the spirit of model-free methods, we introduce algorithms which work only assuming access to cost and gradient evaluation oracles.
We abstract away the particular implementation of these oracles to simplify our presentation and assume that they are exact, in order to focus on the overall optimization landscape of the OE problem.
More formally, for any ﬁlter
K 2 K stab , Eval ( K , L OE ) returns the OE cost, L OE ( K ) and Grad ( K , L OE ) return the gradient of the OE cost, r L OE ( K ) .
Despite this simpliﬁcation, we would like to again emphasize that these can be efﬁciently approximated in ﬁnite samples, and purely on the basis of observations y t subsampled in discrete intervals.
For further discussion, please see Appendix C.
Lastly, in addition to standard cost and gradient evaluations of the L OE loss, as part of our algorithm, we further require access to gradient and cost evaluations of smooth functions of the stationary-state covariance.
Speciﬁcally, if f : S 2 n + !
R is a function of the covariance matrix K , we assume we can compute Eval cov ( K , f ) which returns the f ( ⌃ K ) and Grad cov ( K , f ) which returns r K f ( ⌃ K ) .
In Appendix C.2, we show that these oracles can be implemented without direct state access by “subsampling” multiple observations at different time steps.
Main Results
In this section, we present the main contributions of our work.
After demonstrating that the OE cost function contains stationary points that are not globally optimal, we present informativityregularized policy gradient (IR-PG), a direct policy search algorithm based on a novel regularization strategy to preserve informativity , introduced in Section 1.1.
We state a formal convergence result showing that IR-PG converges to a globally optimal ﬁlter at a O (1 /T ) rate.
The efﬁcacy of
IR-PG is illustrated via numerical experiments in Appendix F.6, cf.
Fig.
3 in particular to see how informativity regularization dramatically improves the convergence of policy gradient methods.
3.1 Existence of suboptimal stationary points
Perhaps the simplest model-free approach to the OE problem is to run gradient descent on L OE ( · ) .
Under mild assumptions on the loss function L OE , gradient descent will converge to a ﬁrst-order stationary point of L OE .
Unfortunately, despite the existence of a convex reformulation [Scherer et al., 1997] and Fact 1.1, the L OE loss function contains suboptimal stationary points:
Example 3.1.
Consider the OE instance given by A
=
I 2 , C
= I 2 , W 1 = 3 ⇥ I 2 , W
= I 2 , and the ﬁlter K bad given by A bad =  " ⇥ I 2 , " > 0 , B bad = 0 2 , C bad = 0 2 .
K bad constitutes a suboptimal stationary point of L OE for this OE instance.
The example is similar in spirit to Tang et al.
[2021, Theorem 4.1]; details are given in Appendix F.1.
Importantly, the cost is invariant under perturbations to any single parameter of the ﬁlter and K bad , being equivalent to the zero-ﬁlter, is suboptimal.
The same is true for any OE instance: every ﬁlter with B bad = 0 , C bad = 0 , and A bad being stable is a suboptimal stationary point of L OE .
We again emphasize that these suboptimal stationary points arise when the change of variables in the convex reparametrization “breaks down”, i.e., these suboptimal ﬁlters are not in the image of the reformulation map ( · ) of Fact 1.1.
The perils of enforcing minimality.
A ﬁlter K is minimal if ( A K , B K ) is controllable, and ( A K , C K ) is observable.
Example 3.1 is the extreme case of a non-minimal ﬁlter, since B bad =
C bad = 0 2 .
Conversely, as a special case of LQG , the OE problem inherits the property that all stationary points corresponding to minimal ﬁlters are globally optimal [Tang et al., 2021, Theorem 4.3].
Therefore, it may be natural to ask: can a local search algorithm enforce minimality to avoid suboptimal stationary points?
A classical result due to Brockett [1976] suggests not: the set of minimal n -th order single-input-single-output transfer functions (e.g. ﬁlters) is the disjoint union of n + 1 open sets.
Thus it is impossible for a continuous path to pass from one of these open sets to another without entering a region corresponding to a non-minimal ﬁlter, suggesting that a local search algorithm regularized to ensure minimality at every iteration may never converge to the optimal solution (unless a sufﬁciently large, and lucky, step allows the iterate to hop over the boundary at non-minimality).
See Appendix F.2 for further discussion and supporting numerical experiments.
Futher challenges.
Given the drawbacks of enforcing minimality, one may wonder whether it is sufﬁcient to enforce controllability or observability alone.
In Example 3.1 above - and indeed, for all the examples in Tang et al.
[2021] of suboptimal stationary points in the LQG landscape - there is a loss of both observability ( C bad = 0 ) and controllability ( B bad = 0 ).
Unfortunately, suboptimal stationary points can occur at controllable policies and observable policies. Example 3.2. Consider the OE instance given by [Equation]
and the ﬁlter K bad given by
[Equation] The ﬁlter K bad is a controllable but strictly suboptimal ﬁrst-order critical point of L OE ( K ) . A formal proof is provided in Proposition F.1, but the intuition is as follows: K bad is suboptimal because no information about the second state of the true system ever enters the ﬁlter (due to sparsity of B bad ).
However, K bad is controllable as  > 0 .
Further discussion appears in Appendix F.3, where it is also shown that one can similarly construct examples of ﬁlters that are observable (but not controllable) that correspond to strictly suboptimal ﬁrst-order critical points of L OE ( K ) .
There we also demonstrate that, for the setup in Example 3.2, the minimum eigenvalue of the Hessian r 2 L OE ( K bad ) can be made arbitrarily close to zero by taking  in A bad to be arbitrarily large.
This casts doubt on whether approaches based on saddle-point escape with appropriate random perturbations [Jin et al., 2017] or acceleration [Jin et al., 2018] should be expected to work.
3.2 A provably convergent algorithm
The previous discussion puts us in a bind: regularizing to preserve minimality may introduce problems related to path-disconnectedness; yet, neither controllability nor observability alone is sufﬁcient to rule out suboptimal stationary points.
We identify a stronger condition called informativity which is sufﬁcient.
We deﬁne the set of informative ﬁlters as [Equation]
First, we check that all optimal ﬁlters lie in this set, under Assumption 2.4 (proof in Appendix E.3).
Lemma 3.1.
Under Assumptions 2.1 to 2.4, K opt ⇢ K info ⇢ K ctrb , and K info is an open set.
Our key insight is that informativity is also sufﬁcient to ensure optimality of stationary points, but does not cause path-connectedness issues as it did for minimality (see Appendix G.3 for the proof).
Theorem 1.
Let K 2 K info ; then (i) there is a continuous path lying in K info connecting K to some K ?
2 K opt and (ii) if r L OE ( K ) = , then K 2 K opt .
Theorem 1 suggests that gradient descent with enforced informativity should converge to optimal ﬁlters.
It is not, however, implied by the landscape analysis of Tang et al.
[2021], which focuses solely on minimal stationary points.
Still, numerous challenges remain: (1) How can one enforce informativity in a smooth fashion?
(2) What quantitative measure of informativity provides quantitative suboptimality guarantees on approximate ﬁrst-order stationary points?
Given that L OE ( K ) need not have compact level sets (see Appendix F.3), how does one ensure that the iterates of policy search do not escape to inﬁnity, or reach regions where the loss of smoothness is arbitrarily poor?
The explained covariance matrix.
In light of Theorem 1, we design a policy search algorithm which ensures that ⌃ 12 , K remains full-rank throughout the search, but does so in a quantitative fashion.
Our central object is the explained covariance matrix , which measures how much of the covariance of the steady-state system x ( t ) is explained by the internal ﬁlter state ˆ x K ( t ) in the large t limit: Z K :=
lim t !
1 (Cov[ x ( t )]
E
[Cov[ x ( t ) | ˆ x K ( t )]]) .
When K 2 K ctrb , Z K admits an elegant closed-form expression, which provides an alternative deﬁnition of K info :
[Equation]
that
[Equation]
[Equation]
Since Z K is invariant under similarity transformations, Z K can be interpreted as a normalized analogue of ⌃ 12 , K .
Informally, the quadratic form v > Z K v is a sufﬁcient statistic for how much information ˆ x ( t ) contains about the “ v -direction” of x ( t ) ; see Appendix E.6 for a precise statement.
Explained-covariance regularization.
We preserve informativity by ensuring our iterates satisfy
Z K  0 .
To this end, we run gradient descent on the regularized objective for some  > 0 :
[Equation]
, where R info ( K ) :
= ⇢ tr[ Z  1 K ] K 2 K info otherwise .
The proposed regularizer has several important properties.
First, though R info is nonconvex in
K , both R info and L OE can be made convex under the same change of variables, in the sense of
Fact 1.1, cf.
Lemma I.2.
In addition, R info is non-negative, and tends to 1 as Z K approaches singularity.
Furthermore, R info is invariant under similarity transformations.
Next, many of the essential quantities arising in our analysis can be bounded in terms of Z  1 K , justifying Z K is a natural quantitative measure of informativity.
Lastly, the set of global-minimizers of R info ( · ) is precisely the optimal ﬁlters for the OE problem, as per the following lemma (see Appendix E.4 for proof).
Lemma 3.2 (Existence of maximal Z K ) .
Under Assumptions 2.1 to 2.3, there exists a unique Z ?
such that Z ?
=
Z K if and only if K 2 K opt , and Z ? ⌫ Z K for all K 2 K ctrb \ K opt .
Consequently,
K opt 2 arg min K 2 K ctrb R info ( K ) .
Lemma 3.2 directly implies that the suboptimality of L  ( · ) upper bounds the suboptimality in L OE ( · ) , so we can minimize
L  as a proxy for minimizing L OE .
Corollary 3.1.
For any K , we have L OE ( K )  min K 0 L OE ( K 0 )
 L  ( K )  min K 0 L  ( K 0 ) .
Reconditioning.
We introduce an additional normalization step between policy updates to ensure the iterates produced by our algorithm have well-conditioned covariance matrices; this in turn ensures the iterates produced by our algorithm remain in a compact set, and that the smoothness of L  is uniformly bounded.
For any ﬁlter K 2 K ctrb such that ⌃ 22 , K  0 , the reconditioning operator recond ( K ) returns a ﬁlter K 0 which is equivalent to K , but for which ⌃ 22 , K
= I n .
Formally 3 , recond ( K ) :
= Sim S ( A K , B K , C K ) , where S = ⌃  1 / 2 22 , K .
Since K and K 0 = recond ( K ) are equivalent realizations, we have L OE ( K ) =
L OE ( K 0 ) , R info ( K ) =
R info ( K 0 ) , and thus L  ( K ) =
L  ( K 0 ) .
Statement of IR - PG.
We can now describe our algorithm IR-PG, whose pseudocode is displayed in Algorithm 1.
IR-PG applies gradient descent on the regularized L OE objective, with an additional balancing step between gradient updates.
Appendix B.1 provides a variant where the step size is chosen by backtracking line-search, which enjoys the same rigorous convergence guarantees.
To guarantee convergence to an optimal ﬁlter (and ﬁniteness of L  ), we need to initialize at a ﬁlter such that Z K 0  0 , i.e. K 0 2 K info .
Fortunately, random initializations from a continuous distribution satisfy this condition with probability 1 (see Appendix E.7 for a formal statement and proof).
Concerning the choice of  , the formal guarantees in Section 3.3 below hold for any  > 0 .
Nonetheless, the choice of  can inﬂuence the numerical properties of the loss function, such as smoothness and the size of the gradients; cf.
Appendix F.6 for further discussion.
3.3 Formal guarantees
We conclude this section by stating the formal convergence guarantee for IR-PG.
Our results depend on natural problem quantities, among which is the minimum singular value of P ?
(as deﬁned in
Eq.
(2.5)), which we show is always strictly positive.
Lemma 3.3.
Let P ?
be the solution to the Riccati equation in Eq.
Then under Assumptions 2. to 2.3,  ? :
=
min ( P ? ) is strictly positive.
Moreover, P ?
=
⌃ 11 , sys  Z ? .
In other words, P ? is the limiting conditional covariance of the true state x ( t ) given the policy-state ˆ x K ( t ) under (any) optimal policy.
Lemma 3.3 states that this covariance is nonsingular, i.e. not even optimal policies contain perfect information about any mode of x ( t ) .
Given an initialization K 0 , our convergence rate depends polynomially on the following problem parameters:
[Equation]
Theorem 2.
Fix  > 0 , K 0 2 K info .
There are terms C 1 , C 2  1 , which are at most polynomial in n, m, C sys ,  ,   1 and L  ( K 0 ) , such that the iterates of IR-PG with any stepsize ⌘  1 C 1 satisfy
[Equation]
The formal guarantee for back-tracking stepsizes is nearly analogous, and given in Appendix B.1.
The oracle complexity of IR-PG is described in Appendix B.2.
We sketch the highlights of the proof in the following section, after introducing our DCL framework.
A rigorous proof overview is deferred to Appendix G.2.
Analysis Framework
In this section, we sketch our analysis in two halves.
In the ﬁrst, we provide a framework for analyzing gradient descent on objectives which admit convex reformulations.
This takes place in two parts: ﬁrst, via our technique of differentiable convex liftings ( DCL s) we establish a form of gradient dominance we term weak-PL.
Second, we show that weak-PL, together with smoothness (i.e. upper bound on the magnitude of the Hessian) implies convergence of gradient descent at a O (1 /t ) rate.
The DCL formalizes the developments in the “our techniques” discussion of Section 1.1.
Next, we instantiate the analysis framework for regularized output estimation loss L  deﬁned in the previous section.
It sufﬁces to establish both (a) smoothness of L  and (b) weak-PL by exhibiting a valid DCL .
Interestingly, the upper bounds on both the Hessian magnitude and the weak-PL factor depend on the magnitude of tr[ Z  1 K ] , which is precisely what is controlled by the informativity regularizer in Eq.
4.1 Differentiable convex liftings ( DCL s)
This section introduces differentiable convex liftings ( DCL s), a rigorous and ﬂexible framework for operationalizing convex reformulations of nonconvex objectives.
Preliminaries.
To neatly accommodate optimization over constrained domains, we express functions f : R d !
¯ R as taking values in the extended reals ¯ R :=
R [ { 1 } .
Given such an f , we denote its domain dom ( f ) : { x : f ( x ) 6 = 1 } as the set on which f is ﬁnite; we say f is proper if dom ( f ) 6 = ; ; we deﬁne its minimal value inf( f )
=
inf x 2 R d f ( x ) .
We say f 2 C k
( K ) on a domain
K if f is k -times continuously differentiable (and ﬁnite) on some open set containing K ⇢ R d .
DCL s .
The DCL is a generic template for convex reformulation that signiﬁcantly generalizes the setup in Fact 1.1.
Rather than relating f directly to a convex f cvx , we “lift” f to a function f lft by appending auxiliary variables.
We then assume a reparametrization  mapping the domain of f lft to that of f cvx in this higher dimensional space; intuitively,  is the “inverse” of in Fact 1.1.
Deﬁnition 4.1.
A triplet of functions ( f cvx , f lft ,  ) is a DCL of a proper function f : R d !
¯ R if
(1) f cvx : R d z !
¯ R is a proper convex function whose minimum is attained by some z ? .
(2) For some additional number of parameters d ⇠  0 , f lft : R d + d ⇠ !
¯ R is related to f via partial minimization: f ( x ) =
min ⇠ 2 R d ⇠ f lft ( x , ⇠ ) .
(3) There is an open set Y ◆ dom ( f lft ) for which  : Y !
dom ( f cvx ) is C 1 and satisﬁes f lft ( · ) =
f cvx (  ( · )) .
Above, d z and d ⇠ are arbitrary dimensions representing the arguments of f cvx ( · ) and f lft ( x , · ) , respectively.
The mere existence of a DCL implies that approximate stationary points of f are also approximate minimizers, under conditions elaborated on below:
Theorem 3.
Let f : R d !
¯ R be a proper function with DCL ( f cvx , f lft ,  ) .
Then, for any xdom ( f ) at which f is differentiable, f satisﬁes the weak-PL condition:
[Equation]
where ↵
[Equation]
x
[Equation]
Theorem 3 strengthens Fact 1.1 in two respects.
For one, it does not impose any smoothness restrictions on f lft or f cvx ; in particular f cvx can be highly non-smooth and, due to the extended-real function formulation, can also include constraints.
And second, the lifting f lft adds considerable ﬂexibility, which we show is necessary to capture the convex reformulation of OE (Appendix I).
The factor ↵ DCL ( x ) depends on two quantities.
The numerator is the d z -th singular value of r  ( x , ⇠ ) for any ⇠ 2 arg min f lft ( x , · ) .
This captures how large perturbations of f lft ’s arguments must be in order to achieve a desired perturbation of the arguments of f cvx , under the reparameterization  .
The additional arguments in f lft compared to f adds additional columns to r  thereby making it easier to ensure  d z ( r  ( · )) > 0 .
On the other hand, the denominator measures the Euclidean distance between any minimizer of the convex function f cvx ( · ) and image of ( x , ⇠ ) under the reparameterization  , and can be bounded under quite benign conditions.
Proof Sketch.
The formal proof of Theorem 3 (given in Appendix H.2) takes special care to handle cases where f lft and f cvx are ﬁnite only on restricted domains, as well as possible nonsmoothness; still, the main ideas behind are intuitive.
For f cvx being convex, f cvx ( z )  inf( f cvx ) =
O ( kr f cvx ( z ) k ) .
Using the DCL deﬁnition and analyzing the inverse image of a point under  , we can also establish a gradient dominance result for f lft .
Weak-PL for the original function f follows since f is related to f lft by partial minimization, so its gradients must be larger than those of f lft .
Gradient descent with DCL s. We now describe how DCL s yield quantitative convergence guarantees for gradient descent.
A more general guarantee accommodating the reconditioning step in
IR-PG is deferred to Appendix G.1, and encompasses the bound below as a special case.
Given ↵ > 0 , we say that proper f : R d !
¯ R satisﬁes ↵ - weak-PL (named after the stronger PolyakŁ ojasiewicz condition) on a domain K ⇢ R d if f 2 C 1 ( K ) and
kr f ( x ) k  ↵ ( f ( x )  inf( f ) .
From Theorem 3, we see f satisﬁes ↵
K -weak PL on K if f has a DCL and ↵ K :=
inf x 2 K ↵ DCL ( x ) > 0 .
To analyze gradient descent, we also require smoothness: we say f is  -upper-smooth on
K if f 2 C 2 ( K ) and for all x 2 K , r 2 f ( x )   I .
The following follows from a standard descent lemma for smooth (though possibly nonconvex) functions.
Proposition 4.1.
Let x 0 2 dom ( f ) , and suppose that the level set K ( x 0 ) :
= { x : f ( x )  f ( x 0 ) } is compact, and f satisﬁes ↵ x 0 -weak PL with ↵ x 0 > 0 , and  x 0 -upper smoothness on K ( x ) with  x 0 > 0 .
Then, x 0 lies in the same path-connected component of some minimizer of f , and for any ⌘  1 /  x 0 the updates x k {++
+}
= x k  ⌘ r f ( x k ) satisfy f
( x k )  inf( f )  2 / ( k · ↵ 2 x 0 ⌘ ) .
4.2 Proof of Theorem 2.
In light of Proposition 4.1 (and its generalization to include reconditioning), it sufﬁces to establish both the weak-PL and smoothness of the loss L  ( · ) on the well-conditioned sublevel set K 0 :
= { K : L  ( K )  L  ( K 0 ) , 12 I n  ⌃ 22 , K  2 I n } .
To establish the weak-PL property, we exhibit a DCL for which ↵ DCL ( K ) depends only on the operator norms of ⌃ K , ⌃  1 K , Z K : as well as other systemquantities.
Here, poly op ( . . . ) denotes a quantity at most polynomial in the operator norms of its arguments.
Proposition 4.2.
For any   0 (non-strict), the objective L  ( K ) admits a DCL ( f cvx , f lft ,  ) where the lifted parameter takes the form ( K , ⌃ K ) 2 K info ⇥ S 2 n ++ , L  ( K )
=
f lft ( K , ⌃ K
[Equation] = min ⌃ [Equation] + f lft ( K , ⌃ ) , and where
[Equation]
Furthermore, the norms of the parameters A K , B K , C K satisfy the following bounds:
[Equation]
Smoothness is established in (Proposition G.5), which relies on a novel (and rather challenging) bound on the solutions to Lyapunov equations involving A cl , K :
Proposition 4.3 (Stability of A cl , K ) .
Suppose that K 2 K info .
Then, for any matrix Y 2 S 2 n , the solution ⌃ K , Y to the Lyapunov equation A cl , K ⌃ K , Y + ⌃ K , Y A > cl , K + Y = 0 satisﬁes k ⌃ K , Y k   C lyap ( K ) · k Y k  , where C lyap ( K ) =
poly op  ⌃ K , ⌃  1 K , Z  1 K , W  1 1 , W  1 2 , C  , and where k · k  denotes either the operator, Frobenius, or nuclear norm.
The regularization R info ( K ) ensures k Z  1 K k remains bounded on the sublevel K 0 ; we further show (Lemma G.7) that 12 I n  ⌃ 22 , K  32 I n implies ⌃ K is invertible, and ensures k ⌃ K k , k ⌃
1 K k are uniformly bounded.
Thus, the weak-PL constant and smoothness parameters are uniformly bounded on K 0 , concluding the proof.
We stress that the proofs of Proposition 4.2 and Proposition 4.3 require several novel technical arguments, which may be of independent interest.
A full proof roadmap, including formal statements of the aforementioned results, is given in Appendix G.2.
The fact that k ⌃ K k , k ⌃  1 K k , k Z  1 K k appear throughout the analysis suggests that (a) informativity as measured by
Z  1 K , and (b) the conditioning of ⌃ 22 , K may be fundamental to the OE landscape.
Proof of Theorem 1.
The proof of Theorem 1 is simpler, as it relies mainly on the existence of a
DCL for which ↵ DCL does not vanish.
See Appendix G.3 for details.
Conclusion
In this work, we introduce the ﬁrst direct policy search algorithm which converges to the globally optimal dynamic policy for the output estimation problem. We hope that our analysis serves as a valuable starting point to study direct policy search for reinforcement learning and control problems with partial observations, in which the relevant classes of policies are dynamic ones that maintain an internal state. We also hope that both our proposed principle of “informativity”, and our technical contributions around convex reformulations, continue to prove useful in future work.
Acknowledgments and Disclosure of Funding
The authors acknowledge Pablo Parrilo and Tobia Marcucci for helpful discussions.
J.U. is supported by NSF Award No. EFMA-1830901, Ofﬁce of Naval Research (ONR) Award No. N0001418-1-2210 and ONR Award No. N00014-17-1-2699.
M.S. is supported by Amazon.com Services
LLC, PO# #D-06310236 and the MIT Quest for Intelligence.
K.Z. is supported by a SimonsBerkeley Research Fellowship.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section.
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] See Section 2.
(b) Did you describe the limitations of your work?
[Yes] See Assumption 2.4 and Appendix D. (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section(b)
Did you include complete proofs of all theoretical results?
[Yes] See appendices.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Code provided in supplementary materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix F.6.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Figs.
3 and 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See beginning of Appendix F.6.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]