Globally Convergent Policy Search for Output Estimation
Abstract
We introduce the ﬁrst direct policy search algorithm which provably convergesto the globally optimal dynamic ﬁlter for the classical problem of predicting theoutputs of a linear dynamical system, given noisy, partial observations.
Despitethe ubiquity of partial observability in practice, theoretical guarantees for directpolicy search algorithms, one of the backbones of modern reinforcement learning,have proven difﬁcult to achieve.
This is primarily due to the degeneracies whicharise when optimizing over ﬁlters that maintain an internal state.
In this paper, we provide a new perspective on this challenging problem basedon the notion of informativity , which intuitively requires that all components of aﬁlter’s internal state are representative of the true state of the underlying dynamicalsystem.
We show that informativity overcomes the aforementioned degeneracy.
Speciﬁcally, we propose a regularizer which explicitly enforces informativity, andestablish that gradient descent on this regularized objective - combined with a“reconditioning step” – converges to the globally optimal cost at a O (1 /T ) rate.
Introduction
Data used for prediction and control of real world dynamical systems is almost always noisy andincomplete (partially observed).
Sensors and other measurement procedures inevitably introduceerrors into the datasets, so designing reliable learning algorithms for these noisy or partially observeddomains requires confronting fundamental questions of disturbance ﬁltering and state estimation.
Despite the ubiquity of partial observation in practice, these concerns are often underexplored inmodern analyses of learning for control that assume perfect observations of the underlying dynamics.
In this work we study the output estimation ( OE ) problem or learning to predict in partially observedlinear dynamical systems.
The output estimation problem is one of the most fundamental problemsin theoretical statistics and learning theory.
Both in theory and in practice, advances in predictingpartially observed linear systems have led to successes in a variety of areas from controls to biologyand economics, (c.f. e.g. Athans [1974], Lillacci and Khammash [2010], Gautier and Poignet[2001]).
We revisit this classical problem from a modern optimization perspective, and study thepossibility of learning the optimal predictor via model-free procedures and direct policy search.
Relative to model-based procedures, which ﬁrst estimate the underlying dynamics and then returna policy by solving an optimization problem using the estimated model, model-free methods offerseveral potential advantages.
First, direct policy search allows one to easily specify the complexity ofthe policy class over which one searches.
For example, the vast majority of industrial control systemsare built upon proportional-integral-derivative (PID) controllers.
Each PID controller comprisesthree scalar variables (gains) yet successfully regulates complex feedback loops in high-dimensionalsystems (e.g chemical plants).
In addition, model-free policy search optimizes for performancedirectly on the true system, rather than an approximate model.
As such, there is no gap between the model used for synthesis and the system on which the controller is deployed.
Such gaps are typicallycovered by robust control techniques
[Zhou et al., 1996], which may introduce conservatism.
In light of these advantages, there has recently been signiﬁcant interest from both theoreticians andpractitioners in understanding the foundations of model-free control.
So far, this attention has focused mostly on problems with full-state observation such as the linear quadratic regulator ( LQR ) orfully-observed Markov Decision Processes (MDPs), for which optimal policies are static .
Progresson partially observed problems has been complicated by the difﬁculties associated with optimizingover dynamic policies that maintain internal state to summarize past observations.
In this paper, weprovide the ﬁrst policy search algorithm which provably converges to the globally optimal ﬁlter forthe OE problem, and shed new light on the intricacies of the underlying optimization landscape.
The Output Estimation problem.
We study one of the simplest and most basic problems withpartial observability: the output estimation ( OE ) problem.
In brief, the goal is to search for a predictor of the output z ( t ) of a linear dynamical system given partial measurements y ( t ) .
For the truesystem with states x ( t ) and dynamics that evolve according to,
[Equation]
the goal is to ﬁnd the parameters
K =
( A K , B K , C K ) of the ﬁlter (interchangably, policy ),
[Equation]
that minimizes the steady-state prediction error,
[Equation]
In this paper, we study solving the OE problem via model-free methods, where the goal is to searchfor the optimal ﬁlter parameters K
=
( A K , B K , C K ) using direct policy search without knowledgeor estimation of the true system parameters A , C , G , W 1 , W 2 ; see Section 2 for further details.
While the OE problem is not a control problem per se, it is an attractive stepping stone for studyingmodel-free control.
Like LQG , optimal policies for OE are dynamic , i.e., maintain an internal state.
It therefore serves as a valuable testbed to develop the theoretical foundations of direct policy searchfor this important class of policies which has thus far remained poorly understood.
Related work.
Recent years have witnessed a resurgence of interest in direct policy search.
For continuous control problems, Fazel et al.
[2018] established global convergence of policy gradient methods on discrete-time LQR .
Subsequent work has sharpened rates [Malik et al.
, 2019],analyzed convergence under more general frameworks [Bu et al., 2019], and extended the analysisto continuous-time [Mohammadi et al., 2021].
Beyond LQR , Zhang et al.
[2020] analyzed policysearch for mixed H 2 / H 1 and risk-sensitive control.
Sun and Fazel [2021] also considered analysis 65 via convex reformulations, for state feedback problems.
For discrete state-action MDP s, Agarwalet al.
[2021] established convergence rates for a variety of policy gradient methods, cf.
also Bhandari and Russo [2019].
All of these works considered static state-feedback policies, with perfectstate information.
For problems with partial observation - which necessitates the use of dynamicpolicies - the work most relevant to our own, is Tang et al.
[2021], which studied policy search for LQG .
They established that all stationary points corresponding to minimal controllers are globallyoptimal, and demonstrated that gradient descent may fail to converge to optimal policies.
1.1 Contributions
We propose a novel policy search method which provably converges to a globally optimal L OEcost.
Despite extensive prior work on static policy search (see above), our result constitutes the ﬁrstrigorous guarantee for policy search over dynamic policies.
While the OE problem admits a convexreformulation [Scherer et al., 1997], it is also well known that its landscape contains suboptimalcriticial points.
This main seem like a contradiction at ﬁrst, but we demonstrate such spuriouscritical points occur precisely when the policy loses what we term “informativity”: that the steadycovariance between the controller state and system state, ⌃ 12 := lim T !
1 1 T [
R
T 0 x ( t )ˆ x ( t ) >
d t ] , becomes rank deﬁcient.
With this insight, we propose a regularizer R that ensures the internalstate of the learned policy remains “uniformly informative” about the state of the true system, andprove that gradient descent on the regularized objective L OE ( · ) +  R ( · ) converges at a O (1 /T ) 83 rate to an optimal policy.
The efﬁcacy of the approach is illustrated via numerical experiments, cf.
Appendix F.6.
Code is provided in the supplementary material.
Our techniques.
Searching over dynamic policies introduces two key challenges: (1) spuriouscritical points can arise when one or more factors become “degenerate” in a certain way; (2) changesof basis produce a continuum of “equivalent realizations” of the ﬁlter K , some of which are poorlyconditioned.
Similar challenges have been observed in problems with rotational symmetries, e.g.nonconvex matrix factorization.
Neither challenge arises when searching over static policies [Fazelet al., 2018].
Our approach is centered around the idea of convex reformulations of control synthesis problems, and the following fact regarding functions which admit these reformulations, cf.
Appendix H. 1 for proof.
Fact 1.1.
Let f : R n x !
R be a differentiable, possibly nonconvex function such that min x f ( x ) isﬁnite.
There exists a differentiable function : R n ⌫ !
R n x satisfying the following two properties:
(i) the mapping is surjective, i e. for all x 2 R n x there exists ⌫ 2 R n ⌫ such that x = ( ⌫ ) , (ii) under the change of variables the function f cvx ( ⌫ ) :
= f
( ( ⌫ )) is differentiable and convex .
Then all ﬁrst-order stationary points, x s.t r f ( x ) = 0 , are globally optimal.
The OE problem, LQG, and many other related control tasks admit convex reformulations [Schereret al., 1997, Masubuchi et al., 1998].
Given that gradient descent (under mild regularity assumptions)converges to stationary points, we might hope that Fact 1.1 guarantees that direct policy search onthe OE ﬁlter will succeed at ﬁnding an optimal policy, when applied to loss functions admitting suchconvex reformulations.
Somewhat surprisingly, we ﬁnd that this is emphatically not the case: gradient descent on the L OE objective fails to reliably converge to optimal solutions (see Section 3.1).
To resolve this paradox, we show that the surjectivity condition of Fact 1.1 may fail for the convexreparametrization of OE : there are ﬁlters K with ﬁnite cost L OE ( K ) , which are not in the image ofthe reformulation map ( · ) .
We ﬁnd that degeneracy occurs precisely when informativity , deﬁnedin Section 1.1 as ⌃ 12 , K having full rank, fails to hold.
Conversely, when ⌃ 12 , K is full-rank, theconditions of Fact 1.1 are met and the parametrization behaves as needed.
Thus, we identify noninformativity - rank deﬁciency of ⌃ 12 , K - as the fundamental notion of degeneracy correspondingto challenge (1).
Motivated by this observation, we introduce a novel “informativity regularizer”
R info ( · ) which enforces that ⌃ 12 , K is full rank.
Our proposed algorithm, IR-PG alternates betweengradient updates on the regularized loss L  ( · ) :
= L OE ( · ) +  R info ( · ) , and “reconditioning” steps 113 to ensure well-conditioned realizations of the ﬁlters K , thereby addressing challenge (2) above.
Westress that our notion of informativity differs from the minimality criterion emphasized in Tang et al.[2021], whose limitations we discuss in Section 3.1.
In order to achieve our quantitative converge guarantees, we establish numerous results which maybe of independent interest, including: (i) a quantitative analysis of the OE convex reformulation dueto Scherer [1995], (ii) novel bounds on the magnitude of solution to Lyapunov equations under theclosed-loop OE ﬁlter dynamics.
Both arguments appeal to a (quantitative measure of) informativity.
Finally, we develop a quantitative analogue of Fact 1.1 via a paradigm we call differentiable convexliftings , or DCL s, described at length in Section 4.1.
Preliminaries
Before presenting our main results in Section 3, we ﬁrst introduce some of the relevant deﬁnitions,and provide the reader with some the relevant background on prediction in partially-observed dynamical systems.
We adopt standard notation wherever possible, and for brevity, defer details to
Appendix A. As outlined in the introduction, we consider the problem of predicting the outputs ofa partially observed linear dynamical system.
We refer to the dynamical system deﬁned in Eq.
(1.1)as the true system , with states x ( t ) 2 R n , observations y ( t ) 2 R m , and performance outputsz ( t ) 2 R p .
To ensure the dynamics have a well-deﬁned steady-state, we assume that A is stable.
Assumption 2.1.
The matrix A is Hurwitz stable .
That is, the real components of all its eigenvaluesare strictly negative: <
[  i ( A )] < 0 for i 2 [ n ] .
Because these policies only access the system outputs, and are only evaluated in relation to systemoutputs, we assume that the true system state is observable .
Further, we assume that dynamics aresubject to sufﬁciently rich noise excitations.
Assumption 2.2.
The pair ( A , C ) in Eq.
(1.1) is observable.
That is, the observability Gramiandeﬁned as G obs :=
[Equation]
exp
[Equation]
> C > C exp( s A )
d s is strictly positive deﬁnite.
Assumption 2.3.
We assume that the noise matrices W 1 and W 2 are strictly positive deﬁnite.
As stated previously, we restrict our attention to ﬁnding the best dynamic ﬁlter within the parametricfamily described in Eq.
Note that this family contains the Bayes optimal predictor for the L OEobjective, as we will later describe in more detail.
We review a number of basic facts:
A. Steady state distributions.
We deﬁne K stab :=
K : A K is Hurwitz stable }
to be the
set of 142 ﬁlters such that
A
K is stable.
Under Assumption 2.1, Appendix E. shows this equivalent to stabilityof the closed-loop matrix A cl , K .
[Equation]
Hurwitz [Equation] Stability of A cl , K is a sufﬁcient condition for L OE ( K ) to be ﬁnite, and for the following limitingcovariance to be well deﬁned, ⌃ K = lim t !
1 E
[Equation]
2 S 2 n + .
This steady-state covariance is given by the solution to the continuous-time Lyapunov equation,
[Equation]
where W cl , K :=
[Equation]
Notice that ⌃ K depends only on ( A K , B K ) , but not on C K , and that the ﬁrst n ⇥ n block of ⌃ K doesnot depend on K at all.
To highlight these distinctions, we partition
[Equation]
and deﬁne K ctrb := { K 2 K stab : ⌃ 22 , K  0 } as the set of ﬁlters whose internal state covariance 150 is full rank.
We refer to these as the controllable policies, as these are precise the policies for whichthe pair ( A K , B K ) is controllable.
B. Equivalent realizations.
There are many different ways of parametrizing a given dynamicfeedback policy, all of which have exactly the same input-output behavior.
In particular, given aninvertible matrix S 2 GL ( n ) , the OE loss of a ﬁlter K is invariant under the following class ofsimilarity transforms: Sim S ( K ) :
( A K , B K , C K ) 7!
( SA K S  1 , SB K , C K S  1 ) .
Formally, for any
K 2 K stab and any S 2 GL ( n ) , L OE ( K )
=
L OE ( Sim S ( K )) .
We say that K and K 0 are equivalent 157 realizations if they are related by a similarity transformation Sim S ( K )
=
K 0 for some S 2 GL ( n ) .
158 Note that the set
K ctrb is also preserved under similarity transformation.
C. Optimal policies.
The landmark result by Kalman shows that for the system deﬁned by( A , C , W 1 , W 2 ) the Kalman ﬁlter K ?
=
( A  L ?
C , L ? , G ) achieves minimal L OE loss.
Here,
L ? is the Kalman gain which is deﬁned in terms of the solution of the following Riccati equation:
[Equation]
We deﬁne the set of optimal ﬁlters K opt to be those which are equivalent to the Kalman ﬁlter:
[Equation]
D. Restricted problem setting.
The problem description outlined above, including Assumptions 2.1 to 2.3, constitutes the standard OE problem, well-known in control theory, cf.
[Doyleet al., 1989, §IV.D].
In this paper, we will make the following additional assumption that restrictsthe class of OE problems we consider.
Further discussion on the utility and necessity of this assumption (for our analysis) is provided in Section 3.2 and Appendix D; the latter also shows that
Assumption 2.4 holds for “generic” problem instances.
Assumption 2.4.
The optimal policy is itself controllable, i.e. for all ( A K ? , B K ? , C K ? )
2 K opt we 170 have that ( A K ? , B K ? ) is controllable.
E. Interaction protocol.
In the spirit of model-free methods, we introduce algorithms whichwork only assuming access to cost and gradient evaluation oracles.
We abstract away the particularimplementation of these oracles to simplify our presentation and assume that they are exact, in orderto focus on the overall optimization landscape of the OE problem.
More formally, for any ﬁlter
K 2 K stab , Eval ( K , L OE ) returns the OE cost, L OE ( K ) and Grad ( K , L OE ) return the gradient of the 176 OE cost, r L OE ( K ) .
Despite this simpliﬁcation, we would like to again emphasize that these can beefﬁciently approximated in ﬁnite samples, and purely on the basis of observations y t subsampled inin discrete intervals.
For further discussion, please see Appendix C.
Lastly, in addition to standard cost and gradient evaluations of the L OE loss, as part of our algorithm,we further require access to gradient and cost evaluations of smooth functions of the stationary-statecovariance.
Speciﬁcally, if f : S 2 n + !
R is a function of the covariance matrix K , we assume wecan compute Eval cov ( K , f ) which returns the f ( ⌃ K ) and Grad cov ( K , f ) which returns r K f ( ⌃ K ) .
In Appendix C.2, we show that these oracles can be implemented without direct state access by“subsampling” multiple observations at different time increments.
Main Results
In this section, we present the main contributions of our work.
After demonstrating that the OEcost function contains stationary points that are not globally optimal, we present informativityregularized policy gradient (IR-PG), a direct policy search algorithm based on a novel regularization strategy to preserve informativity , introduced in Section 1.1.
We state a formal convergenceresult showing that IR-PG converges to a globally optimal ﬁlter at a O (1 /T ) rate.
The efﬁcacy of
IR-PG is illustrated via numerical experiments in Appendix F.6, cf.
Fig.
3 in particular to see howinformativity regularization dramatically improves convergence of policy gradient.
3.1 Existence of suboptimal stationary points
Perhaps the simplest model-free approach to the OE problem is to run gradient descent on
L OE ( · ) .Under mild assumptions on the loss function L OE , gradient descent will converge to a ﬁrst- 196 order stationary point of
L OE .
Unfortunately, despite the existence of a convex reformulation[Scherer et al., 1997] and Fact 1.1, the L OE loss function contains suboptimal stationary points:
Example 3.1.
Consider the OE instance given by A
=
I 2 , C
= I 2 , W 1 = 3 ⇥ I 2 , W
= I 2 ,and the ﬁlter K bad given by A bad =  " ⇥ I 2 , " > 0 , B bad = 0 2 , C bad = 0 2 .
K bad constitutes asuboptimal stationary point of L OE for this OE instance.
The example is similar in spirit to Tang et al.
[2021, Theorem 4.1]; details are given in Appendix F.1.
Importantly, the cost is invariant under perturbations to any single parameter of the ﬁlter and K bad ,being equivalent to the zero-ﬁlter, is suboptimal.
The same is true for any OE instance: every ﬁlterwith B bad = 0 , C bad = 0 , and A bad being stable is a suboptimal stationary point of L OE .
The perils of enforcing minimality.
A ﬁlter K is minimal if ( A K , B K ) is controllable, and( A K , C K ) is observable.
Example 3.1 is the extreme case of a non-minimal ﬁlter, since B bad =
C bad = 0 2 .
Conversely, as a special case of LQG , the OE problem inherits the property that allstationary points corresponding to minimal ﬁlters are globally optimal [Tang et al., 2021, Theorem4.3].
Therefore, it may be natural to ask: can a local search algorithm enforce minimality to avoidsuboptimal stationary points?
A classical result due to Brockett [1976] suggests not: the set ofminimal n -th order single-input-single-output transfer functions (e.g. ﬁlters) is the disjoint unionof n + 1 open sets.
Thus it is impossible for a continuous path to pass from one of these opensets to another without entering a region corresponding to a non-minimal ﬁlter, suggesting that alocal search algorithm regularized to ensure minimality at every iteration may never converge to theoptimal solution.
See Appendix F.2 for further discussion and supporting numerical experiments.
Futher challenges.
Given the drawbacks of enforcing minimality, one may wonder whether it issufﬁcient to enforce controllability alone.
In Example 3.1 above - and indeed, for all the examplesin Tang et al.
[2021] of suboptimal stationary points in the LQG landscape - there is a loss of bothobservability ( C bad = 0 ) and controllability ( B bad = 0 ).
Even controllable suboptimal stationarypoints exist, as shown in Proposition F.1.
Moreover, as argued in Appendix F.3, approaches basedon saddle-point escape with appropriate random perturbations [Jin et al., 2017] or acceleration [Jinet al., 2018] are also unlikely to be successful.
3.2 A provably convergent algorithm
The previous discussion puts us in a bind: we cannot regularize to preserve minimality because ofpath-disconnectedness.
Yet, controllability is not enough to rule out suboptimal stationary points.
We identify a stronger condition called informativity which is sufﬁcient.
We deﬁne the set of informative ﬁlters as K info := { K 2 K stab : rank( ⌃ 12 , K ) = n } .
First, we check that all optimal ﬁlters 228 lie in this set, under Assumption 2.4 (see Appendix E.3 for proof)
Lemma 3.1.
Under Assumptions 2.1 to 2.4, then K opt ⇢ K info ⇢ K ctrb , and K info is an open set.
Our key insight is that informativity is also sufﬁcient to ensure optimality of stationary points, butdoes not cause path-connectedness issues as it did for minimality (see Appendix G.3 for the proof).
Theorem 1.
Let K 2 K info ; then (i) there is a continuous path lying in K info connecting K to some 233 K ?
2 K opt and (ii) if r L OE ( K ) = 0 , then K 2 K opt .
Theorem 1 suggests that gradient descent with enforced informativity should converge to optimalﬁlters.
It is not, however, implied by the landscape analysis of Tang et al.
[2021], which focusessolely on minimal stationary points.
But numerous challenges remain: (1) how can one enforceinformativity in a smooth fashion?
(2) what quantitative measure of informativity provides quantitative suboptimality guarantees on approximate ﬁrst-order stationary points?
Given that L OE ( K )need not have compact level sets (see Appendix F.3), how does one ensure that the iterates of policysearch do not escape to inﬁnity, or reach regions where the loss of smoothness is arbitrarily poor?
The explained covariance matrix.
In light of Theorem 1, we design a policy search algorithmwhich ensures that ⌃ 12 , K remains full-rank throughout the search, but does so in a quantitativefashion.
Our central object is the explained covariance matrix , which measures how much of thecovariance of the steady-state system x ( t ) is explained by the internal ﬁlter state ˆ x K ( t ) in the large tlimit: Z K :=
lim t !
1 (Cov[ x ( t )]
E
[Cov[ x ( t ) | ˆ x K ( t )]]) .
When K 2 K ctrb , Z K admits an elegantclosed-form expression, which provides an alternative deﬁnition of K info :
[Equation]
so that
[Equation]
Since Z K is invariant under similarity transformations, Z K can be interpreted as a normalized analogue of ⌃ 12 , K .
Informally, the quadratic form v > Z K v is a sufﬁcient statistic for how much information ˆ x ( t ) contains about the “ v -direction” of x ( t ) ; see Appendix E.6 for a precise statement.
Explained-covariance regularization.
We preserve informativity by ensuring our iterates satisfy
Z K  0 .
To this end, we run gradient descent on the regularized objective for some  > 0 :
[Equation]
The proposed regularizer has several important properties.
First, though R info is nonconvex in
K , both R info and L OE can be made convex under the same change of variables, in the sense of
Fact 1.1, cf.
Lemma I.2.
In addition, R info is non-negative, and tends to 1 as Z K approachessingularity.
Furthermore, R info is invariant under similarity transformations.
Next, many of theessential quantities arising in our analysis can be bounded in terms of Z  1 K , justifying Z K is a naturalquantitative measure of informativity.
Lastly, the set of global-minimizers of R info ( · ) are preciselythe optimal ﬁlters for the OE problem, as per the following lemma (see Appendix E.3 for proof).
Lemma 3.2 (Existence of maximal Z K ) .
Under Assumptions 2.1 to 2.3, there exists a unique Z ?
0such that Z ?
=
Z K if and only if K 2 K opt , and Z ? ⌫ Z K for all K 2 K ctrb \ K opt .
Consequently,
K opt = arg min K 2 K ctrb R info ( K ) .
Lemma 3.2 directly implies that the suboptimality of L  ( · ) upper bounds the suboptimality in L OE ( · ) , 263 so we can minimize
L  as a proxy for minimizing L OE .
Corollary 3.1.
For any K , we have L OE ( K )  min K 0 L OE ( K 0 )
 L  ( K )  min K 0 L  ( K 0 ) .
Reconditioning.
We introduce an additional normalization step between policy updates to ensurethe iterates produced by our algorithm have well-conditioned covariance matrices; this in turn ensures the iterates produced by our algorithm remain in a compact set, and that the smoothness of L  e K t  ⌘ r is uniformly bounded.
For any ﬁlter K 2 K ctrb such that ⌃ 22 , K  0 , the reconditioning operatorrecond ( K ) returns a ﬁlter K 0 which is equivalent to K , but for which ⌃ 22 , K
= I n .
Formally 3 ,recond ( K ) :
= Sim S ( A K , B K , C K ) , where S = ⌃  1 / 2 22 , K .
Since K and K 0 = recond ( K ) are equivalent realizations, we have L OE ( K ) =
L OE ( K 0 ) , R info ( K ) =
R info ( K 0 ) , and thus L  ( K ) =
L  ( K 0 ) .
Statement of IR - PG.
We can now describe IR-PG, whose pseudocode is displayed in Algorithm 1.
IR-PG applies gradient descent on the regularized L OE objective, with an additional balancing step between gradient updates.
Appendix B.1 provides a variant where the step size is chosenby backtracking line-search, which enjoys the same rigorous convergence guarantees.
To guaranteeconvergence to an optimal ﬁlter (and ﬁniteness of L  ), we need to initialize at a ﬁlter such that
Z K 0  0 , i.e. K 0 2 K info .
Fortunately, random initializations from a continuous distribution satisfy 278 this condition with probability 1 (see Appendix E.7 for a formal statement and proof).
3.3 Formal guarantees
We conclude this section by stating the formal convergence guarantee for IR-PG.
Our results dependon natural problem quantities, among which is the minimum singular value of P ?
(as deﬁned in
Eq.
(2.3)), which we show is always strictly positive.
Lemma 3.3.
Let P ?
be the solution to the Riccati equation in Eq.
Then under Assumptions 2.1to 2.3,  ? :
=
min ( P ? ) is strictly positive.
Moreover, P ?
=
⌃ 11 , sys  Z ? .
In other words, P ? is the limiting conditional covariance of the true state x ( t ) given the policy-stateˆ x K ( t ) under (any) optimal policy.
Lemma 3.3 states that this covariance is nonsingular, i.e. not evenoptimal policies contain perfect information about any mode of x ( t ) .
Given an initialization K 0 , ourconvergence rate depends polynomially on the following problem parameters:
[Equation]
Theorem 2.
Fix  > 0 , K 0 2 K info .
There are terms C 1 , C 2  1 , which are at most polynomial in 290 n, m, C sys ,  ,   1 and L  ( K 0 ) , such that the iterates of IR-PG with any stepsize ⌘  1 C 1 satisfy
[Equation]
The formal guarantee for back-tracking stepsizes is nearly analogous, and given in Appendix B.1.
The oracle complexity of IR-PG is described in Appendix B.2.
We sketch the highlights of theproof in the following section, after introducing our DCL framework.
A rigorous proof overview isdeferred to Appendix G.2.
Analysis Framework
4.1 Differentiable convex liftings ( DCL
s)
This section introduces differentiable convex liftings ( DCL s), a rigorous and ﬂexible framework foroperationalizing convex reformulations of nonconvex objectives.
Preliminaries.
To neatly accommodate optimization over constrained domains, we express functions f : R d !
¯ R as taking values in the extended reals ¯ R :=
R [ { 1 } .
Given such an f , wedenote its domain dom ( f ) : { x : f ( z ) 6 = 1 } as the set on which f is ﬁnite; we say f is proper ifdom ( f ) 6 = ; ; we deﬁne its minimal value inf( f )
=
inf x 2 R d f ( x ) .
We say f 2 C k
( K )
on a if f isk -times continuously differentiable (and ﬁnite) on some open set containing K ⇢ R d .
DCL s .
The DCL is a generic template for convex reformulation that signiﬁcantly generalizes thesetup in Fact 1.1.
Rather than relating f directly to a convex f cvx , we “lift” f to a function f lft byappending auxiliary variables.
We then assume a reparametrization  mapping the domain of f lftto that of f cvx in this higher dimensional space; intuitively,  is the “inverse” of in Fact 1.1.
Deﬁnition 4.1.
A triplet of functions ( f cvx , f lft ,  ) is a DCL of a proper function f : R d !
¯ R if(1)
f cvx : R d z !
¯ R is a a proper convex function whose minimum is attained by some z ? . For some
additional number of parameters d ⇠  f lft : R d + d ⇠ ! ¯ R
is related to f via partial minimization: f ( x ) =
min ⇠ 2 R d ⇠ f lft ( x , ⇠ ) .
(3) There is an open set Y ◆ dom ( f lft ) for which  : Y !
dom ( f cvx ) is C 1 and satisﬁes f lft ( · ) =
f cvx (  ( · )) .
The mere existence of a DCL implies that approximate stationary points of f are also approximateminimizers, under conditions elaborated on below:
Theorem 3.
Let f : R d !
¯ R be a proper function with DCL ( f cvx , f lft ,  ) .
Then, for any x 2dom ( f ) at which f is differentiable, f satisﬁes the weak-PL condition:
[Equation]
where ↵ DCL ( x [Equation] =
[Equation]
Theorem 3 strengthens Fact 1.1 in two respects.
For one, it does not impose any smoothness restrictions on f lft or f cvx ; in particular f cvx can be highly non-smooth and, due to the extended-realfunction formulation, can also include constraints.
And second, the lifting f lft adds considerableﬂexibility, which we show is necessary to capture the convex reformulation of OE (Appendix I).
The factor ↵ DCL ( x ) depends on two quantities.
The numerator is the d z -th singular value ofr  ( x , ⇠ ) for any ⇠ 2 arg min f lft ( x , · ) .
This captures how large perturbations of f lft ’s argumentsmust be in order to achieve a desired perturbation of the arguments of f cvx , under the reparameterization  .
The additional arguments in f lft compared to f adds additional columns to r  therebymaking it easier to ensure  d z ( r  ( · )) > 0 .
On the other hand, the denominator measures the Euclidean distance between any minimizer of the convex function f cvx ( · ) and image of ( x , ⇠ ) under thereparameterization  , and can be bounded under quite benign conditions.
Proof Sketch.
The formalproof of Theorem 3 (given in Appendix H.2) takes special care to handle that allow f lft and f cvxto be ﬁnite only on restricted domains, and possible non-smoothness; still, the main ideas behindare intuitive.
For f cvx convex, f cvx ( z )  inf( f cvx ) =
O ( kr f cvx ( z ) k ) .
Using the DCL deﬁnitionand analyzing the inverse image of a point under  , we can also establish a gradient dominationresult for f lft .
Weak-PL result for the original function f follows since f is related to f lft by partialminimization, so its gradients must be larger than those of f lft .
Gradient descent with DCL s. We now describe how DCL s yield quantitative convergence guarantees for gradient descent.
A more general guarantee accommodating the reconditioning step in
IR-PG is deferred to Appendix G.1, and encompasses the bound below as a special case.
Given↵ > 0 , we say that proper f : R d !
¯ R satisﬁes ↵ - weak-PL (named after the stronger PolyakŁ ojasiewicz condition) on a domain K ⇢ R d if f 2 C 1 ( K ) and
kr f ( x ) k  ↵ ( f ( x )  inf( f ) .
340 From Theorem 3, we see f satisﬁes ↵
K -weak PL on K if f has a DCL and ↵ K :=
inf x 2 K ↵ DCL ( x ) > 341 0 .
To analyze gradient descent, we also require smoothness: we say f is  -upper-smooth on
K if 342 f 2 C 2 ( K ) and for all x 2 K , r 2 f ( x )   I .
The following follows from a standard descent 343 lemma for smooth (though possibly nonconvex) functions.
Proposition 4.1.
Let x 0 2 dom ( f ) , and suppose that the level set K ( x 0 ) :
= { x : f ( x )  f ( x 0 ) } 345 is compact, and f satisﬁes ↵ x 0 > 0 -weak PL and  x 0 > 0 -upper smoothness on K ( x ) .
Then, x 0lies in the same path-connected component of some minimizer of f , and for any ⌘  1 /  x 0 theupdates x k +1 =
x k  ⌘ r f ( x k ) satisfy f (
[Table]
4.2 Proof of Theorem 2.
In light of Proposition 4.1 (and its generalization to include reconditioning), it sufﬁces to establishboth the weak-PL and smoothness of the loss L  ( · ) on the well-conditioned sublevel set K 0 :
={ K : L  ( K )  L  ( K 0 ) , 12 I n  ⌃ 22 , K  2 I n } .
To establish weak-PL, we exhibit a DCL for which↵ DCL ( K ) depends only on the operator norms of ⌃ K , ⌃  1 K , Z K : as well as other system-quantities.
Here, poly op ( . . . ) denotes a quantity at most polynomial in the operator norms of its arguments.
Proposition 4.2.
For any   0 (non-strict), the objective L  ( K ) admits a DCL ( f cvx , f lft ,  )where the lifted parameter takes the form ( K , ⌃ K ) 2 K info ⇥ S 2 n ++ , L  ( K )
=
f lft ( K , ⌃ K ) = 356 min ⌃ 2 S 2 n + f lft ( K , ⌃ ) , and where
[Equation]
Furthermore, the norms of the parameters A K , B K , C K satisfy the following bounds:
[Equation]
Smoothness is established in (Proposition G.5), which relies on a novel (and rather challenging)bound on the solutions to Lyapunov equations involving A cl , K :
Proposition 4.3 (Stability of A cl , K ) .
Suppose that K 2 K info .
Then, for any matrix Y 2 S 2 n , thesolution ⌃ K , Y to the Lyapunov equation A cl , K ⌃ K , Y + ⌃ K , Y A > cl , K + Y = 0 satisﬁesk ⌃ K , Y k   C lyap ( K ) · k Y k  , where C lyap ( K ) =
poly op  ⌃ K , ⌃  1 K , Z  1 K , W  1 1 , W  1 2 , C  , and where k · k  denotes either the operator, Frobenius, or nuclear norm.
The regularization R info ( K ) ensures k Z  1 K k remains bounded on the sublevel K 0 ; we further show(Lemma G.7)
that 12 I n  ⌃ 22 , K  32 I n implies ⌃ K is invertible, and ensures k ⌃ K k , k ⌃
1 K k are 365 uniformly bounded.
Thus, the weak-PL constant and smoothness parameters are uniformly boundedon K 0 , concluding the proof.
We stress that the proofs of Proposition 4.2 and Proposition 4.3 requireseveral novel technical arguments, which may be of independent interest.
A full proof roadmap,including formal statements of the aforementioned results, is given in Appendix G.2.
The fact thatk ⌃ K k , k ⌃  1 K k ,
k Z  1 K k appear throughout the analysis suggests that (a) informativity as measured by
Z  1 K , and (b) the conditioning of ⌃ 22 , K may be fundamental to the OE landscape.
Proof of Theorem 1.
The proof of Theorem 1 is simpler, as it relies mainly on the existence of a
DCL for which ↵ DCL does not vanish.
See Appendix G.3 for details.
Conclusion
The work introduces the ﬁrst policy search algorithm which converges to the globally optimal dynamic ﬁlter for the output estimation problem.
We hope that our analysis serves as a valuable startingpoint to study direct policy search for reinforcement learning and control problems with partial observations, in which the relevant class of policies are dynamic and maintain an internal state.
Wealso hope that both our proposed principle of informativity, and our technical contributions aroundconvex reformulations, continue to prove useful in future work.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for informationon how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section.•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] See Section 2.(b)
Did you describe the limitations of your work?
[Yes] See Assumption 2.4 and Appendix D.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section
(b) Did you include complete proofs of all theoretical results?
[Yes] See appendices.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Code providedin supplementary materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix F.6.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Figs.
3 and 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See beginning of Appendix F.6.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data
you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁ-
able information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]