E NSEMBLE K ALMAN F ILTER (E N KF) FOR R EINFORCEMENT L EARNING (RL)
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper is concerned with representing and learning the optimal control law for the linear quadratic Gaussian (LQG) optimal control problem.
In recent years, there is a growing interest in re-visiting this classical problem, in part due to the successes of reinforcement learning (RL).
The main question of this body of research (and also of our paper) is to approximate the optimal control law without explicitly solving the Riccati equation.
For this purpose, a novel simulation-based algorithm, namely an ensemble Kalman ﬁlter (EnKF), is introduced in this paper.
The algorithm is used to obtain formulae for optimal control, expressed entirely in terms of the EnKF particles.
For the general partially observed LQG problem, the proposed EnKF is combined with a standard EnKF (for the estimation problem) to obtain the optimal control input based on the use of the separation principle.
The theoretical results and algorithms are illustrated with numerical experiments.
I NTRODUCTION
This paper is concerned with the problem of reinforcement learning (RL) in continuous-time and continuous (Euclidean) state-space settings.
A special case is the linear quadratic Gaussian (LQG) problem where the dynamic model is a linear system, the cost terms are quadratic, and the distributions – of the random initial condition and the noise – are Gaussian.
The LQG problem has a rich and storied history in modern control theory going back to the very origin of the subject (Kalman, 1960).
To obtain the optimal control, the bottleneck is to solve the
Riccati equation – the differential Riccati equation (DRE) in ﬁnite-horizon settings or the algebraic
Riccati equation (ARE) in the inﬁnite-horizon settings of the problem.
There is a body of literature devoted to the study of these equations (Bittanti et al., 2012; Lancaster & Rodman, 1995) with specialized numerical techniques to compute the solution (Laub, 1991; Benner & Bujanovi ´ c, 2016).
There are two issues which makes the LQG and related problems a topic of recent research interest: (i) In high-dimensions, the matrix-valued nature of the DRE or ARE means that any algorithm is
O ( d 2 ) in the dimension d of the state-space; and (ii) the model parameters may not be explicitly available to write down the DRE (or the ARE) let alone solve it.
The latter is a concern, e.g., when the model exists only in the form of a black-box numerical simulator.
These two issues have motivated the recent research on the inﬁnite-horizon linear quadratic regulator (LQR) problem (Fazel et al., 2018; Mohammadi et al., 2021b;
Tu & Recht, 2019).
In LQR, the bottleneck is to solve an ARE.
The algorithms studied in the recent papers seek to bypass solving an ARE by directly searching over the space of stabilizing gain matrices.
Global convergence rate estimates are established for both discrete-time (Fazel et al., 2018;
Dean et al., 2020; Malik et al., 2020; Mohammadi et al., 2021a) and continuous-time (Mohammadi et al., 2020a;b; 2019; 2021b) settings of the LQR problem.
Extensions to the H ∞ regularized LQR (Zhang et al., 2020) and
Markov jump linear systems (Jansch-Porto et al., 2020) have also been carried out.
The motivation and goals of the present work are related to these recent papers, albeit the proposed solution approach is very different.
Our inspiration comes from data assimilation (Reich & Cotter, 2015), where in practical applications, e.g., in weather prediction, (i) only simulation-based models are available, and (ii) these are very high-dimensional.
The ensemble Kalman ﬁlter (EnKF) is an efﬁcient simulation-based algorithm to assimilate sensor data in these applications without the need to explicitly solve a DRE (Evensen, 1994; 2006; Houtekamer & Mitchell, 2001).
Our contribution:
A novel extension of the EnKF algorithm is proposed to learn the optimal control law for the general (stochastic and partially observed) setting of the LQG optimal control problem.
The proposed algorithm is simulation-based (using particles) and, in particular, avoids the need to solve the DRE.
Assuming full-state feedback (e.g, setting of the LQR problem), the optimal control law is directly obtained.
For the partially observed case, the proposed EnKF is combined with a standard EnKF (for the estimation problem) to obtain the optimal control input, based on the use of the separation principle.
The algorithm is shown to be exact in its mean-ﬁeld (when the number of particles N = ∞ ) limit (Prop. 2).
For the ﬁnite- N approximation, an error bound is obtained (Eq. (14)).
An extensive discussion is included to provide an intuitive explanation of the algorithm, situate the algorithm in the RL landscape (see discussion after Eq.
(12) in Sec. 2.2), and to compare and contrast the algorithm with the state-of-the-art (Sec. 2.5).
For this purpose, numerical comparisons with competing algorithms are also included (Fig.
2 (c)).
The outline of the remainder of this paper is as follows.
The LQG optimal control problem and its simulation-based solution is described in Sec. 2.
The algorithms are illustrated with numerical examples in Sec.
3 where comparisons with the state-of-the-art algorithms are also described.
L INEAR Q UADRATIC G AUSSIAN (LQG) PROBLEM
Problem: The partially observed linear Gaussian model is expressed using the It ˆ o-stochastic differential equations (SDE) as
[Equation]
(1a)
[Equation]
(1b) where X = { X t ∈ R d : t ≥ 0 } is the hidden state process,
U
=
{ U t ∈ R m : t ≥ 0 } is the control input, and Z = { Z t ∈ R p : t ≥ 0 } is the observation process.
The model parameters A, B, H are matrices of appropriate dimensions, the process noise ξ = { ξ t ∈ R d : t ≥ 0 } and the observation noise ζ = { ζ t ∈ R p : t ≥ 0 } are Wiener processes (w.p.) with covariance Q and R , respectively.
It is assumed that X 0 , ξ and ζ are mutually independent and R (cid:31) 0 .
The LQG optimal control objective is to minimize
[Equation]
It is assumed that ( A, B ) is controllable, ( A, C ) and ( A, H ) are observable, and the matrices P T (cid:31)and R (cid:31) 0 .
In the fully observed settings, U t is allowed to be a function of X t .
In the partially observed settings, U t is a function of the past observations { Z s : 0 ≤ s ≤ t } .
For this purpose, it is convenient to denote
[Equation]
) as the sigma-ﬁeld generated by the observations,
and consider control inputs U which are adapted to the ﬁltration
[Equation]
Classical solution: Using the seperation principle 1 the LQG controller is obtained in three steps:
Step 1.
Filter design: The objective of the ﬁlter design step is to compute the causal estimate ˆ X t := E ( X t |Z t ) for t ≥ 0 .
The evolution equation for { ˆ X t : t ≥ 0 } is the Kalman-Bucy ﬁlter.
Notably, the optimal gain matrix for the ﬁlter is obtained by solving a forward (in time) DRE.
Step 2.
Control design: The objective of the control design step is to compute the optimal feedback control law { u t ( x ) : 0 ≤ t ≤ T, x ∈ R d } for the fully observed LQG problem.
It is well known to be of the linear form
[Equation]
where the optimal gain matrix K t is obtained by solving a backward (in time) DRE.
Step 3.
Certainty equivalence:
The optimal control input is obtained by combining the results from steps 1 and 2:
[Equation]
The bottleneck is to solve the DREs – the forward DRE for the optimal ﬁlter gain and the backward
DRE for the optimal control gain 2 .
In the following three sections, we describe a simulation-based algorithm for these three steps which avoids the need to explicitly solve the DREs.
2. S TEP 1. F ILTER DESIGN USING E N KF
The ﬁlter design objective is to compute the causal estimate ˆ X t =
E ( X t |Z t ) .
In the linear Gaussian settings, the conditional distribution of the X t is Gaussian whose mean and variance are denoted as m t and Σ t , respectively.
These evolve according to the Kalman-Bucy ﬁlter:
[Equation]
where L t = Σ t H (cid:62) R − 1 is the Kalman gain.
Note that (3b) is a forward (in time) DRE and its solution Σ t is used to compute the optimal Kalman gain L t .
The EnKF is a simulation-based algorithm to approximate the Kalman ﬁlter, that does not require an explicit solution of the DRE (3b).
The design of an EnKF proceeds in two steps:
Construct a stochastic process, denoted by ¯ X :=
{ ¯ X t ∈ R d : t ≥ 0 } , such that the conditional distribution (given Z t ) of ¯ X t is equal to the conditional distribution of X t ; 2.
Simulate N stochastic processes, denoted by { X it : t ≥ 0 , 1 ≤ i ≤ N } , to empirically approxi- mate the distribution of ¯ X t .
The process ¯ X is referred to as the mean-ﬁeld process.
The N processes in the step 2 are referred to as particles.
The construction ensures that the EnKF is exact in the mean-ﬁeld ( N = ∞ ) limit.
That is, for any bounded and continuous function f ,
[Equation]
The details of the two steps are as follows:
Mean-ﬁeld process: The mean-ﬁeld process is constructed as
[Equation]
where ¯ ξ := { ¯ ξ t : t ≥ 0 } is an independent copy of the process noise ξ , ¯ L t :=
¯Σ t H (cid:62) R − 1 is the
Kalman gain and
[Equation]
are the conditional mean and the conditional covariance, respectively, of ¯ X t .
The right-hand side of (4) depends upon both the process ( ¯ X t ) as well as the statistics of the process ( ¯ m t , ¯Σ t ).
Such an
SDE is an example of a McKean-Vlasov SDE.
The proof of the following proposition is included in the Appendix A (see also Taghvaei & Mehta (2020, Theorem 1)).
Proposition 1 (Exactness of EnKF) .
Consider the mean-ﬁeld EnKF (4) initialized with a Gaussian initial condition ¯ X 0 ∼ N ( m 0 , Σ 0 ) .
Suppose the control input U is a Z -adapted stochastic process.
Then its solution ¯ X t is a Gaussian random variable whose conditional mean and variance
[Equation]
evolve the same as the Kalman ﬁlter (3) .
Finite- N approximation: The mean-ﬁeld process is simulated as an interacting particle system:
[Equation]
where ˆ X ( N ) t
[Equation]
X it is the empirical mean and
[Equation]
is the empirical approximation of the optimal Kalman gain matrix.
The system (5) is referred to as the square root form of the EnKF (Bergemann & Reich, 2012, Eq (3.3)).
Note that the gain is approximated entirely in terms of particles without solving the DRE (3b).
The EnKF (5) is an example of a simulation-based algorithm in the sense that N copies of the model (1a) are simulated in parallel.
The simulations are coupled through a term which is referred to as the data assimilation step.
This term has a gain times error feedback control structure.
2. S TEP 2. C ONTROL DESIGN USING E N KF
This section contains the main contribution of this paper.
Assuming full-state feedback, the optimal control law for the LQG problem (1a)-(2) is
[Equation]
where
[Equation]
t
[Equation]
P t is the optimal gain matrix and P t is a solution of the backward (in time) DRE
[Equation]
It is known that P t (cid:31) 0 for 0 ≤ t ≤ T whenever P T (cid:31)
(Brockett, 2015, Sec. 24).
Therefore, S t =
P − 1 t is well-deﬁned.
It is straightforward to verify that S t also solves a backward DRE
[Equation]
The bottleneck is to solve the DRE (7).
In the following, an EnKF is proposed to obtain a simulationbased approximation of the optimal control law u t .
As before, the construction proceeds in two steps: (i) deﬁnition of an exact mean-ﬁeld process and (ii) its ﬁnite- N approximation.
Mean-ﬁeld process: The objective is to construct a stochastic process, denoted ¯ Y t ∈ R d at time t , whose variance equals P t , the solution of the DRE (7).
This is done by constructing ¯ Y =
{ ¯ Y t ∈ R d :≤ t ≤ T } as a solution of the following backward (in time)
McKean-Vlasov SDE:
[Equation]
and deﬁning
[Equation]
where ¯ η = { ¯ η t ∈ R m : t ≥ 0 } is a w.p. with covariance matrix R − 1 , d (cid:0) ¯ η in (9) denotes the backward Itˆo-integral (Nualart & Pardoux, 1988, Sec. 4.2), and
[Equation]
The proof of the following proposition is included in Appendix B.
Proposition 2. Consider the mean-ﬁeld EnKF (9) initialized with a Gaussian initial condition ¯ Y T ∼
N (0 , S T ) .
Then its solution ¯ Y t is a Gaussian random variable whose mean and variance
[Equation]
Consequently, ¯ Y t is also a Gaussian random variable with mean and variance
[Equation]
and therefore (i)
If the matrix B is explicitly known then the optimal gain matrix
[Equation]
or else (when it is not) then (ii) deﬁne the Hamiltonian 3 (or the Q-function)
(Mehta & Meyn, 2009)
[Equation]
from which the optimal control law is obtained as u t ( x ) =
arg min a ∈ R m H ( x, a, t ) .
Finite- N approximation: The mean-ﬁeld process is empirically approximated by simulating a system of interacting particles {Y it ∈
R d : 0 ≤ t ≤ T, i = 1 , . . .
, N } according to
[Equation]
where η i is a copy of ¯ η , and n ( N ) t
[Equation]
N
[Equation] [Equation]
Y it
[Equation]
Relationship to RL: The following intuitive explanation is included to situate the proposed simulation-based algorithm in the RL landscape:
Representation:
In designing any RL algorithm, the ﬁrst issue is representation – how does one represent the unknown value function ( P t in the linear case)?
Our novel idea – the ﬁrst key innovation of this paper – is to represent P t is in terms of statistics (variance) of the particles.
Such a representation is fundamentally distinct from representing the value function, or its proxies, such as the Q function, in terms of a set of basis functions (Bradtke et al., 1994; Devraj et al., 2020; Maei et al., 2010; Fujimoto et al., 2018; Melo et al., 2008).
The algorithm is entirely simulation based: N copies of the model (1a) are simulated in parallel where the terms on the righthand-side of (11) have the following intuitive explanation:
Dynamics:
The ﬁrst term “ A Y it d t ” on the right-hand side of (11) is simply a copy of uncontrolled dynamics in the model (1a).
Control:
The second term is the control input “ BU t d t ” implemented as “ B d η it ”.
That is, the control input U for the
i -th particle is a white noise process with covariance R − 1 .
One may interpret this as an exploration step whereby the cheaper control directions are explored more.
In summary, for the i -th particle, the dynamics and control are the same as any RL algorithm (white noise is used for exploration).
The difference arises due to the third term (which is the second key innovation of this paper).
RL step: The third term indicated as the RL step engenders a particle ﬂow that effectively implements the value iteration step of the RL.
There are several points to be made:
The RL step is a function of the state cost term in (2).
This is most easily seen by writing the equation for the empirical mean
[Equation]
where η ( N ) t = N − 1 (cid:80) i η it .
Noting that the state cost term is x (cid:62) C (cid:62) Cx , the RL step implements a gradient with S ( N ) t
C (cid:62) as the gain matrix.
The RL step has a linear feedback control structure and serves to couple the particles.
Without the RL step, the particles are independent of each other.
This form of the RL step is possible only if one has access to a simulator and an ability to add additional terms outside the control channel (same as the data assimilation step in estimation).
For example, this is not possible when the system exists only as an experiment
Arrow of simulation time: The particles are simulated backward – from terminal time t
=
T to initial time t = 0 .
This is consistent with the dynamic programming (DP) equation which also proceeds backward in time.
However, because the focus of the RL is on the inﬁnite-horizon problem, this important property of DP is not considered in algorithm design.
It is noted that the inﬁnite-horizon problem is easily handled in our formulation by taking T as suitably large (which is how the inﬁnitehorizon limit is deﬁned in DP).
It remains to use the ﬁnite- N system to approximate the optimal control law.
Optimal control: There are two cases to consider: (i)
If the matrix B is explicitly known then
[Equation]
(13) or else (when it is not) then (ii) approximate the Hamiltonian as
[Equation]
from which the optimal control law is obtained as
[Equation]
There are several zeroth-order approaches to solve the minimization problem, e.g., by constructing 2-point estimators for the gradient.
Since the objective function is quadratic and the matrix R is known, m queries of H ( N ) ( x, · , t ) are sufﬁcient to compute u ( N ) t ( x ) .
2. E N KF- BASED ALGORITHM FOR THE LQG CONTROLLER
By combining the result of steps 1 and 2, the optimal control input
[Equation]
The overall algorithm including steps 1, 2 and 3 is tabulated in the Appendix C where additional remarks on numerical approximation of backward SDEs also appear.
2. C ONVERGENCE AND ERROR ANALYSIS
The mean-ﬁeld process (9) represents the mean-ﬁeld limit of the ﬁnite-
N system (11), as the number of particles N → ∞ .
The convergence analysis is a delicate matter based on the propagation of chaos (Bishop & Del Moral, 2018).
In Appendix D, under certain additional assumptions on system matrices, the following error bound is derived:
[Equation]
(14) where µ and c var are positive constants and (cid:107) · (cid:107) F is the Frobenius (matrix) norm.
The computational complexity of an ENKF is O ( Nd ) .
EnKF is a workhorse in applications such as weather prediction where models are simulation-based (Evensen, 2006; Reich & Cotter, 2015).
In these applications, the number of simulations N << d .
2. C OMPARISON WITH RELATED WORK
Classical RL algorithms for the LQR problem are based on a linear function approximation, using quadratic basis functions, of the value function or its surrogate the Q-function (Bradtke et al., 1994).
Convergence guarantees typically require (i) a persistence of excitation condition, and (ii) use of the on policy methods whereby the parameters are learned for a given ﬁxed policy (which is subsequently improved).
For the deterministic LQR problem, the persistence of excitation condition is difﬁcult to justify theoretically, and in practice can lead to poor performance related to slow convergence rates.
These limitations have spurred recent research on the LQR problem.
Our algorithm is compared to the model-free policy optimization based methods of Mohammadi et al.
(2021a)
and Fazel et al.
In policy optimization, the value J ( U ) is minimized over the search space of stabilizing gain matrices, using a gradient-descent approach, starting from a stabilizing controller (which is not straightforward to obtain in a model-free setting).
For each perturbation of the gain matrix, the gradient is estimated by simulating N particles (trajectories) over a time-horizon [0 , T ] .
Multiple iterations are needed to converge to the minimizer.
In contrast, using the EnKF, the matrix P t is approximated by simulating N particles once over the time-horizon [0 , T ] .
The optimal control is obtained from an application of the minimum principle that only requires m evaluations at each t (the minimization of Hamiltonian is an online calculation).
The trade-off between the two approaches is as follows: While policy optimization methods require multiple iterations with a small number of particles, the EnKF requires only a single iteration with relatively larger number of particles.
As illustrated with the aid of numerical examples in Sec. 3, this can lead to an order of magnitude gain in the computation time.
As a ﬁnal point, our paper provides a simulation-based algorithm for the most general class of linear quadratic problem: fully or partially observed, ﬁnite or inﬁnite-horizon, stochastic or deterministic, and furthermore does not require an initial stabilizing controller.
Duality: In Appendix E, it is shown that the Gaussian density of the random variable ¯ Y t equals a log transformation of the value function (at time t ) of the LQG optimal control problem.
Additional remarks are also included to expound the duality roots of the proposed algorithm, including a survey of the relevant literature on this beautiful subject.
N UMERICAL S IMULATIONS
In all the following three numerical examples, we consider the inﬁnite-horizon fully observed deterministic LQR problem.
Although our algorithms are more generally applied, such a choice allows us to compare and contrast with the recent RL work on the LQR problem.
For the LQR problem, the solution of the ARE is denoted P ∞ and the associated optimal feedback gain is denoted K ∞ .
In each of the three numerical examples, a time-horizon T is ﬁxed.
Over the time-horizon, the ﬁniteN EnKF algorithm (11) is run to obtain an empirical approximation { P ( N ) t ∈ R d × d : 0 ≤ t ≤ T } .
For the sake of comparison, the exact { P t ∈ R d × d : 0 ≤ t ≤ T } is obtained by numerically solving the backward DRE (7).
Typically, we chose P T
=
I , the identity matrix.
3. E XPONENTIAL CONVERGENCE
An attractive feature of the proposed EnKF algorithm is that the mean-ﬁeld limit is exact.
This means the EnKF recovers the properties of the DRE in the limit.
One such property is the exponential convergence: For any ﬁxed t , P t → P ∞ as T → ∞ , starting from any initialization P T (Ocone & Pardoux, 1996, Remark 2.1).
Moreover, because of the error formula (14), P ( N ) t → P ∞ with a small additional bias that decays as O ( 1 N ) .
For a d = 2 dimensional system, Fig. 1 depicts this exponential convergence of the four entries of the symmetric P ( N ) t matrix (using N = 100 particles).
As part of Appendix F.1, additional ﬁgures are included to depict exponential convergence of the 100 entries of the 10 × 10 matrix P ( N ) t for a d = 10 dimensional system (using N = 1000 particles).
In these evaluations, the system ( A, B ) is chosen to be in its controllable canonical form where the appropriate entries (last row) of the A matrix are randomly generated.
Because these are randomly generated, the matrix A is
− 0 .0 .
0 (a) − 0 . [Figure] − 0 .0 .0 .0 .
OL
CL  8 (b) 1 .1 .1 .1 .1 .
P
EnKF
DRE
ARE  8 (c) 1 .1 .1 .1 .2 .2 .
P
T − t 8 (d) 0 .0 .0 .0 .0 .1 .
Im
Re
P 21 =
P often unstable (see also Fig.
1 (a)) which does not pose any problem with our algorithm.
Additional details on the numerics appear as part of Appendix F.1.
3. E VALUATION AND COMPARISON WITH LITERATURE : M ASS SPRING DAMPER SYSTEM
In order to evaluate and compare the performance of the proposed algorithm, we describe next the results for the coupled mass-spring system studied in Mohammadi et al.
For each mass, there are two states, position and velocity.
A system with d/ 2 masses is d -dimensional.
To evaluate the performance of the ﬁnite- N algorithm, we consider the following error metric:
[Equation]
The expectation is approximated empirically by averaging over 100 simulation runs.
See Appendix
F.2 for details on modeling, parameter values, and the numerical discretization.
Fig.
2 depicts the results of the numerical experiments showing the O ( 1 N ) decay of MSE as N increases (for d ﬁxed).
Part (c) of the ﬁgure depicts a comparison with the two state-of-the-art algorithms, namely Mohammadi et al.
(2021b)
and Fazel et al.
In the ﬁgure, computation time is plotted against the relative error in approximating the LQR gain K ∞ (plots are qualitatively similar using other error metrics).
The computation time is obtained using the Python process time() function from the time library for measuring execution time.
The result depicted in the ﬁgure is for d = 10 but qualitatively similar results are also obtained for other values of d .
See Appendix F.3 for details on numerical implementation of the algorithms.
We also implemented the classical RL algorithm in Bradtke et al.
but its convergence was not reliable because of the persistence of excitation issues.
[Figure]
the relative error in approximating the LQR gain K ∞ .
Based on the preliminary analysis, the main reason for the order of magnitude improvement in computational time is as follows:
An EnKF requires only a single iteration over a ﬁxed time-horizon [0 , T ] .
Although the number of particles ( N ) is an order of magnitude larger for the EnKF algorithm, certain vectorisation features of numpy yield signiﬁcant gains in computational time.
Because the other algorithms require multiple iterations which must necessarily be carried out serially, these computations are slower.
In our comparisons, we used the same time-horizon T and discretization time-step δt for all the algorithms.
It is certainly possible that some of these parameters can be optimized to improve the performance of the other algorithms.
In particular, one may consider shorter or longer time-horizon T or use parallelization to speed up the gradient calculation.
3. E VALUATION OF E NK F FOR A NONLINEAR CART - POLE SYSTEM
It is a time-honored practice to design the optimal control law for a (linearized) LQR model and then implement this control law on a nonlinear system.
By the principle of linearization, the control law works well (in theory) for small perturbations and often (in practice) for even large ones.
We consider the nonlinear conservative cart pole model (Tedrake; Rawlik et al., 2013).
The control acts as external force applied to the cart.
The four-dimensional state for the system is ( θ, x, ω, v ) , where θ ∈ S 1 (the circle) is the angle of the pole (pendulum) as measured from the stable equilibrium, x ∈ R is the displacement of cart along the horizontal, and ( ω, v ) :
= ( ˙ θ, ˙ x ) ∈ R 2 is the velocity vector.
The control objective is to balance the pole – stabilize the system at the inverted equilibrium ( π, 0 , 0 , 0) , assuming full state feedback.
For the purposes of control design, the nonlinear system is ﬁrst linearized at the desired equilibrium and an LQR problem is formulated (see Appendix F. for details).
For the purposes of evaluation, the optimal control is applied to the nonlinear model.
Figure 3 depict the results of numerical experiments for three different choices of N .
It is seen that as few as N = 10 particles are sufﬁcient to stabilize the equilibrium.
Using N = 1000 particles, the closed-loop trajectories are virtually indistinguishable from the DRE-based solution.
C ONCLUSIONS
In recent years, there has been a concerted effort to revisit the LQR problem in the context of
RL (Fazel et al., 2018; Mohammadi et al., 2021b; Tu & Recht, 2019; Dean et al., 2020; Malik et al., 2020; Zhang et al., 2020; Jansch-Porto et al., 2020).
The effort is largely in response to the well known issues that arise when the state and action space are continuous (Lu et al., 2021).
In this paper, we present a new paradigm for RL.
There are two key innovations: (i) the representation of the unknown value function in terms of the statistics (variance) of the particles; and (ii) design of interactions between simulations to solve the optimal control problem.
For the LQR problem, this is shown to yield a learning rate that closely approximates the exponential rate of convergence of the solution of the DRE.
In numerical examples, this property is shown to lead to an order of magnitude better performance than the state-of-the-art algorithms.
Given the enormous success of EnKF in data assimilation (Evensen, 2006; Reich & Cotter, 2015), the contributions of this paper potentially open up new opportunities for RL.
It is our hope that the paper will engender new synergies between the data assimilation and the RL communities.
R EPRODUCIBILITY STATEMENT
The implementable pseudo-code of the algorithm is tabulated in Appendix C. For the three examples in Sec.
3, the numerical parameters are tabulated in Appendix
F.
The codes for all numerical simulations are provided as part of the supplementary material.
All the codes have a README ﬁle associated with it.
Table 1 provides a list of the Appendix containing the simulation parameters and location of codes for each numerical example
[Table]
APPENDIX A P ROOF OF P ROP .
It is ﬁrst shown that the conditional mean ¯ m t and the conditional covariance ¯Σ t evolve according to the Kalman ﬁlter equations (3).
The SDE for the mean is obtained by taking the conditional expectation of (4),
[Equation]
where we used the fact that U t is assumed to be Z t measurable.
To obtain the equation for the covariance, deﬁne the error process e t
= ¯ X t − ¯ m t which evolves according to
[Equation]
Then, upon an application of the It ˆ o rule
[Equation]
and taking the conditional expectation, the equation for the conditional covariance ¯Σ t
= E [ e [Equation] e (cid:62) t |Z t ] is obtained as
[Equation]
where we used the deﬁnition ¯ L t =
¯Σ t H (cid:62) .
The equation for the conditional covariance ¯Σ t is identical to the DRE (3b).
Therefore, if the initial condition ¯Σ
= Σ 0 then ¯Σ t = Σ t for all t > 0 .
This also implies ¯ L t =
L t , which in turn implies that the SDE for the conditional mean ¯ m t is identical to Kalman ﬁlter equation for m t .
If ¯ m
= mthen ¯ m t = m t for all t > 0 .
By replacing the mean-ﬁeld terms ¯ m t and ¯Σ t by exogenous processes m t and Σ t , the McKeanVlasov SDE (4) simpliﬁes to an Ornstein-Uhlenbeck SDE.
Because the distribution of the initial condition ¯ X 0 is Gaussian, the distribution of ¯ X t is also Gaussian and given by N ( m t , Σ t ) .
B P ROOF OF P ROP .
The proof is similar to the proof of Prop. 1.
The equation for the mean ¯ n t is obtained by taking the expectation of SDE (9),
[Equation]
Because ¯ n T = 0 , we have ¯ n t = 0 for all t ∈ [0 , T ] .
The equation for the covariance ¯ S t is obtained by writing the SDE for the error e t :
= ¯ Y t − ¯ n t :
[Equation]
Using the Itˆo rule for e t e (cid:62) t ,
[Equation]
The Itˆo correction term appears with a negative sign because the SDE involves a backward Wiener process (cid:0) ¯ η t (Nualart & Pardoux, 1988, Sec. 4.2).
Taking an expectation yields the following equation for ¯ S t :
[Equation]
The SDE is identical to the SDE for S t .
Because ¯ S T = S T , we have ¯ S t =
S t for all t ∈ [0 , T ] .
The conclusion that ¯ Y t is Gaussian follows from the fact that with ¯ n t = n t and ¯ S t = S t , the SDE for ¯ Y t is an Ornstein-Uhlenbeck SDE with a Gaussian terminal condition.
The proof for the rest of proposition is straightforward.
By deﬁnition,
[Equation]
Since E [ ¯ Y t ¯ Y (cid:62) t ] = P t , the optimal gain matrix K t = − R − 1 B (cid:62) E
[ ¯ Y t ¯ Y (cid:62) t ] .
For a given x ∈ R d , the optimal control u t ( x ) =
K t x = − R − 1 B (cid:62) E
[ ¯ Y t ¯ Y (cid:62) t ] x is the unique minimizer of the Hamiltonian.
This is referred to as the minimum principle of optimal control.
C D ETAILS OF THE ALGORITHMS TO SOLVE THE PARTIALLY OBSERVED LQG PROBLEM IN S EC .
In this section, the implementation details of the EnKF-based algorithms are presented to numerically solve the partially observed LQG problem (1)-(2) introduced in the main body of the paper.
For better readability, the overall algorithm is broken down into three separate algorithms: 1.
Algorithm 1 is an ofﬂine algorithm.
It is based on the ﬁnite- N approximation of the (control) EnKF as described in Sec.
2.2 of the main body of the paper.
The algorithm is run ofﬂine to obtain an approximation of the { P t : 0 ≤ t ≤ T } .
Algorithm 2 is an online algorithm.
It is based on the ﬁnite- N approximation of the (ﬁlter) EnKF as described in Sec.
2.1 of the main body of the paper.
The algorithm is run online in a real-time manner.
At each time step, it processes the sensor measurements from the true system (plant) and computes the optimal control input by calling Algorithm 3. 3.
Algorithm 3 computes the optimal control input.
It is based upon minimization of the Hamiltonian function.
In an online implementation, the optimal control input is applied to the plant at each time step.
The input structure of each of the three algorithms is clearly delineated.
In particular, the algorithms require only the simulator in the form of the function evaluator f ( x, u ) =
Ax + Bu for the dynamics, c ( x ) =
Cx for the cost function, and h ( x ) =
Hx for the observation function.
The algorithms do not require solution of the DRE.
Remark 1.
In a numerical implementation of step 1 and step 2, there are two sources of error: (i) error on account of ﬁnite-
N approximation; and (ii) error on account of time discretization which depends upon the step size ∆ t .
The ﬁnite- N approximation error has been studied in the
EnKF literature where it is shown that the error in approximating (cid:107) P t − P ( N ) t (cid:107) (in step 1) and the error in approximating the gain the mean (cid:107)
ˆ X t −
ˆ X
( N ) t (cid:107) (in step 2) converges to zero with the rate
O ( 1 √ N ) Del Moral & Tugaut (2018); Bishop et al.
; Bishop & Del Moral (2018); Taghvaei & Mehta (2018);
Taghvaei (2019).
The time discretization error is also expected to be bounded and converges to zero with the rate O (∆ t ) when the system is stable Kloeden & Platen (1999).
Input: Simulation time T , simulation step-size ∆ t , number of particles N , simulator f ( x, u ) = Ax + Bu , terminal cost P T , cost function c ( x ) = Cx , and control cost matrix R .
Output: { P ( N ) k : k = 0 , 1 , 2 , . . .
, T ∆ t } .
1: T F = T ∆ t 2: P ( N ) T F = P T 3: Initialize {Y iT F } Ni =1 i.i.d ∼ N (0 , P − 1 T ) 4: calculate n ( N ) T F = N − 1 (cid:80) Ni
=
Y iT F 5: for k = T F to 1 do 6:
Calculate ˆ c ( N ) k
= N − 1 (cid:80) Ni
=1 c ( Y ik ) 7:
Calculate M ( N ) k
= ( N − 1) − 1 (cid:80) Ni =
( Y ik − n ( N ) k )( c ( Y ik ) − ˆ c ( N ) k ) (cid:62) 8: for i = 1 : N do 9:
∆ η ik i.i.d ∼ N (0 , 1∆ t R − 1 ) 10: ∆ Y ik = f ( Y ik , ∆ η ik )∆ t + 12 M ( N ) k
( c ( Y ik ) + ˆ c ( N ) k )
∆ t 11:
Y ik − 1 = Y
ik − ∆ Y ik 12: end for 13:
Calculate n ( N ) k − 1 = N − 1 (cid:80) Ni
=1 Y ik −14:
Calculate S ( N ) k − 1 =
( N − 1) − 1 (cid:80)
Ni =
( Y ik − 1 − n ( N ) k − 1 )
( Y ik − 1 − n ( N ) k − 1 ) (cid:62) 15:
P ( N ) k − 1 = ( S ( N ) k − 1 ) −16: end for algorithm to approximate state estimate ˆ X t and optimal control u t
Input: Simulation time T , simulation step-size ∆ t , number of particles N , simulator f ( x, u ) =
Ax + Bu , initial distribution N ( m 0 , Σ 0 ) , process noise covariance R , observation function h ( x ) =
Hx , { P ( N ) k : k = 0 , 1 , 2 , . . .
, T ∆ t } from the ofﬂine algorithm F.1.
Output: estimate { ˆ X ( N ) k : k = 0 , 1 , 2 , . .
, T ∆ t } and optimal control input { u ( N ) k : k = 0 , 1 , 2 , . . .
, T ∆ t − 1 } .
1: Deﬁne T F := T ∆ t 2: Initialize particles { X i 0 } Ni =1 i.i.d ∼ N ( m 0 , Σ 0 ) 3: for k = 0 to T F − 1 do 4:
Calculate ˆ X ( N ) k
= 1 N (cid:80) Ni =1 X ik 5:
Calculate u ( N ) k
= arg min a H ( ˆ X ( N ) k , P ( N ) k
ˆ X ( N ) k , a ) from algorithm6:
Apply control u ( N ) k to the true system and obtain the observation ∆ Z k = Z ( k +1)∆ t − Z k ∆ t 7:
Calculate ˆ h ( N ) k
= 1 N (cid:80) Ni
=1 h ( X ik ) 8:
Calculate L ( N ) k
= {+1 N − 1 (cid:80) Ni =
+}
( X ik − ˆ X
( N ) k )
( h ( X ik ) − ˆ h ( N ) k ) (cid:62) R −9: for i = 1 to N do 10:
∆ ξ ik i.i.d ∼ N (0 , Q ∆ t ) 11:
∆ X i k = f ( X i k , u ( N ) k ) ∆ t + ∆ ξ i k + L ( N ) k (∆ Z k − h
( X ik )+ˆ h ( N ) k∆ t ) 12:[-X ik +
-]
[Equation] 13: end for 14: end for
Input: state x , momentum y , control cost matrix R ,
Hamiltonian H ( x, y, α ) = y T ( a ( x ) + b ( x )
[Equation]
+
[Equation]
c ( x )
[Equation]
+
[Equation]
(cid:62)
[Equation]
Output: optimal control u = arg min α H ( x, y, α ) . 1: if b ( x ) is known then 2: u = − R − 1 b ( x ) (cid:62) y 3: else 4: for k
= 1 to m do 5: e k = [0 , . .
. , 0 , 1 (cid:124)(cid:123)(cid:122)(cid:125) k -th component , 0 , . .
. , 0] ∈ R m 6: u k = H ( x, y, R − 1 e k ) − H ( x, y, 0) − 12 ( R − 1 ) kk 7: end for 8: end if
D E RROR ANALYSIS
The objective of the error analysis is to study the convergence of the empirical variance of the particles S ( N ) t to its mean-ﬁeld limit ¯ S t as N → ∞ .
According to the Proposition 2, the mean-ﬁeld limit ¯ S t =
S t , hence it evolves backward in time according to
[Equation]
In order to study the error S ( N ) t − ¯ S t , we obtain a stochastic differential equation for the evolution of S ( N ) t .
Lemma 1.
The empirical covariance matrix S ( N ) t , deﬁned in (12) , evolves according to the stochastic differential equation
[Equation]
16) where M t is a Martingale given by
[Equation]
with quadratic variation
[Equation]
Proof.
The evolution for the error process e it :
= Y it − n ( N ) t is obtained by subtracting the evolution for the empirical mean n ( N ) t from (11):
[Equation]
The evolution for the empirical covariance S ( N ) t =
[Equation]
[Equation]
Ni =1 e it
[Equation]
T is obtained by application of the Itˆo rule.
Note that the ﬁrst term in the evolution of S ( N ) t in (16) is exactly identical to that of ¯ S t in (15), while the second term is an additional stochastic term that converges to zero as N → ∞ .
Even though the ﬂuctuations scale as O ( N − 12 ) , the analysis is challenging as has been noted in literature (see the remark after Theorem 3.1 in (Del Moral & Tugaut, 2018)).
Error analysis of the (stochastic) ensemble Kalman ﬁlter has been carried out in the literature (Del Moral & Tugaut, 2018; Bishop &
Del Moral, 2018) under additional assumptions which we also make below:
Assumption 1.
The matrix C (cid:62) C is identity.
And the control matrix B is full rank.
The assumption regarding the cost matrix C can be relaxed to C T C being non-singular by a change of coordinates.
The assumption regarding the control matrix is strong.
It is an open problem in the literature regarding stability of EnKF to carry out the error analysis under the more natural assumption that the system is controllable and observable.
Under assumption 1, we can prove the main result stated in the following proposition.
Proposition 3.
Let ¯ S t be the mean-ﬁeld covariance deﬁned in (10) and
S Nt be the empirical covariance of the particles deﬁned in (16) .
Then, under Assumption 1, the error between S ( N ) t and ¯ S t satisﬁes the upper-bound
[Equation]
where µ ∞ , c µ , c var are positive constants that are independent of time t .
In the remainder of this section, we prove the proposition 3.
To simplify the presentation, we use the time-reversed quantitative Ω ( N ) t := S ( N ) T − t and Ω t := ¯ S T − t which evolve forward in time according to
[Equation]
where M t is deﬁned in Lemma 1.
First, we show some basic results regarding the asymptotic properties of the Ricatti equation that governs Ω t .
These results are straightforward application of the classical stability theory of the
Ricatti equation.
Lemma 2.
Consider the Ricatti equation (18) that governs Ω t .
Then,
There exists a positive deﬁnite matrix solution Ω ∞ to the ARE
[Equation]
such that − A − Ω ∞ C (cid:62) C is Hurwitz.
Ω t converges to Ω ∞ exponentially fast 3.
Let µ t denote the minimum eigenvalue for the matrix F t :
= A + 12 C (cid:62) C Ω t .
Then, there exists a positive constant µ ∞ > 0 and c µ such that
[Equation]
Proof.
Under Assumption 1, the pair ( A, BR − 1 2 ) is controllable and the pair ( A, C ) is observable.
Part (1) follows from (Kwakernaak & Sivan, 1972, Theorem 3.7) and part (2) follows from (Ocone & Pardoux, 1996, Lemma 2.2 and Theorem 2.3).
The proof of part (3) follows from the following two facts (i) ( − A − 12 Ω ∞ C (cid:62) C ) is Hurwitz because
[Equation]
is a Lyapunov equation, with BR − 1 B (cid:62) positive deﬁnite since B is assumed to be full rank; and (ii) Ω t converges to Ω ∞ exponentially fast.
Therefore, there exists a time τ 1 > 0 such that ( − A − 12 Ω t C (cid:62) C is Hurwitz for t ≥ τ 1 .
As a result, the minimum eigenvalue µ t of F t =
A + 12 Ω t C (cid:62) C is lower-bounded µ t ≥ µ ∞ > 0 for t ≥ τ 1 .
This concludes the bound (20) where c [Equation] e − (cid:82) τ [Equation]
µ τ d τ
Next, we obtain a bound for E
[ Tr (Ω ( N ) t )] .
Upon taking the expectation and trace of (19)
[Equation]
where we used E
[ Tr ( X (cid:62) X )]
≥
Tr ( E [ X ] E
[ X ] (cid:62) ) for X = Ω ( N ) t
C (cid:62) to conclude the inequality.
Then, by inspecting the equation for Tr (Ω t ) , it follows that
[Equation]
where E 0 is a uniformly in time bounded constant, because Ω t converges to Ω ∞ .
Next, we obtain the bound for the error Ω ( N ) t − Ω t .
Subtracting (18) from (19) yields
[Equation]
Then, by application of the Itˆo rule to (Ω ( N ) t
[Equation]
− Ω t ) 2 and taking the trace,
[Equation]
Upon taking the expectation and deﬁning Θ t = E [ tr ((Ω ( N ) t
[Equation]
[Equation] where we used the bound (21), and Tr ( XY ) ≤ 12 Tr ( X (cid:62) X )
+ 12 Tr ( Y (cid:62) Y ) for X = Ω ( N ) t − Ω t and
Y = BR − 1 B (cid:62) .
In order to continue with proving the bound, we use the assumption C (cid:62) C
= I to conclude
[Equation]
because tr (Ω ( N ) t (Ω ( N ) t − Ω t ) 2 ) ≥ 0 .
Morevoer, due to exponential convergence of Ω t to Ω ∞ , tr (Ω 2 t ) ≤ E 1 uniformly.
Thus,
[Equation]
where µ t is the minimum eigenvalue for A + 12 Ω t .
Therefore,
[Equation]
where c var = 2 tr (( BR − 1 B (cid:62) ) 2 )
+ 2 E 0 tr ( BR − 1 B (cid:62) )
+ E 1 and we used (20).
The proof of the proposition follows by substitution S ( N ) T − t − S t
= Ω ( N ) t − Ω t and the bound Θ 0 ≤ 3 (cid:107) P T (cid:107) F N .
E D UALITY ROOTS OF THE PROPOSED ALGORITHM
For t ∈ (0 , T ) , the LQG value function is deﬁned as follows
[Equation]
[Equation]
From the DP optimality principle, it is easily shown Liberzon (2012) that
• The value function v t ( x ) = 12 x (cid:62) P t x + (constant) is quadratic where •
The matrix-valued process { P t ∈ R d × d : 0 ≤ t ≤ T } solves the DRE with terminal condition P T at time t = T .
Log transformation: The log transformation is a manifestation of the duality between optimal control and optimal estimation Mitter & Newton (2003);
Todorov (2008).
Deﬁne the probability density p t ( x ) ∝ e − v t ( x ) Fleming & Mitter (1982); Fleming (1977).
For the LQG problem, v t is quadratic and therefore, p t is a Gaussian density with variance P − 1 t =
S t .
Indeed, the Gaussian density of the random variable ¯ Y t equals p t .
The following proposition gives the precise connection:
Proposition 4.
Suppose p t
∝ e − v t where v t is the value function deﬁned in (22) .
Let ¯ p t be the density of the random variable ¯ Y t deﬁned in (9) .
Then, provided p T = ¯ p T ,
[Equation]
Proof.
By Proposition 2, ¯ Y t is a Gaussian random variable N (0 , S t ) where S t =
P − 1 t by deﬁnition.
Therefore,
[Equation]
Therefore, since p t ( x ) and ¯ p t ( x ) are both normalized to one, the constant is zero and ¯ p t ( x ) = p t
[Equation] x ) .
In the past, duality has been used to:
(i) Obtain linearly solvable sampling algorithms to solve optimal control problems Kappen & Ruiz (2016)
; Rawlik et al.
; Theodorou et al.
; Sch ¨ utte et al.
; Todorov
; and (ii) Set up optimal control problems for the purposes of estimation and simulation Sutter et al.
; Hartmann & Sch ¨ utte (2012); Van Handel (2006); Kim & Mehta (2020).
(iii)
A parallel approach to duality involves posing optimal control problems as problems of prob - abilistic inference, and using methods like expectation maximisation (Toussaint & Storkey, 2006), message passing (Hoffmann & Rostalski, 2017; Watson & Peters, 2021; Watson et al., 2021) or minimisation of KL divergence (Kappen et al., 2012;
Rawlik et al., 2013; Watson et al., 2020).
Although novel and distinct from the algorithms described in these earlier works – which instead rely on the use of a Feynman-Kac representation – the proposed algorithm is an example of a sampling algorithm to solve the optimal control problem.
In a related work, we have extended the results presented here to a class of nonlinear optimal control problems.
These nonlinear generalization more clearly reveals the duality roots of the proposed algorithm.
The EnKF algorithm arises as a special case in the LQG settings.
The nonlinear generalization will be a subject of future publication.
F D ETAILS OF THE NUMERICAL EXAMPLES IN S EC .
Notation: I n ∈ R n × n denotes the identity matrix and 1 d ∈ R d denotes a vector with all entries equal to 1.
F. E XPONENTIAL CONVERGENCE
The EnKF algorithm is implemented for the linear system (1) to approximate the optimal gain matrix.
The model matrices A and B are in the control canonical form
[Equation]
 
[Equation]
B
[Equation]
  ..
[Equation]
  where the entries ( a 1 , . .
a d ) ∈ R d is selected randomly from N (1 .
65 1 d , I d ) .
Additional model and simulation parameters are summarized in Table 2 and Table 3.
The numerical result for a single realization of the algorithm, for d = 2 , is depicted in Figure 1.
Part-(a) depicts the spectral properties of the open-loop ( ˙ x = Ax ) and the closed-loop ( ˙ x =
( A +
BK ( N )) x ) dynamics (recall that due to the backward in time nature of the EnKF, K 0 is the terminal gain which the algorithm yields).
It is observed that the algorithm learns the gain matrix that make the closed-loop dynamics stable, while the open loop dynamics A is unstable.
The entry-wise convergence of the matrix P ( N ) t as t goes from T to 0 is depicted in part (b)-(c)-(d).
The result is compared with the exact solution to the DRE P t and the asymptomatic limit, as the time horizon T → ∞ , which is the solution to the ARE.
Additional numerical results, for the case where d = 10 , are depicted in Fig.
4 for the and closed-loop eigenvalues and in Fig.
5 for the convergence of the 100 entries of the 10 × 10 matrix
[Table] [Table]
Solution of the ARE is depicted as the red dashed line.
As in Fig.
1 for the 2-dimensional system, these plots are depicted with respect to T − t .
The initialization is P T
=
I , the identity matrix.
F. C OUPLED MASS SPRING DAMPER SYSTEM
This system is taken from Mohammadi et al.
The matrices A and B are as follows:
[Equation]
where d s = d 2 is the number of masses and T ∈ R d s × d s is a Toeplitz matrix with 2 on the main diagonal and − 1 on the ﬁrst sub-diagonal and ﬁrst super-diagonal.
Additional model and simulation parameters are listed in Table 4 and 5, respectively.
[Table] [Table]
F. C OMPARISON BETWEEN E N KF AND POLICY - GRADIENT METHODS
The performance of the EnKF algorithm is compared with policy gradient algorithms presented in Mohammadi et al.
(2021c) (denoted as [M21]) and Fazel et al.
(2018) (denoted as [F18]).
These two algorithms are based on gradient free optimization for the gain matrix, with the difference that [M21] is for continuous-time systems while [F18] is for discrete-time systems.
We used the discretized linear system x t +∆ t =
( I + A ∆ t ) x t + Bu t ∆ t to implement [F18].
The two algorithms involve similar set of hyper-parameters that are selected optimally for each experiment for fair comparison.
The three algorithms, EnKF, [M21], and [F18] are implemented for the mass-spring system of Appendix F.2 for d = 2 , 4 , 10 .
The comparison is made by studying the relationship between the computational time and the error in approximating the optimal gain matrix.
The error is measured with the formula error =  
[Equation]
; for EnKF [Equation] ; for [M21] and [F18] where K ( N )is the terminal gain output by EnKF, K ( N ) T is the terminal gain output by the [M21] and [F18] algorithms, and K ∞ is the analytical gain obtained through the ARE.
In order to study the error vs computational time relationship for EnKF algorithm, the number of particles N is varied, while for [M21] and [F18] the number of iterations is varied.
The simulation time horizon T = 10 , and the step-size ∆ t = 0 .
01 is the same for all algorithms.
The hyperparameters of the [M21], and [F18] algorithms are presented in Table 6.
The initial guess K 0 = 0 , initial distribution D 0 = N (0 , I d ) , and gradient descent step α = 0 .
for both [M21] and [F18].
The values of the other hyper parameters are in Table 7.
The numerical results for d = 10 are depicted in Figure 2 and for d = 2 , 4 in Figure 6.
Additionally, Figure 7 shows comparison when the error is measured as
[Equation]
for EnKF
[Equation]
for [M21] and [F18] where c ( N )is the optimal control cost corresponding to the terminal gainas per EnKF, and by the initial gain as per [M21] and [F18] algorithms; and c ( N ) T is the optimal control cost corresponding to the initial gain as per EnKF, and terminal gain as per the [M21] and [F18] algorithms; and c ∞ is the optimal control cost achieved by analytical gain obtained through the ARE.
The simulations are implemented in Python 3 on a Intel Xeon E3-1240 V2 3.40 Ghz CPU, and the process time() function from the time module is used to evaluate the execution time. [Table]
N
[Figure]
[Figure]
F. C ART - POLE SYSTEM
The nonlinear model is taken from (Tedrake, Chapter 3.2.1): ˙ θ = ω ˙ ω = [Equation] θ [Equation]
˙ x = v ˙ v =
[Equation]
Hyper-param
[M21] [F18] d      r 10 −10 −10 −10 −10 −10 −
N For the LQG control design, we linearize the equations about the desired inverted equilibrium ( π, 0 , 0 , 0) .
The associated A and B matrices are as follows:
[Equation]


The model parameters are listed in Table 8 and the simulation parameters are in Table 9.
[Table]
(¯ θ, ¯ x, ¯ ω, ¯ v ) ( π, 0 , 0 , 0)
[Table]
( θ (0) , x (0) , ω (0) , v (0)) (1 . 25 π, − 0 .
[Table]
diag ([10 , 10 , 1 , 1])
[Table] [Table] [Table]