Online Learning of Optimal Control Signals in Stochastic Linear Dynamical Systems
Abstract
Among the most canonical systems are linear time-invariant dynamics governed by differential equations and stochastic disturbances.
Since in many application the true dynamics are not known, an interesting problem in this class of systems is learning to minimize a quadratic cost function when system matrices are unknown.
This work initiates theoretical analysis of implementable reinforcement learning policies for balancing exploration versus exploitation in such systems.
We present an online policy that learns the optimal control actions fast by carefully randomizing the parameter estimates to explore.
More precisely, we establish performance guarantees for the presented policy showing that the regret grows as the square-root of time multiplied by the number of parameters .
Implementation of the policy for a ﬂight control task shows its efﬁcacy.
Further, we prove tight results that ensure stability under inexact system matrices and fully specify unavoidable performance degradations caused by a non-optimal policy.
To obtain the results, we conduct a novel analysis for matrix perturbation, bound comparative ratios of stochastic integrals, and introduce the new method of policy differentiation.
These technical novelties are expected to provide a useful cornerstone for similar continuous-time reinforcement learning problems.
Introduction
State-space models are widely-used for decision-making in dynamic environments.
A popular model is the one that represents the continuous-time dynamics of the environment by linear stochastic differential equations.
In this setting, the multidimensional state of the system is driven by the control action and the Brownian noise, according to an Ito stochastic differential equation.
The range of applications is extensive, including chemistry, biology, ﬁnance and engineering [23, 38, 36, 29].
In many applications, uncertainties about the true dynamics necessitate adaptive methods that learn optimal actions, often relying on parameter estimation.
Accordingly, reinforcement learning policies are extensively studied for decision-making under uncertainty in discrete time systems.
The existing literature is fairly rich, including efﬁcient algorithms that use optimism in the face of uncertainty and posterior sampling [1, 2, 35, 22].
Further, regret bounds are available in the presence of domain knowledge, partial observation, and multiple agents [31, 10, 42, 43, 3, 28, 24].
On the other hand, the existing literature for continuous-time systems is still immature.
Early papers focus on estimating unknown parameters after an inﬁnitely long interaction with the environment [33, 32, 18, 17, 16].
Recently, sub-optimal policies with linear regrets are proposed and consistency is shown for systems with full-rank input matrices [9].
Ensuing papers study ofﬂine algorithms for learning to control continuous-time systems from a batch of data.
It is shown that computation of optimal policies can be performed using dynamic programming, temporal difference, entropy regularization, and system identiﬁcation [15, 37, 40, 5].
However, online reinforcement learning policies that can learn optimal actions from a single state trajectory without imposing additional costs to the system are currently unavailable.
The existing results rely on asymptotic estimation, require strong assumptions, and provide sub-optimal performance [16, 9].
A fundamental challenge compared to batch learning is that an online policy needs to simultaneously minimize the cost and estimate the unknown dynamics parameters.
These two goals are contradictory and constitute the important exploration-exploitation dilemma.
That is, estimation accuracy is necessary for optimal decision-making, while sub-optimal actions are required for obtaining accurate estimates.
We provide the ﬁrst online continuous-time reinforcement learning policy that balances the exploration-exploitation trade-off in a provably efﬁcient manner and under minimal assumptions.
The main contributions of this paper can be summarized as follows.
In Algorithm 1, we propose an efﬁcient online reinforcement learning policy based on randomized estimates of the unknown system matrices.
Then, we establish the rates for both the error in learning the dynamics matrices, as well as scaling of the regret that the algorithm incurs.
Algorithm 1 is easy to implement, yet, it learns the optimal control actions fast so that its regret at time T is O (cid:16) √ T log T (cid:17) (Theorem 4).
Furthermore, we study stability of linear systems for inaccurate system matrices and establish the minimal stabilizability margin (Theorem 2).
Finally, a reciprocal expression is provided for regret that fully captures sub-optimalities due to uncertainty or inaccuracy in approximating the optimal control action (Theorem 3).
The results provide both the average-case as well as worst-case analyses, the presented bounds are tight, and the assumptions are minimal.
To study online policies, one needs to address important challenges.
First, sensitivity analysis of (complex) eigenvalues of matrices with perturbed entries is needed.
Further, anti-concentration bounds for smallest singular values of partially-random matrices are required.
Finally, we need to fully characterize the cumulative sub-optimalities in cost function in terms of model uncertainties.
Thus, we develop multiple novel techniques for matrix-perturbation analysis, for comparative ratios of stochastic integrals , and for spectral properties of random matrices .
We also introduce the method of policy differentiation to precisely capture the cost of sub-optimal control actions.
En route, different tools from stochastic control, Ito calculus, and stochastic analysis are used, including HamiltonJacobi-Bellman equations, Ito Isometry, as well as martingale convergence theorems [41, 34, 4].
This paper is organized as follows.
In Section 2, we discuss the problem and the preliminary materials.
Then, in Section 3, we study system stability when the control action is designed based on dynamics matrices other than the true ones and establish stabilizability guarantees.
Next, effects of sub-optimal actions and the regret they cause are examined in Section 4.
Section 5 contains the randomizedestimates policy of Algorithm 1, as well as the theoretical analysis showing its efﬁciency.
Because of space limitations, all proofs are delegated to the appendices, as outlined on page 13.
Notation.
The smallest (largest) eigenvalue of A , in magnitude, is λ min ( A ) ( λ max ( A ) ).
For v ∈ C d , its norm is deﬁned as || v ||
2 = (cid:80) di
=1 | v i | 2 .
Moreover, we write ||| A ||| for the operator norm of matrices; ||| A ||| =
sup ||
v ||
=
||
Av || , and A † for Moore-Penrose
generalized inverse.
The sigma-ﬁeld generated by the stochastic process { Y s } 0 ≤ s ≤ t is denoted by σ ( Y 0: t ) .
A multivariate normal distribution with mean µ and covariance matrix Σ is shown by N ( µ, Σ) .
For λ ∈ C , we use (cid:60) ( λ ) , (cid:61) ( λ ) to denote the real and imaginary parts of λ , respectively.
The symbol ∨ (resp., ∧ ) is used to show the maximum (resp., minimum).
Finally, O ( · ) refers to the order of magnitude.
Problem Statement
We study reinforcement learning policies for a multidimensional Ito stochastic differential equation with unknown drift matrices.
That is, the state vector at time t is X t ∈ R d X , which follows the unknown dynamics equation
[Equation]
Above, the vector U t ∈ R
d U is the control action at time t , and the disturbance { W t } t ≥ 0 is a standard
Brownian motion in a d W dimensional space.
Technically, by ﬁxing the probability space (Ω , F , P ) which is completed by adding the null-sets of P , let all stochastic objects belong to this probability space, and let E [ · ] be expectation with respect to P (unless otherwise explicitly stated).
The Brownian motion { W t } t ≥ 0 starts from the origin and has independent normally distributed increments.
That is, W 0 = 0 , for all 0 ≤ t 1 ≤ t 2 ≤ t 3 ≤ t 4 , the vectors W t 2 − W t 1 and W t 4 − W t 3 are statistically independent, and for all non-negative reals s < t , it holds that W t − W s ∼ N (0 , ( t − s ) I d W ) .
Furthermore, C ∈ R d X × d W reﬂects the effect of W t on the state evolution.
We aim to design computationally tractable and provably efﬁcient reinforcement learning policies for the system in (1).
The transition matrix A (cid:63) , the input matrix B (cid:63) , and the noise-coefﬁcient matrix C , all are unknown .
The goal is to minimize the expected average cost
[Equation]
where c π ( X t , U t ) =
X (cid:62) t QX t
+ U (cid:62) t RU t is the cost at t .
The value of c π ( X t , U t ) is determined by the positive deﬁnite matrices Q, R of proper dimensions and by the policy π , as deﬁned below.
Technically, π is a non-anticipative closed-loop policy: At every time t , π determines U t according to the information available at the time.
More precisely, π maps the state observations (i.e., X s for s ∈ [0 , t ] ) and the previously taken actions (i.e., U s for s in the open interval [0 , t [ ) to the current control action U t .
This mapping can be stochastic or deterministic.
Importantly, π faces the fundamental exploration-exploitation dilemma for minimizing the expected average cost, because the dynamics matrices A (cid:63) , B (cid:63) are unknown and need to be learned based on the state and action observations.
The details of this dilemma are discussed in Section 5.
We assume that Q, R are known to the policy, the rationale being that the decision-makers are aware of the objective they aims to achieve, while their uncertainty about the true dynamics impedes them from deciding optimally.
The benchmark for assessing reinforcement learning policies is the optimal policy π (cid:63) that designs U t having access to A (cid:63) , B (cid:63) .
To deﬁne π (cid:63) , let Φ A (cid:63) ,B (cid:63) ( · ) :
R d X × d X → R d X ×
d X be
[Equation]
The above mapping is vital for ﬁnding π (cid:63) .
To see the intuition, ﬁrst note that a control action U t directly inﬂuences the current cost c π ( X t , U t ) , and indirectly affects the future cost values according to (1).
So, future consequences of decisions need to be considered for minimizing J π , and Φ A (cid:63) ,B (cid:63) ( · ) is employed for this purpose [12, 41].
To proceed towards identifying π (cid:63) , let
M = K
( A (cid:63) , B (cid:63) ) solve the equation Φ A (cid:63) ,B (cid:63) ( M ) = 0 .
In order to investigate existence and uniqueness of K ( A (cid:63) , B (cid:63) ) , we need the following deﬁnition and assumption.
Deﬁnition 1 (Notations λ ( · ) , E ( · ) ) Let λ ( D ) be the largest real-part of the eigenvalues of an arbitrary square matrix D : λ ( D )
=
max {(cid:60) ( λ ) : det ( D − λI ) = 0 } .
Further, for arbitrary matrices A ∈ R d X × d X , B ∈ R d X × d U , deﬁne E ( A, B )
= |||
A − A (cid:63) ||| + ||| B − B (cid:63) ||| .
So, E ( A, B ) measures the deviation of A, B from the true dynamics matrices A (cid:63) , B (cid:63) .
Note that unlike λ min ( · ) , λ max ( · ) that consider only magnitudes of the eigenvalues, λ ( · ) reﬂects the signs of the eigenvalues as well, and so can be either positive, zero, or negative.
However, they are related according to λ max (cid:0) e D (cid:1) =
e λ ( D ) .
We assume that the true dynamics matrices A (cid:63) , B (cid:63) are stabilizable:
Assumption 1 (Stabilizability)
There exists some L ∈ R d U × d X , such that λ ( A (cid:63) + B (cid:63) L ) < 0 .
Assumption 1 expresses that by applying U t =
LX t , the system can operate without any explosion.
Technically, it is straightforward to see that the solution of the differential equation (1) under the feedback policy U t =
LX t is
[Equation]
So, because of λ ( A (cid:63) + B (cid:63) L ) < 0 , X t does not grow unbounded with t .
Importantly, existence of a stabilizing matrix L is necessary for the problem to be well-deﬁned.
Otherwise, state explosion renders the average cost inﬁnite for all decision-making policies [12, 41].
Now, recall that
Φ A (cid:63) ,B (cid:63) ( K ( A (cid:63) , B (cid:63) ))
= 0 , let L ( A (cid:63) , B (cid:63) )
= − R − 1 B (cid:62) (cid:63) K ( A (cid:63) , B (cid:63) ) and deﬁne the feedback policy π [Equation]
[Equation]
We show that Assumption 1 sufﬁces for unique existence of K ( A (cid:63) , B (cid:63) ) and for optimality of π (cid:63) .
Theorem 1 (Optimal policy)
The matrix K ( A (cid:63) , B (cid:63) ) uniquely exists, and π (cid:63) in (3) gives
[Equation]
To compute K ( A (cid:63) , B (cid:63) ) , it sufﬁces to solve the differential equation ˙
M t = Φ A (cid:63) ,B (cid:63) ( M t ) starting from a positive semideﬁnite M 0 .
Note that it is equivalent to the integration M t
=
M 0 + t (cid:82) Φ A [Equation] ,B (cid:63) ( M s )d s .
In the proof of Theorem 1, we show that lim t →∞ M t =
K ( A (cid:63) , B (cid:63) ) .
Next, we focus on learning A (cid:63) , B (cid:63) and the additional cost compared to the cost of π (cid:63) .
To that end, we formulate sub-optimalities of decision-making policies and the increase in cost due to lack of knowledge about the optimal actions U t =
L ( A (cid:63) , B (cid:63) )
X t .
For a general policy π , the regret of π at time T is denoted by R π ( T ) and is deﬁned as the cumulative increase in the cost by T .
That is, the difference between the instantaneous costs of π and π (cid:63) in (3) is integrated over 0 ≤ t ≤ T :
[Equation]
Clearly, randomness of state and action leads to that of regret.
So, regret analyses for reinforcement learning policies include worst-case analyses that establish upper-bounds for R π ( T ) , as well as average-case analyses that provide bounds for E
[ R π ( T )] .
Further, for unknown A (cid:63) , B (cid:63) , we hope that the increasing observations of state and action over time will be effectively leveraged so that eventually, the policy makes near-optimal decisions.
So, as t grows, we expect c π ( X t , U t ) − c π (cid:63) ( X t , U t ) to shrink.
Accordingly, R π ( T ) is desired to scale sub-linearly with T .
In the rest of this paper, we study R π ( T ) , E [ R π ( T )] , and their dependence on T and the problem parameters.
Another quantity of interest is the accuracy of estimating the unknown dynamics.
So, letting A t , B t be estimates of A (cid:63) , B (cid:63) based on the state-action observations by time t ; i.e., { X s , U s } 0 ≤ s ≤ t , we study the decay rate of the estimation error E ( A t , B t ) , as deﬁned in Deﬁnition 1.
Stability Analysis for Perturbed Dynamics Matrices
In this section, we study the effects of uncertainties about the dynamical model on the system stability.
We specify the minimal information one needs to possess in order to ensure stabilization, and show that a coarse-grained approximation of the truth is sufﬁcient for that purpose.
Results of this section will be used later in the design of randomized-estimates policy in Section 5.
Importantly, the stability analysis in this section is general, captures effects of all involved quantities, and provides tight results in the sense that the conditions of Theorem 2 are required for guaranteeing stabilization.
In addition, the results presented here are of independent interests, because stability is required for letting the system operate for a reasonable time period.
To proceed, note that if hypothetically the optimal linear feedback in (3) is applied to the system in (1), then stability is guaranteed.
More precisely, applying L ( A (cid:63) , B (cid:63) ) , the resulting closed-loop transition matrix D (cid:63) =
A (cid:63) + B (cid:63) L ( A (cid:63) , B (cid:63) ) has all its eigenvalues on the open left half-plane of the complex plane, as stated in Theorem 1.
The issue is that the true dynamics matrices A (cid:63) , B (cid:63) are unknown and need to be learned.
However, if some matrices A, B meet the conditions we shortly discuss, one can stabilize the system by applying the linear feedback U t =
L ( A, B ) X t .
To proceed, let D
=
A + B L ( A, B ) be the closed-loop transition matrix of a system with dynamics matrices A, B , under the feedback U t =
L ( A, B ) X t .
Then, let ρ > 0 and ζ < ∞ satisfy
[Equation]
The quantities in (5) both are required for studying stability of the matrix A (cid:63) + B (cid:63) L ( A, B ) , as follows.
Remember that we have a closed-loop stability result in Theorem 1: λ ( D ) < 0 .
So, the ﬁrst inequality in (5) quantiﬁes the extent to which L ( A, B ) is able to stabilize the system of parameters
A, B .
Intuitively speaking, λ ( D ) is the smallest (i.e., most negative) upper-bound one can hope for the eigenvalues of A (cid:63) + B (cid:63) L ( A, B ) , since the optimal feedback L ( A, B ) is purposefully designed for the certainly known matrices A, B .
It can be shown that the positive quantity ρ enjoys a uniform positive lower-bound as long as A, B live in some neighborhoods of A (cid:63) , B (cid:63) , as expressed in (31).
The second inequality in (5) is somewhat guaranteed by the ﬁrst one.
Technically, it is shown in (22) in the proof of Theorem 1 that
[Equation]
Therefore, λ ( D ) ≤ ρ < 0 implies that for some ζ < ∞ , we have |||K ( A, B ) ||| ≤ ζ .
So, ζ is merely used for simplifying the expressions.
Towards stability analysis, we need further information of D =
A + B L ( A, B ) that the Jordan form of this matrix provides.
Suppose that eigenvalues of D are λ 1 , · · · , λ k , and let the Jordan decomposition be D
= P − 1 Λ P .
So, Λ = diag (Λ 1 , · · · , Λ k ) is a block-diagonal matrix, and all diagonal entries of Λ i are λ
i , the immediate off-diagonal entries above the diagonal of Λ i are 1 , and all other entries of Λ i are 0 (as shown in (24)).
Now, a consequential quantity for determining the stability margin is the largest block-size µ D deﬁned in Deﬁnition 2.
It depends on the abovementioned blocks
Λ i , i = 1 , · · · , k , by letting µ i denote the dimension of the square matrix Λ i , and referring to the largest among µ 1 , · · · , µ k by µ D .
Deﬁnition 2 (Largest block-size µ D ) Letting P and Λ i ∈ C µ i × µ i be as in the Jordan decomposition D =
P − 1 Λ P explained above, deﬁne µ D = max 1 ≤ i ≤ k µ i .
The quantity µ D is the largest size of the blocks Λ 1 , · · · , Λ k in the Jordan form and crucially determines the order of stability margin, as established in the following theorem.
Theorem 2 (Stability margin) Let P, µ D and ρ, ζ be as in Deﬁnition 2 and (5) , respectively.
Then, we have λ
( A (cid:63) + B (cid:63) L ( A, B ))
< − δ , as long as
[Equation]
Note that in (7), we can upper-bound |||L ( A, B ) ||| − 1 with λ min ( R ) ζ − 1 ||| B ||| − 1 .
Theorem 2 states that if E ( A, B ) is sufﬁciently small to satisfy (7), then A (cid:63) + B (cid:63) L ( A, B ) is stable and all of its eigenvalues in the complex plane lie on the left-hand-side of the vertical line (cid:60) = − δ .
In addition, (7) reﬂects effects of different factors, as follows.
First, the reason for the decrease in the stability margin on the right-hand-side of (7) as |||L ( A, B ) ||| increases, is that the difference between the closed-loop matrices is the product of B (cid:63) − B and L ( A, B ) .
Further, as can be seen in Deﬁnition 2, the quantity µ 1 / 2 D ||| P ||| (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) P − 1 (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≥ 1 quantiﬁes non-diagonality of D , and it is 1 for fully diagonal matrices D where P is the identity matrix and µ D = 1 .
Therefore, more non-diagonal closed-loop matrices D lead to smaller stability margins and make stabilization harder to learn.
The dependence on ρ corroborates the intuition that systems whose optimal closed-loop matrices has eigenvalues of larger real-parts (i.e., smaller ρ ), are harder to stabilize.
Moreover, the expression ( ρ − δ ) ∧ ( ρ − δ ) µ D indicates that µ D is very important for determining the rates of bounding the eigenvalues of A (cid:63) + B (cid:63) L ( A, B ) .
The rates for ρ − δ < 1 and ρ − δ > 1 are different, because of a similar phenomena in sensitivity of eigenvalues of matrices to perturbations in their entries.
This result is of independent interest as it is a generalization of Bauer-Fike Theorem [6] to asymmetric matrices we establish in the proof of Theorem 2.
Intuitively, we prove that larger blocks in the Jordan decomposition can lead to drastic ﬂuctuations in eigenvalues.
To close this section, we provide uniform lower and upper bounds for ρ > 0 and ζ < ∞ , respectively.
For that purpose, similar to Deﬁnition 2, deﬁne the largest block size µ (cid:63) = µ D (cid:63) based on the
Jordan decomposition D (cid:63) =
A (cid:63) + B (cid:63) L ( A (cid:63) , B (cid:63) ) =
P − 1 (cid:63) Λ (cid:63) P (cid:63) .
Then, we show in the proof of
Theorem 2 that E ( A, B ) ≤ (cid:15) 0 is sufﬁcient for satbilization, ρ ≥ λ min ( Q ) 4 − 1 |||K ( A (cid:63) ζ ≤ 2 |||K ( A (cid:63) , B (cid:63) ) ||| , where
[Equation]
(cid:1) µ (cid:63) µ 1 / 2 (cid:63) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) P − 1 (cid:63) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ||| P (cid:63) ||| ∧  ∞ (cid:90) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) e D (cid:63) t (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) 2 d t   −. [Equation] [Equation]
Reciprocal Regret Bounds and Policy Differentiation
In this section, we investigate sub-optimalities and provide a tight expression for the regret that non-optimal control actions cause.
Such investigation is vital since reinforcement learning policies need to learn the unknown dynamics A (cid:63) , B (cid:63) and so their non-optimal actions are unavoidable.
To proceed, let U t be the control action of the policy π at time t .
In the following theorem, we quantify R π ( T ) in terms of deviations U t −L ( A (cid:63) , B (cid:63) )
X t , and introduce α T that fully assesses the regret of π .
In fact, α T uniﬁes average-case and worst-case analyses by capturing both E [ R π ( T )] and R π ( T ) .
Further, Theorem 3 provides scalings with different problem parameters.
For example, it shows that the difference between R π ( T ) and
E
[ R π ( T )] scales linearly with the dimension of the Brownian motion.
After the statement of Theorem 3, we discuss its intuitions and implications.
To establish Theorem 3, we utilize the theory of continuous-time martingales and (in Lemma 2) develop novel results on comparative ratios of stochastic integrals.
More importantly, we construct the new framework of policy differentiation for ﬁnding tight and accurate regret bounds.
Broadly speaking, policy differentiation precisely evaluates the regret in terms of inﬁnitesimal suboptimalities.
Then, we integrate these inﬁnitesimal deviations and obtain α T , in which the integrand [Equation]
2 plays a role similar to the derivative of the regret.
This framework can be used for an analogous regret analysis in other continuous-time reinforcement problems.
Theorem 3 (Regret analysis) Suppose that L t is a bounded piecewise continuous function of t , and π is the policy U t =
L t X t .
Then, we have E [ R π ( T )]
=
E [ α T ] , and
[Equation]
where D (cid:63) =
A (cid:63) + B
[Equation]
B
[Equation]
E
[Equation]
= e
[Equation]
ω
[Equation]
and α T =
T (cid:90)
[Equation]
[Table]
The boundedness and piecewise continuity conditions are somewhat natural to ensure that the state process in (2) is well-deﬁned.
Because the optimal policy is a time-invariant feedback, violating these conditions does not lead to smaller regret.
Furthermore, since by Theorem 1 we have λ ( D (cid:63) ) < 0 , the matrix E t exponentially decays as t grows.
Therefore, the second integral in the deﬁnition of α T is dominated by the ﬁrst one.
So, Theorem 3 shows that the sub-optimality π incurs at time t scales as square of the deviation L t − L
( A (cid:63) , B (cid:63) ) .
Finally, the constant ω R reﬂects effects of different problem parameters and indicates that R π ( T ) − α T scales linearly with d W .
The results of Theorem 3 are interesting along different directions.
First, the exact equality E [ R π ( T )]
=
E [ α T ] can be used for establishing regret lower-bounds by studying the fastest rates π (cid:63) is learned and the deviation L t −L ( A (cid:63) , B (cid:63) ) shrinks.
Further, since α 1 / 2 T log α T
=
O ( α T ) , not only the average-case criteria E
[ R π ( T )] , but also the worst-case regret R π ( T ) are captured by α T .
In other words, Theorem 3 indicates that the ﬂuctuations of R π ( T ) around its expectation E [ R π ( T )] are in magnitude smaller than the expected value itself.
Thus, studying α T is sufﬁcient and necessary for regret analysis, and it provides a general reciprocal relationship for all policies.
Moreover, when A (cid:63) , B (cid:63) are unknown, a reinforcement learning policy becomes progressively more capable of narrowing down the sub-optimality gap as time goes by.
More precisely, as soon as having sufﬁciently long trajectories to learn A (cid:63) , B (cid:63) to satisfy (cid:12)(cid:12)(cid:12)(cid:12) R 1 /
( L t − L ( A (cid:63) , B (cid:63) ))
X t (cid:12)(cid:12)(cid:12)(cid:12) < 1 , the regret grows much slower since α T integrates the squares of these deviations.
For example, if the estimation accuracy satisﬁes the ideal square-root rate E ( A t , B t ) =
O (cid:0) t 1 / 2 (cid:1) , then the regret scales logarithmic with time; α T =
O (log T ) .
However, due to the trade-off between the exploration and exploitation, this is not the case (see Proposition 1).
Technically, to obtain the above error rate, U t needs to persistently deviate from L ( A (cid:63) , B (cid:63) )
X t , which causes a linearly growing regret.
Theorem 3 provides both a general result for analyzing online policies, as well as a useful insight on how to design them.
We utilize this insight to design Algorithm 1 and to establish Theorem 4 in the next section.
Indeed, we randomize the parameter estimates so that U t appropriately deviates from π (cid:63) , leading to E ( A t , B t ) =
O (cid:0) t − 1 / 4 (cid:1) .
So, we obtain the efﬁcient regret bound α T = (cid:101) O (cid:0) T 1 / 2 (cid:1) .
Randomized-Estimates Policy
In this section, we discuss fast and tractable algorithms with efﬁcient performances for cost minimization subject to uncertainties about the dynamics matrices A (cid:63) , B (cid:63) .
First, we explain the fundamental exploration-exploitation dilemma.
Then, we investigate a procedure for estimating the unknown dynamics using the data of state-action trajectory.
Based on that, an online reinforcement learning policy that employs randomizations of the parameter estimates for balancing exploration versus exploitation is presented in Algorithm 1.
Next, a regret bound is established in Theorem 4 indicating that Algorithm 1 efﬁciently minimizes the cost function so that the regret scales as the square-root of the time.
We also specify the rates of identifying the dynamics matrices.
First, by Theorem 3, the policy needs to ensure that U t ≈ L ( A (cid:63) , B (cid:63) )
X t to incur a small regret.
Further, since A (cid:63) , B (cid:63) are unknown, the policy needs to estimate them based on { X s , U s } 0 ≤ s ≤ t by time t .
However, if U s ≈ L ( A (cid:63) , B (cid:63) )
X s , then some of the coordinates of the data point X s , U s become (almost) uninformative as they are (approximately) linear transformations of the rest of them.
This defeats the purpose and renders accurate estimation of A (cid:63) , B (cid:63) infeasible.
Note that accurate approximations of A (cid:63) , B (cid:63) are needed for taking near-optimal actions.
This, known as the explorationexploitation dilemma, is the main obstacle in online reinforcement learning and shows that a low-regret policy needs to diversify the actions { U s } 0 ≤ s ≤ t by deviating from {L ( A (cid:63) , B (cid:63) )
X s } 0 ≤ s ≤ t .
Now, we discuss the learning procedure in Algorithm 1, based on extensions of the least-squares estimates.
Suppose that instead of the full data X s , U s for real values of s ≥ 0 , one has access to samples at a discrete set of time points; X k(cid:15) , U k(cid:15) for k = 0 , 1 , · · · .
Then, (1) for a small (cid:15) gives the approximate data generation mechanism X
[Equation]
X
[Equation] = [Equation] So, a natural approach is to form a linear regression problem and estimate A (cid:63) , B (cid:63) by a minimizer of (cid:80) k (cid:12)(cid:12)(cid:12)(cid:12)
X ( k +1) (cid:15) − X k(cid:15) − ( AX k(cid:15) + BU k(cid:15) )
(cid:15) (cid:12)(cid:12)(cid:12)(cid:12) 2 over A, B .
Denoting Y s = (cid:2) X (cid:62) s , U (cid:62) s (cid:3) (cid:62) , this gives the estimates [ A, B ] = (cid:80)
k (cid:0) X ( k +1)
(cid:15) − X k(cid:15) (cid:1) Y (cid:62) k(cid:15) (cid:15) (cid:0)(cid:80)
k Y k(cid:15) Y (cid:62) k(cid:15) (cid:15) 2 (cid:1) † .
Therefore, letting (cid:15) → 0 , the summations become integrations and we get the continuous-time estimate based on the full trajectory { X s , U s } 0 ≤ s ≤ t .
The result is shown in (9) below and will be used by Algorithm 1.
To ensure that the system evolves stable, Algorithm 1 projects the parameter estimates on the stabilization oracle S 0 in lights of Theorem 2.
In the sequel, we explain how one can learn S 0 fast.
Deﬁnition
For a ﬁxed δ 0 > 0 , let S 0 be a set containing matrices A, B that satisfy (7) for δ =
δ 0 .
Intuitively, the system is stabilized by having access to S 0 , despite uncertainties about the true dynamics matrices A (cid:63) , B (cid:63) .
Note that the condition in (7) is veriﬁable since ρ, ζ depend on the known parameter estimates A, B .
Availability of a stabilization set is a common assumption in the literature of online reinforcement learning policies for linear systems [33, 32, 8, 2, 35, 22, 42, 43].
For example, if an initial stabilizing feedback L 0 is available, we can devote a short time period to explore by applying sub-optimal control actions and use the resulting observations to learn S 0 .
In systems operating prior to running Algorithm 1 or in open-loop-stable systems, this condition automatically holds (in the latter case, L 0 = 0 is an initial stabilizer).
Similarly, in systems with a reset option that can immediately steer the system-state to small values, S 0 can be learned fast [9, 5, 19].
In Section 3 we saw that an (cid:15) 0 neighborhood of A (cid:63) , B (cid:63) is sufﬁcient for bounding ρ, ζ , for (cid:15) 0 in (8).
That is, a coarse-grained approximation of A (cid:63) , B (cid:63) sufﬁces for S 0 .
So, if S 0 is not available in advance, it can be learned from the state-input trajectories of short lengths [16, 9, 20, 21, 27, 13, 19, 25].
In addition, in Theorem 4 we show that the algorithm learns A (cid:63) , B (cid:63) with the rate t − 1 / 4 .
So, the projection on S 0 will be automatically performed after the time t = (cid:101) O (cid:0) (cid:15) − 4 0 (cid:1) .
The algorithm proceeds as follows.
For some ﬁxed γ > 1 , the sequence { γ n } ∞ n =0 contains the time instants at which the algorithm updates the parameter estimates.
In fact, Algorithm 1 applies control actions U t =
L ( A n , B n ) X t during the time period γ n ≤ t < γ n +1 , where A n , B n are estimates of
A (cid:63) , B (cid:63) , based on the trajectory up to time γ n .
Further, to ensure that the policy commits sufﬁciently to explore the environment, a random matrix Θ n is added to the parameter estimates at time t = γ n .
Then, Algorithm 1 projects the resulting d X × ( d X + d U ) matrix onto S 0 .
Formally, letting Π S 0 ( · ) denote projection on S 0 , deﬁne
[Equation]
where Y s = (cid:2) X (cid:62) s , U (cid:62) s (cid:3) (cid:62) and the d X × (
d X + d U ) matrices { Θ n } ∞ n =0 are independent of everything else and of each others.
Further, the random matrix Θ n that is used at time t = γ n has independent
Gaussian entries of mean zero and standard deviation σ n = σ 0 ( γ − n n ) 1 / 4 =
σ 0 (cid:0) t − 1 log γ t (cid:1) 1 / 4 , for some ﬁxed σ 0 .
This decay rate of σ n is delicately adjusted for two purposes.
On one hand, Θ n is sufﬁciently large for randomizing the parameter estimates to ensure that effective exploration occurs and the current data is diverse enough so that we obtain accurate estimates in the future.
On the other hand, Θ n is sufﬁciently small to let the current estimates remain accurate and prevent signiﬁcant deviations.
Otherwise, large values of Θ n deteriorate the current efﬁcient exploitation.
Select γ > 1 and A 0 , B 0 ∈ S 0 arbitrarily
For 0 ≤ t < 1 = γ 0 , apply U t =
L ( A 0 , B 0 )
X t for n = 0 , 1 , 2 , · · · do
Update parameter estimates A n , B n by (9) while γ n ≤ t < γ n +1 do
Take action U t =
L ( A n , B n ) X t end while end for
The memory that Algorithm 1 occupies is remarkably small since in (9) it can update the involved matrices by continuously integrating the stream of the observations Y s , d X s .
Furthermore, calculations can be done fast, making update of the parameter estimates at time γ n immediately effective.
The rationale for freezing the parameter estimates for exponentially growing time intervals γ n ≤ t < γ n + is that Algorithm 1 can defer the learning step until collecting enough observations Y s so that a new update of parameter estimates is effectively more accurate than the previous one.
The following result provides performance guarantees for the randomized estimates policy.
Theorem 4 (Analysis of Algorithm 1) Let the policy π and the estimates A n , B n be those in Algorithm 1.
Assume that n satisﬁes γ n ≤ T < γ n +1 .
Then, using Deﬁnition 1 and (4) , we have
[Equation]
where
[Equation]
E .
Theorem 4 indicates efﬁciency of Algorithm 1:
At time T , the sub-optimality gap is as small as
O (cid:0) ω π T − 1 / 2 log T (cid:1) .
It also provides ω π , ω E that reﬂect the dependence of estimation error and regret on different parameters in the problem.
So, the regret scales linearly with the number of unknown parameters in A (cid:63) , B (cid:63) , while the estimation error dwindles linearly with the dimension.
To establish Theorem 4, we study the learning step in (9) and determine the rates Algorithmestimates A (cid:63) , B (cid:63) .
For that purpose, we prove concentration bounds for the empirical covariance matrices of the state vectors and also show anti-concentration of the Gram matrix t (cid:82)
Y s Y (cid:62) s d s of the signal Y s = (cid:2) X (cid:62) s , U (cid:62) s (cid:3) (cid:62) , as t grows.
Then, we establish bounds on the comparative ratios of stochastic integrals and use that for controlling the estimation error.
Furthermore, we show that the optimal feedback matrices have a Lipschitz property with respect to the dynamics matrices and leverage that for ﬁnding the deviation rates from the optimal feedback.
Finally, we utilize policy differentiation and Theorem 3 for getting the regret bounds in Theorem 4.
Note that for obtaining descending estimation errors and sub-linear regret bounds we need λ min (cid:0) CC (cid:62) (cid:1) > 0 .
This is a standard requirement in estimation and control of stochastic linear systems and expresses that all coordinates of the state vectors are randomized by the Brownian motion { W t } t ≥ 0 in a relatively short time [30, 39, 9].
So, all state variables have signiﬁcant roles in the dynamics.
From a modeling point of view, λ min (cid:0) CC (cid:62) (cid:1) > 0 indicates that the stochastic differential equation in (1) is irreducible in the sense that a smaller subset of state variables is insufﬁcient for capturing the stochastic dynamical behavior of the environment.
To close this section observe that the estimation error of Algorithm 1 shrinks as T − 1 / 4 .
So, it does not decay with the ideal square-root rate because the main priority of Algorithm 1 is to minimize the regret by exploring minimally.
However, if the randomization matrices Θ n are persistent and their standard deviations do not diminish as n grows, then we obtain the square-root consistency.
This is formalized in Proposition 1.
Of course, the compromise is that R π ( T ) grows linearly with T if Θ n does not dwindle as n grows.
Numerical Illustrations.
Here, we provide numerical analyses of Algorithm 1.
The details of the experiments are provided in Appendix A.
The normalized rates of the estimation error are plotted in Figure 1, versus the continuous time T .
To illustrate the rates in Theorem 4, the ﬁgure contains multiple trajectories of T 1 / 2 E ( A n , B n ) 2 , while n, T satisfy γ n ≤ T < γ n +1 .
The normalized estimation errors in Figure 1 are (almost) bounded as time grows, corroborating Theorem 4.
Figuredepicts normalized regret versus time.
That is, the horizontal axis is T , while the vertical one presents the values of T − 1 / 2 R π ( T ) , where π is the online reinforcement learning policy in Algorithm 1.
Again, it is clear that Theorem 4 holds, as the normalized regret remains bounded as time grows.
Concluding Remarks
We studied online reinforcement learning policies for unknown continuous-time stochastic linear systems and presented algorithms that learn to minimize quadratic costs.
Three important problems are fully investigated, followed by the intuitions and implications of the presented analyses.
First, we studied stabilization of stochastic linear systems based on incorrect dynamics matrices and proved Theorem 2 that speciﬁes the coarse-grained approximations for guaranteeing stability.
Then, proposing the novel approach of policy differentiation, we established a reciprocal result in
Theorem 3 for the regret that policies cause by taking sub-optimal actions.
More importantly, we presented the online reinforcement learning Algorithm 1 and established its performance guarantees.
Indeed, Theorem 4 expresses that the estimation rate of Algorithm 1 is O (cid:16) dT − 1 / 4 log 1 / 2 T (cid:17) and it enjoys the efﬁcient regret bound O (cid:0) d 2 T 1 / 2 log T (cid:1) , where T is the time and d is the dimension.
As an initiating paper on design and analysis of online reinforcement learning policies for continuoustime stochastic systems, this study introduces interesting directions for future work.
That includes establishing anytime regret bounds that hold uniformly over time, investigating high-dimensional systems with structured dynamics such as low-rank or sparse matrices, designing efﬁcient policies under imperfect state-observations, and extending the analysis to non-linear systems. References Appendices
A Numerical Illustrations of Estimation Error and Regret
B Proof of Theorem 1 (Optimal policy)
C Proof of Theorem 2 (Stability margin)
C.1 Proof of sufﬁciency of (8) for stabilization bounds . . . . . . . . . . . . . . . . . . D Proof of Theorem 3 (Regret analysis) E Proof of Theorem 4 (Analysis of Algorithm 1) F Estimation Rates under Persistent Randomization
G Auxiliary Lemmas G.1 Difference in regrets of two policies . . . . . . . . . . . . . . . . . . . . . . . . G.2 Scaling of self-normalized stochastic integrals . . . . . . . . . . . . . . . . . . . G.3 Lipschitz continuity of optimal feedback . . . . . . . . . . . . . . . . . . . . .
G.4 Effects of sub-optimal linear feedback policies . . . . . . . . . . . . . . . . . .
G.5 Convergence of empirical covariance matrix of the state vectors . . . . . . . .
G.6 Manifolds of dynamical systems with equal optimal feedback matrices . . . . . A Numerical Illustrations of Estimation Error and Regret
Now, we provide numerical analyses for showcasing the performance of Algorithm 1 for estimating the unknown dynamics matrices and learning the optimal policy.
For this purpose, we assume that the true continuous-time system matrices are lateral-directional state-space matrices of X-29A airplane at 4000 ft altitude [7].
The system is of dimension d X = 4 , is controlled by two dimensional commands; d U = 2 , and the transition and input matrices in (1) are
A (cid:63) =
[Table]
Note that the above dimensions of the control action and the state vector, as well as open-loop instability of A (cid:63) and the notably small entries of B (cid:63) , render control of the above-mentioned airplane challenging.
Further, we let the coefﬁcient matrix of the Brownian disturbance W t be C = 0 .
2 ×
I 4 , and employ Algorithm 1 to learn to control a quadratic cost with the weight matrices Q = 10 ×
I d X , R = I d U .
The online reinforcement learning policy of Algorithm 1 is run for 500 seconds, while the parameter estimates are updated at times t = γ n , for integer values of n and γ = 1 . 2 .
In order to ﬁnd an initial stabilizing feedback, we run the Bayesian-learning stabilization algorithm forseconds [19].
B Proof of Theorem 1 (Optimal policy)
Fixing (cid:15) > 0 , suppose that the control inputs U t are frozen in intervals of length (cid:15) and can change only at times k(cid:15) , for k = 0 , 1 , · · · .
That is, for all times t satisfying k(cid:15) ≤ t < ( k + 1) (cid:15) , the action vector is ﬁxed; U t = U k(cid:15) .
Next, we proceed towards ﬁnding a decision-making policy for minimizing the expected average cost.
Note that due to the above-mentioned freezing during (cid:15) -length intervals, resulting decision-making policies can be sub-optimal, and indeed provide an upper bound for the optimal cost value.
Next, ﬁx an arbitrary time horizon T , and denote the minimum cost-to-go at time t by
[Equation]
where the inﬁmum is taken over non-anticipating policies that freeze the control action in (cid:15) -length intervals, as elaborated above, and the information at time t is F t
= σ ( X 0: t , U 0: t ) ; the sigma-ﬁeld generated by the state and action vectors up to the time.
Now, ﬁnding an optimal policy is equivalent to applying dynamic programming principle and writing Bellman optimality equations [26, 12].
So, we have
[Equation]
subject to the dynamics equation in (1).
For the sake of simplicity, suppose that T/(cid:15) is an integer.
Solving (10) for k = T/(cid:15) − 1 , we get the optimal control action U (cid:63)k(cid:15)
= 0 .
Accordingly, this gives
[Equation]
for k = T/(cid:15) − 2 , which, after substituting in (10), becomes
[Equation]
where we applied Fubini’s Theorem to derive
[Equation]
[Equation]
However, solving the dynamics (1) for k(cid:15) ≤ t ≤ ( k + 1) (cid:15) , we obtain
[Equation]
which together with Ito’s Lemma [Equation] d W s d W (cid:62) s =
I d W d s [34] [Equation] yields to
[Equation]
Plugging these results in the dynamic programming equation in (11), the expression in front of the minimum becomes the following quadratic function of U k(cid:15) :
[Equation]
(cid:101) GU k(cid:15) + U (cid:62) k(cid:15) (cid:101) RU k(cid:15) + (cid:16) (cid:101) AX k(cid:15) + (cid:101) BU k(cid:15) (cid:17) (cid:62) P k +1 (cid:16) (cid:101) AX k(cid:15) + (cid:101) BU k(cid:15) (cid:17) + tr (cid:16) (cid:101) P k +1 CC (cid:62) (cid:17) , where P k +1 = Q(cid:15) , and (cid:101)
A = e A (cid:63) (cid:15) , (cid:101) B = (cid:15) (cid:90) e A (cid:63) s d sB (cid:63) , (cid:101)
Q = (cid:15) (cid:90)
e A (cid:62) (
cid:63) t
Qe A (cid:63) t d t, (cid:101) G =
(cid:15) (cid:90) e A (cid:62) (cid:63) t Q   t (cid:90) e A (cid:63) ( t − s )
d s 
 B (cid:63) d t, (cid:101) R = R(cid:15)
+ (cid:15) (cid:90)
B (cid:62) (cid:63)   t (cid:90) e A (cid:62) (cid:63) ( t − s )
Qe A (cid:63) ( t − s )
d s 
 B (cid:63) d t, (cid:101) P k + = P k + (cid:15) (cid:90)
e A (cid:62) (cid:63) s e A (cid:63)
s d s + (cid:15) (cid:90)
[Equation]
Note that in the last equation above
[Equation]
we used Ito Isometry [4] to ﬁnd (cid:101)
P k +1 .
Now, performing the minimization the optimal control action is
[Equation]
and (11) leads to
[Equation]
where P k is calculated according to the discrete time Riccati equation
[Equation]
P k +1 (cid:101)
A + (cid:101) G (cid:62) (cid:17) .
It is shown that if there is some matrix L such that λ max
[Equation]
+
[Equation]
BL
[Equation]
< 1 , then as k → −∞ , the matrix P k in the above discrete time Riccati equation converges to a uniquely existing matrix P that solves the algebraic Riccati equation
[Equation]
regardless of the terminal matrix for the largest value k + 1 , which here corresponds to P T/(cid:15)
Next, we show that if (cid:15) is sufﬁciently small, then the matrix L mentioned above exists.
To that end, write
[Equation]
where
[Equation]
Then, letting L be as in Assumption 1, if (cid:15) is small enough, it holds that
[Equation]
That is because the eigenvalues of the matrix M ( (cid:15) )
( A (cid:63) + B (cid:63) L ) are continuous functions of (cid:15) , and for (cid:15)
=
we have λ ( M (0)
( A (cid:63) + B (cid:63) L ))
=
λ (( A (cid:63) + B (cid:63) L )) < 0 , according to Assumption 1.
Hence, (cid:101)
A + (cid:101)
BL =
I d X + M ( (cid:15) )
( A (cid:63) + B (cid:63) L ) (cid:15) implies that eigenvalues of (cid:101)
A + (cid:101)
BL are exactly one plus the eigenvalues of M ( (cid:15) )
( A (cid:63) + B (cid:63) L ) (cid:15) .
So, it holds that
[Equation]
Now, putting (16) and (17) together, if (cid:15) is small enough, then λ max (cid:16) (cid:101)
A + (cid:101) BL (cid:17) < 1 .
Henceforth, suppose that (cid:15) is sufﬁciently small so that the latter inequality holds true.
As long as (cid:15) > 0 is small enough as described above, letting the time horizon T tend to inﬁnity, the (cid:15) -length frozen optimal policy for minimizing the expected average cost is
[Equation]
18) where P is the unique solution of (15).
On the other hand, for a ﬁxed time horizon T , as (cid:15) shrinks the discrete-time Riccati equation in (14) becomes a continuous-time Riccati equation as follows.
First, we have
[Equation]
Using these limits, letting (cid:15) → 0 in (14) leads to lim
[Equation]
That is [Equation] the backward differential equation
[Equation]
) with the terminal condition P T = 0 . Thus
[Equation]
as (cid:15) → [Equation] the optimal policy becomes
[Equation]
, where P t is the solution of (19).
Similarly, letting (cid:15) → 0 in (15) [Equation] we get the optimal policy
U (cid:63)t =
L ( A (cid:63) , B (cid:63) )
X t for minimizing the inﬁnite horizon expected average cost, where
[Equation]
B (cid:62) (cid:63) K ( A (cid:63) , B (cid:63) ) , and K ( A (cid:63) , B (cid:63) ) solves Φ A (cid:63) ,B (cid:63) ( P ) = 0 .
Equivalently, letting P t,T be the solution of (19) when the time horizon is T , it holds that lim T →∞ P 0 ,T =
K ( A (cid:63) , B (cid:63) ) , where K ( A (cid:63) , B (cid:63) ) solves Φ A (cid:63) ,B (cid:63) ( P )
= 0 .
Note that all these relationships rely on the convergence of discrete time Riccati equation (14) to the algebraic Riccati equation (15), as T → ∞ .
Next, subtracting V ( k +1) (cid:15) ( X k(cid:15) ) from both sides of (10), dividing by (cid:15) , and letting (cid:15) → 0 , Ito Isomery implies that
[Equation]
where we used the limits of the matrices (cid:101) Q, (cid:101) G, (cid:101) R as (cid:15) → 0 to ﬁnd the expression on the right-handside of the above equality.
Note that the above partial derivatives exist according to (13) together with Dominated Convergence Theorem.
Hence, substituting for d X t from the dynamics (1), and leveraging Ito’s Lemma, we obtain the Hamilton-Jacobi-Bellman [41] equation
[Equation]
Further, letting (cid:15) → 0 , the expression in (13) gives
[Equation]
where P t solve (19).
This can be equivalently obtained using the fact that a quadratic function of the form V t ( X t ) =
X (cid:62) t F t X t + ϕ t solves the partial differential equation (20) [Equation] as long as
[Equation]
which after solving for U t gives the optimal policy U (cid:63)t = − R − 1 B (cid:62) (cid:63) F t X t , as well as
[Equation]
Because the equation above needs to hold for an arbitrary X t , it splits to
[Equation]
that is, F t solves (19).
Further, note that cost-to-go at time T is zero because time-to-go is zero, which provides the terminal condition V T ( X T ) = 0 , implying that ϕ t = T (cid:82) t tr [Equation] CC
[Equation] F s (cid:1) d s . Therefore, the solutions F t , ϕ t of (20) lead to the same expression as in (21).
Finally, the expected average cost of the policy U t =
L ( A (cid:63) , B (cid:63) )
X t is the limit of the expected average cost of the policy U t = − R
[Equation]
t
[Equation]
, as
[Equation]
Moreover, suppose that C = 0 , and apply the policy U t =
L ( A
[Equation]
B [Equation]
X t .
Then, the state trajectory becomes X t =
e D (cid:63) t X 0 , where D (cid:63) =
A (cid:63) + B
[Equation]
L ( A (cid:63) , B (cid:63) ) .
So, by (21), we have
[Equation]
for an arbitrary initial state X 0 .
Thus, (6) holds:
[Equation]
Since Q is positive deﬁnite, the above equality implies that λ ( D (cid:63) ) < 0 , as well as
[Equation]
So far, we have shown that by restricting our search for an optimal decision-making policy to the class of policies that the control action is frozen during intervals of length (cid:15) , and then letting (cid:15) decay to vanish, we obtain optimal policies given by (19).
Next, we show that these policies are optimal in the larger class of all control policies satisfying the information criteria at every time.
That is, for all t , the control action U t can be determined using F t = σ ( X 0: t , U 0: t ) .
For this purpose, ﬁrst note that the decision-making policy U t
=
R − 1 B
[Equation]
K
[Equation]
A [Equation] , B [Equation]
X t provides an upper-bound for the optimal expected average cost.
That is,
[Equation]
Now, suppose that there is another policy, denoted by (cid:101) π , that satisﬁes
J (cid:101) π ≤ tr (cid:0) CC (cid:62) K
( A (cid:63) , B (cid:63) ) (cid:1) .
Deﬁne cost-to-go of the policy (cid:101) π by
[Equation]
where T is large enough to satisfy
[Equation]
tr
[Equation] CC [Equation] K ( A
[Equation]
, B
[Equation]
) (cid:1) , for all 0 ≤ t ≤ 1 .
Note that such T exists since (cid:101)
π provides a smaller expected average cost than the policy U t
=
R − 1 B (cid:62) (cid:63) K
( A (cid:63) , B (cid:63) )
X t , and the desired upper-bound for (cid:101) V t
( X t ) is 2 V t ( X t ) ; two times the cost-to-go of the policy U t =
R − 1 B (cid:62) (cid:63) K ( A (cid:63) , B (cid:63) )
X t .
Next, writing
[Equation]
subtract (cid:101) V t + (cid:15) ( X t ) from both sides, and divide by (cid:15) .
Letting (cid:15) decay to zero, the upper-bound for (cid:101) V t
( X t ) in terms of V t ( X t ) implies that according to Dominated Convergence Theorem, the following derivatives exist and it holds that
[Equation]
Now, note that since c ( X t , U t ) as well as B (cid:63) U t are continuous functions of U t , the above partial differential equation for (cid:101) V t
( X t ) indicates that (cid:101) π ( F t ) is a continuous function of X t .
This, together with the fact that W t is an almost surely continuous function of time t , in lights of the dynamics equation in (1), leads to continuity of state trajectory X t ; i.e., U t = (cid:101) π ( F t ) is continuous as t varies.
Thus, decision-making policies that freeze for (cid:15) -length intervals provide accurate approximations of U t = (cid:101) π ( F t ) in a sense that there exists a sequence (cid:110) U ( n ) t (cid:111) ∞ n =1 such that U ( n ) t freezes during intervals of the length 1 /n , and it holds that
[Equation]
Therefore, we have
[Equation]
where the inner inﬁmum is taken over all policies that freeze during (cid:15) -length intervals.
This shows that the policy U t = − R − 1 B [Equation] (cid:63) K [Equation]
A (cid:63) , B (cid:63) )
X t is an optimal one, which completes the proof.
C Proof of Theorem 2 (Stability margin)
First, we study eigenvalues of the sum of two matrices.
Suppose that M, ∆ are arbitrary square matrices of the same size, and let
M =
P − 1 Λ P be the Jordan decomposition.
That is, λ 1 , · · · , λ k are eigenvalues of M , Λ ∈ C d
X ×
d X is a block diagonal matrix with blocks Λ 1 , · · · , Λ k , and
[Equation]
i [Equation] [Equation] Further, similar to Deﬁnition 2, let µ M = max 1 ≤ i ≤ k µ i .
We prove that λ ( M − ∆) is at most
[Equation]
To show the above inequality, ﬁrst let λ be an eigenvalue of M − ∆ that satisﬁes (cid:60) ( λ ) > λ ( M ) .
So, M − λI is an invertible matirx, and there exists at least one vector v , such that v (cid:54) =
and ( M − ∆ − λI ) P − 1 v = 0 .
Then, ( M − λI ) P − 1 v = ∆ P − 1 v implies that
[Equation]
[Equation] Because Λ = diag (Λ 1 , · · · , Λ k ) , the matrix Λ − λI is block diagonal as well, and we have
[Equation]
Λ − [Equation] = diag (cid:16)(cid:0)
[Equation]
1 , · · · , (cid:0)
[Equation]
− λI µ k (cid:1) − 1 (cid:17) .
Further, it is straightforward to see that (cid:0) Λ i − λI µ i (cid:1) − 1 is
[Equation]

[Equation]
Therefore, we have
[Equation]
Using this bound for the operator norms of blocks of the block-diagonal matrix (Λ − λI ) − 1 , since µ i ≤ µ M and (cid:60) ( λ ) > λ ( M ) , the equation in (26) implies
[Equation]
[Table]
So, letting λ be an eigenvalue of M − ∆ that satisﬁes (cid:60) ( λ ) = λ ( M − ∆) , we obtain (25)
[Table]
Now, using (25) [Table] we compare A (cid:63) + B (cid:63) L ( A, B ) and D =
A + B L ( A, B ) .
Since A (cid:63) + B (cid:63) L ( A, B ) −
D is
[Table]
, (27) using (5), and letting M = D in (25), we have λ
[Equation]
A
[Equation]
[Equation] P −
[Equation]
|
[Equation]
(cid:63)
[Equation]
|| [Equation] 1 / 2 D (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) P − 1 (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ||| P |||||| ∆ (cid:63) ||| (cid:17) 1 / µ D .
So, in order to have λ
( A (cid:63) + B (cid:63) L ( A, B ))
< − δ , it sufﬁces to show that
[Equation]
||| P |||||| ∆
[Equation]
||| <
[Equation] ∧ [Equation] However, since ||| ∆ (cid:63) ||| ≤ E ( A, B ) (cid:16) 1 ∨ ||| B ||| ζ λ min ( R ) (cid:17) , (7) provides (28), which leads to the desired result.
C.1 Proof of sufﬁciency of (8) for stabilization bounds
Next, we show that E ( A, B ) ≤ (cid:15) 0 is sufﬁcient for stabilization and express uniform bounds for ρ, ζ in (5).
Let D (cid:63) =
A (cid:63) + B (cid:63) L ( A (cid:63) , B (cid:63) ) =
P − 1 (cid:63) Λ (cid:63) P (cid:63) be the Jordan decomposition as deﬁned in the beginning of the proof [Equation] and deﬁne the largest block size µ (cid:63) = µ D (cid:63) , similar to Deﬁnition 2.
Further, suppose that the following is satisﬁed:
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table]
[Equation]
The inequality in (29) implies that if we write D 1 =
A
[Equation]
B
[Equation]
+∆ 1 =
D (cid:63) + ∆ 1 , then, the matrix ∆ 1 =
A − A [Equation]
satisﬁes
[Equation]
So, taking M = D (cid:63) , the bound in (25) implies that λ ( D 1 ) < 0 .
Hence, we can employ Lemma 4 to study consequences of applying the linear feedback L ( A (cid:63) , B (cid:63) ) to a system of dynamics matrices
A, B , and get
[Equation]
where
[Equation]
Above, we used the fact that the initial state X 0 = x in Lemma 4 is arbitrary, and so, the involved matrices are themselves equal.
Further, similar to Lemma 4, it is straightforward to see that
[Equation]
This leads to
[Equation]
Because λ ( D (cid:63) ) <
[Equation]
the latter equation and (6) provide
[Equation]
Therefore
[Equation]
it holds that ||| M ||| ≤ |||K ( A (cid:63) , B (cid:63) ) ||| +2 ||| ∆ 1 |||||| M ||| ∞ (cid:82)
[Equation]
e D (cid:63) t (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) 2 d t , which, according to (29) and K ( A, B ) ≤ M , yields to
[Equation]
To proceed, suppose that v ∈ C d X satisﬁes || v || = 1 and Dv = λv .
Now, (6) implies that
[Equation]
where v ∗ is the transposed complex conjugate of v .
Thus, maximizing the left-hand-side above while taking minimum on the right-hand-side, it holds that
[Equation]
Putting (30) and (31) together, we obtain λ ( D ) ≤ − λ [Equation] ( Q ) (4 |||K ( A (cid:63) , B (cid:63) ) ||| )
− 1 .
This and (30) imply that E ( A, B ) ≤ (cid:15) 0 is sufﬁcient for (5), with ρ = λ min ( Q ) 4 − 1 |||K ( A (cid:63) , B (cid:63) ) ||| − 1 , ζ = 2 |||K ( A (cid:63) , B (cid:63) ) ||| .
[Equation]
D Proof of Theorem 3 (Regret analysis)
Let M = Q + L
( A (cid:63) , B (cid:63) ) (cid:62) R L ( A (cid:63) , B (cid:63) ) .
Recall that π applies U t =
L t X t at time t .
Now, for a given T , suppose that (cid:15) > 0 is a ﬁxed small real, and let N = (cid:100) T/(cid:15) (cid:101) .
Then, deﬁne the sequence of policies { π i } Ni =0 :
π i = (cid:26) U t =
L t X t t < i(cid:15)
U t =
L ( A (cid:63) , B (cid:63) )
X t t ≥ i(cid:15) .
Note that as long as one concerns about times t ≤ T , it holds that π (cid:63) =
π 0 , π N = π .
Clearly, since R π 0 ( T ) = 0 , we have R π ( T ) =
N − 1 (cid:80) i = (cid:0) R
π i +1 ( T )
− R
π
i ( T ) (cid:1) .
Thus, Lemma 1 gives
R π ( T ) =
N − 1 (cid:80) i = (cid:0) X (cid:62) i(cid:15) F i(cid:15)
X i(cid:15)
+ 2 X (cid:62) i(cid:15)
g i(cid:15) + β i(cid:15) (cid:1) , where the matrix F i(cid:15) , the vector g i(cid:15) , and the scalar β i(cid:15) are deﬁned in (41), (42), and (43), respectively.
Now, letting (cid:15) → 0 , since L t is piecewise continuous, we have R [Equation] [Equation]
=
[Equation]
32) where (cid:101) F t = [Equation] →
[Equation]
(cid:15) −
[Equation]
F i(cid:15) , (cid:101) g
[Equation] = lim (cid:15) [Equation]
0 ,i(cid:15) → t (cid:15)
[Equation]
1 g i(cid:15) , and (cid:101) β t = lim (cid:15) → 0 ,i(cid:15) → t (cid:15) − 1 β i(cid:15) .
Note that the above limits exist, since F i(cid:15) , g i(cid:15) , β i(cid:15) are continuous.
To calculate (cid:101) F t , (cid:101) g t , (cid:101) β t , using Lemma 1 and the piecewise continuity of L t , we obtain (cid:101) β t = 0 ,
[Equation]
where S t = L (cid:62) t RL t − L ( A (cid:63) , B (cid:63) ) (cid:62) R L ( A (cid:63) , B (cid:63) ) , and
H
[Equation]
= lim
[Equation]
A
[Equation]
−
[Equation]
D
[Equation]
=
[Equation]
Now, by (6) and ∞ (cid:82)
[Equation]
, the expression for (cid:101) F t becomes
[Equation]
So, after doing some algebra (see (51)), we get
[Equation]
Since W u has independent increments and in (cid:101) g t we have u ≥ t , Fubini’s Theorem gives
[Equation]
Hence, (32), (33), (34), and Fubini’s Theorem imply that E
[ R π ( T )] =
E [ α T ] .
To proceed towards establishing the second result, apply Stochastic Fubini Theorem [34, 4] to get
T (cid:90)
X
[Equation]
where, using the expression for H t [Equation] the vector Y u can be written as
[Equation]
for P
[Equation]
,u =
[Equation]
C d s .
Now, letting V T
= T (cid:82) ||
Y u
|| 2 d u , for V T < 1 , Ito Isometry [4], and for V T ≥ 1 , Lemma 2, imply that
[Equation]
However, by using the triangle inequality and Fubini’s Theorem
[Equation]
we obtain [Equation]  
[Equation]
The second part of the integrand above appears in α T .
So, we proceed by ﬁnding an upper-bound for the ﬁrst part.
For this purpose, we use the triangle inequality and (6) to get the equation λ max 
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
Therefore, by using (31), we get
[Equation]
since E t decays exponentially with t .
So, (35) gives the desired result.
[Equation]
E Proof of Theorem 4 (Analysis of Algorithm 1)
In order to establish Theorem 4, we study the estimation procedure in (9) and specify the accuracy at which the algorithm is able to estimate A (cid:63) , B (cid:63) .
To that end, Lemma 5 and Lemma 6 are utilized to study the Gram matrix V n = [Equation] n (cid:82) [Equation]
Y s Y (cid:62) s
d s in (9), while Lemma 2 is used for bounding the estimation error.
Then, by leveraging Lemma 3, we ﬁnd the rates of deviating from the optimal policy in (3).
Finally, the resulting regret of Algorithm 1 is investigated in lights of Theorem 3.
By using (1) to substitute for d X t ,
[Equation]
is
[Equation]
In (38), we show that V n is non-singular.
So, we have
[Equation]
Because [ A (cid:63) , B (cid:63) ] ∈ S 0 , (9) and (36) lead to
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
Since entries of Θ n are N (cid:0) 0 , γ − n/ 2 n 1 / 2 (cid:1) , we have
[Equation]
This, by Borel-Cantelli Lemma, leads to
[Equation]
Thus, letting d = d X + d U , according to Lemma 2, E ( A n , B n ) is at most
[Equation]
Now, Lemma 5 provides O (log λ max ( V n )) = n log γ .
Further, we will shortly show that
[Equation]
Thus, (37) and (38) yield to the upper-bound
[Equation]
This gives the ﬁrst result in Theorem 4.
To prove the other statement, let β (cid:63) be as deﬁned in Lemma 3.
So, Lemma 3 implies that
[Equation]
Now, since during the time period γ n − 1 ≤ t < γ n the feedback matrix is frozen to L ( A n − 1 , B n − 1 ) , according to Lemma [Equation] we have
[Equation]
where
[Equation]
Moreover, since by Theorem 1 we have λ ( D (cid:63) ) < 0 , the matrix E t in Theorem 3 decays exponentially with t .
So, it holds that
[Equation]
Therefore, according to Theorem 3, we have the following:
[Equation]
This, according to β (cid:63) in Lemma 3, completes the proof.
To prove (38), let D k − 1 =
A (cid:63) + B (cid:63) L ( A k − 1 , B k − 1 ) .
Then, by Lemma 5, we have
[Equation]
where η k =
[Equation]
d s (cid:19) − 1 .
Hence, (39) implies that to establish (38), it sufﬁces to show that the following inequality holds for some 0 ≤ (cid:96) < n − 1 : [Equation]
[Equation]
For an arbitrary ﬁxed (cid:15) > 0 , consider the event that the above-mentioned smallest eigenvalue is less than (cid:15) , and let M n ( (cid:15) ) be the set of matrices [ A k , B k ] n − 1 k = (cid:96) for which this event occurs:
[Equation]
where the ( d X + d U ) ×
d X ( n − (cid:96) ) matrix P (cid:96),n is
[Equation]
Now, note that the set of all matrices
[Equation]
2 − n/ [-
-]4 I d X · · · γ ( n − 1) / 2 − n/ 4 I d X γ (cid:96)/ 2 − n/ 4 L (cid:96) · · · γ ( n − 1) / 2 − n/ 4 L n −(cid:21) , that there exists v ∈ R d X + d U satisfying || v || = 1 and F (cid:62) n v = 0 , is of dimension d X + d U − 1 + ( n − (cid:96) )
( d U − 1) .
To show that, on one hand, the set of unit d X + d U dimensional vectors is (a sphere) of dimension d X + d U − 1 .
On the other hand, by writing v = (cid:2) v (cid:62) 1 , v (cid:62) 2 (cid:3) (cid:62) , for v 1 ∈ R
d
X
and v 2 ∈ R d U , clearly, F (cid:62) n v = 0 is equivalent to L (cid:62) k v 2 = − v 1 , for all k = (cid:96), · · · , n − 1 .
The latter enforces every column of L k to be in a certain hyperplane in R d U .
Thus, according to Lemma 6, the dimension of M n (0) is at most d X
+( d U − 1)( n − (cid:96) +1)+( n − (cid:96) )
d 2 X .
Further, if (cid:96) is sufﬁciently large so that γ − (cid:96) + n/ 2 (cid:15) < 1 , then for every [ A k , B k ] n − 1 k = (cid:96) ∈
M n ( (cid:15) ) , there exists some (cid:104) (cid:101)
A k , (cid:101) B k (cid:105) n −k = (cid:96) ∈
M n (0) , such that for all k = (cid:96), · · · , n − 1 , it holds that (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Table]
The random matrices { Θ k } n − 1 k =0 are independent, and entries of Θ k are independent identically distributed N
[Equation]
0 , γ
[Equation] [Equation]
random variables.
Hence, we have
[Equation]
where m = ( d X d U − d U + 1) ( n − (cid:96) ) − d X − d U + 1 .
To see that, note that M n (0) is a d X + ( d U − 1)( n − (cid:96) + 1) + ( n − (cid:96) )
d 2 X dimensional object in a d X ( d X + d U )( n − (cid:96) ) dimensional space.
So, the exponent is at least m .
Letting (cid:96) =
n − 5 , we have m ≥ 5 .
Further, as n grows, O (cid:0) (cid:96) − 1 / 4 γ ( n − (cid:96)
[Equation]
[Equation]
(cid:1) < 1 holds for (cid:15) = max
[Equation]
k − 1 .
So, we have
[Equation]
[Equation] = ∞ (cid:80) n =5 O (cid:0) n
[Equation]
(cid:1) 5 < ∞ , which by Borel-Cantelli Lemma implies (40).
[Equation]
F Estimation Rates under Persistent Randomization
Proposition 1 Assume that in Algorithm 1 the variance of entries of Θ n is σ 2 n , where [Equation]
Then, letting ω E be as in Theorem 4, Y s = (cid:2) X (cid:62) s , U (cid:62) t (cid:3) (cid:62) , and V n = γ n (cid:82)
Y s Y (cid:62) s d s , we have (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) [Equation] (cid:90)
[Equation] (cid:62) [Equation]

 (cid:62)
V
[Equation] (cid:63) [Equation] (cid:63) ] (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
O (cid:0)
[Equation] (cid:1) .
Proof.
By (36), it sufﬁces to study ∆ =
V † n γ n
[Equation]
s C (cid:62) .
In the sequel, we show that
[Equation]
So, putting Lemma 2 and Lemma 5 together, we obtain the desired result, since they give
[Equation]
Thus, by (39), it is enough to show that for some 0 ≤ (cid:96) < n − 1 ,
[Equation]
is at least (cid:15) = max
[Equation]
≤ k ≤ n − 1 η k − 1 .
Let M n ( (cid:15) ) be the set of [ A k , B k ] n − 1 k = (cid:96) that the above does not hold:
M
[Equation]
λ min
[Equation]
(cid:17) ≤ (cid:15) (cid:111) , where P (cid:96),n is
[Equation]
Similar to the proof of Theorem 4, for [ A k , B k ] n − 1 k = (cid:96) ∈
M n ( (cid:15) ) , there is (cid:104) (cid:101)
A k , (cid:101) B k (cid:105) n − 1 k = (cid:96) ∈
M n (0) , such that (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[ A k , B k ] − (cid:104) (cid:101)
A k , (cid:101) B k
(cid:105)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) 2 =
O (cid:0) γ n − k n − 1 (cid:15) (cid:1) .
Thus, lim inf n →∞ σ n > 0 , together with the dimension of M n (0) that we calculated in the proof of Theorem 4, leads to
[Equation]
for m =
( d X d U − d U + 1) ( n − (cid:96) ) − d X − d U + 1 .
Finally, (cid:96) =
n − 4 gives ∞ (cid:80) n =4 P ( M n ( (cid:15) )) < ∞ .
Therefore, Borel-Cantelli Lemma implies the desired result.
(cid:4)
G Auxiliary Lemmas
In this section, we state the auxiliary lemmas used for establishing the main results and provide their proofs, each subsection corresponding to one lemma.
First, in Lemma 1 in Subsection G.1, we provide expressions for the difference between the regrets of two policies.
Study of self-normalized stochastic integrals is the content of Lemma 2, while Lemmaon Lipschitz continuity of the optimal feedback with respect to the dynamics matrices is established in Subsection G.3.
Next, in Lemma 4, we consider the total cumulative cost for the case of applying a sub-optimal time-invariant linear feedback policy to a deterministic system.
Then, Lemma 5 focuses on explicit calculation of the empirical covariance matrix of the state vectors.
Finally, in Lemma 6 in Subsection G.6 we specify the set of dynamics matrices that possess the same optimal linear feedback matrix.
G.1 Difference in regrets of two policies
Lemma 1 For ﬁxed 0 ≤ t 1 ≤ t 2 ≤ T , deﬁne the policies π 1 , π 2 according to
[Equation]
Further, let D (cid:63) =
A (cid:63) + B (cid:63) L ( A (cid:63) , B (cid:63) ) , D =
A (cid:63) + B (cid:63) L , M (cid:63) =
Q + L ( A (cid:63) , B (cid:63) ) (cid:62) R L ( A (cid:63) , B (cid:63) ) ,
M = Q + LRL , ∆ t = [Equation] D (
[Equation]
1 ) − e
[Equation]
(cid:63) (
[Equation] 1 ) , Z t = t (cid:82) [Equation] (cid:2)
[Equation]
( t
[Equation]
− e D (cid:63) ( t
[Equation]
s
[Equation]
(cid:3) C d W s , and S =
M − M (cid:63) =
L (cid:62)
[Equation]
[Equation]
(cid:63)
[Equation]
(cid:63)
[Equation] Then [Equation] we have R π
[Equation]
T ) − R
[Equation]
[Equation]
T
[Equation]
=
X
[Equation]
F
[Equation]
1 X t 1 + 2 X (cid:62) t 1 g t 1 + β t 1 , where F t 1 , g t 1 , and β t 1 are
[Equation]
[Equation]
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12)
[Equation]
[Equation]
Proof.
Letting X
π i t be the state of the system under the policy
π i , clearly, for t ≤ t 1 , it holds that
X π 1 t =
X π 2 t .
So, we use X t 1 for both states at time t 1 .
Moreover, for t 1 ≤ t ≤ t 2 , we have
[Equation]
where Y t = X
π 2 t − X π 1 t .
So, by denoting the instantaneous cost of policy π i at time t by c π i ( t ) , we get Y t =
∆ t X t 1 + Z t , as well as
[Equation]
On the other hand, for t ≥ t 2 , we have
T
[Equation]
and
[Equation]
Thus, putting (44) and (45) together, we obtain the desired results.
(cid:4)
G.2 Scaling of self-normalized stochastic integrals
Lemma 2 Suppose that Y t ∈ R m is a vector-valued stochastic process such that Y t is F t -measurable for the natural ﬁltration F t = σ (cid:16) { W s } 0 ≤ s ≤ t (cid:17) .
Then, letting V t =
t (cid:82)
Y s Y (cid:62) s d s , we have (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ( I
[Equation]
[Equation]
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) =
O ( md W log λ max ( V t )) .
Proof.
First, ﬁx t > 0 , and for an arbitrary (cid:15) > 0 , let n = (cid:98) t/(cid:15) (cid:99) .
So, for k = 0 , 1 , · · · , n , consider the sequence of matrices
M k
=
(cid:15) −
I + k (cid:80) i =
Y i(cid:15) Y (cid:62) i(cid:15) .
Then, for k = 1 , · · · , n , consider the sequence of scalars β k deﬁned according to β k = Y (cid:62) k(cid:15) M − 1 k − 1 Y k(cid:15) .
Using the formula for determinants of the products of matrices, we have
[Equation]
Since all eigenvalues of I + M − 1 k − 1 Y k(cid:15) Y
[Equation]
k(cid:15) are unit, except one of them which is 1 + β k , we have (1 + β k ) det M k
[Equation]
1 = det M k .
On the other hand, matrix inversion formula gives
[Equation]
which leads to
[Equation]
Further, by using the inequality 1 − β ≤ − log β for β > 0 , the latter equality gives
[Equation]
[Equation]
Now, let F k = k (cid:80)
[Equation]
Using the facts that Y k(cid:15) , F k − 1 , and M k all are F k(cid:15) - measurable, the Brownian motion W t has independent increments, and its covariance matrix is a multiple of identity, properties of conditional expectations give
[Equation]
So
[Equation]
using (46) together with the fact that (as positive semideﬁnite matrices) the order M k − 1 ≤ M k holds, we get the telescopic relationships [Equation] ( (cid:15)M
[Equation]
1 ) (cid:19) .
Since F (cid:62) k ( M k ) − 1 F k is positive semideﬁnite, its trace is larger than its largest eigenvalue.
Hence, adding up for k = 0 , 1 , · · · , n , by interchanging trace and expectation, we obtain E
[Equation]
(cid:105)(cid:17) , which, by (cid:15)M 0 ≥ I , leads to
[Equation]
Thus, according to Doob’s Martingale Convergence Theorem [34, 4], we have
[Equation]
Finally, letting (cid:15) → 0 , we obtain the desired result, because (cid:15)M n , F n are (cid:15) -approximations of the corresponding integrals.
(cid:4)
G.3 Lipschitz continuity of optimal feedback
Lemma 3 Using the Jordan decomposition D (cid:63) =
A (cid:63) + B (cid:63) L ( A (cid:63) , B (cid:63) ) =
P − 1 (cid:63) Λ (cid:63) P (cid:63) , deﬁne µ (cid:63) =
µ D (cid:63) , similar to Deﬁnition 2, and suppose that E ( A, B ) ≤ κ (cid:63) , for κ (cid:63) =
[Equation]
Then, letting
[Equation]
, we have
[Equation]
In general, without the condition E ( A, B ) ≤ κ (cid:63) , the constant β (cid:63) is replaced with
[Equation]
for some convex combination [ A 0 , B 0 ] =
η [ A, B ] + (1 − η )
[ A (cid:63) , B (cid:63) ] , and 0 ≤ η ≤ 1 .
Proof.
Fix the matrices A, B , and consider the matrix-valued curve
[Equation]
For an arbitrary A 0 , B 0 ∈ ϕ , we ﬁnd the derivative of the matrix K
( A 0 , B 0 ) at A 0 , B 0 , assuming that the matrices A 0 , B 0 vary along ϕ .
For this purpose, letting ∆ A
=
A − A (cid:63) , ∆ B = B − B (cid:63) , we ﬁrst calculate K ( A 1 , B 1 ) for A {+
+} =
A 0 + η ∆ A , B 1 = B 0 + η ∆ B , and then let η → 0 .
First, letting
P = K
( A 1 , B 1 ) − K ( A 0 , B 0 ) , we get
[Equation]
The above expression, because of
[Equation]
implies that the followings hold true:
[Equation]
By plugging (47) and A
[Equation]
=
A
[Equation]
in Φ A i ,B i ( K ( A i , B i ))
=
for i = 0 , 1 , we obtain 0 = (cid:2) A [Equation] B
[Equation]
B
[Equation]
B
[Equation]
(cid:3)
[Equation]
+
[Equation]
or equivalently,
[Equation]
for
[Equation]
= A
[Equation]
B
[Equation]
B
[Equation]
K
[Equation]
, B
[Equation]
, and
[Equation]
Suppose that η is sufﬁciently small so that λ (cid:16) (cid:101) A (cid:17) < 0 .
Note that it is possible thanks to stabilizability of A 0 , B 0 , Theorem 1, and lim η → 0 (cid:101)
A =
A 0 + B
L
A 0 , B 0 )
=
D 0 .
So, since PB 1 R − 1 B (cid:62) 1 P is a positive semideﬁnite matrix, (48) implies that
[Equation]
This, because of lim η → 0 (cid:101) Q = 0 , leads to lim η → 0 P = 0 .
Thus, letting M = ∆ A + ∆ B L ( A 0 , B 0 ) and ∆ K ( A 0 ,B 0 ) =
lim η → 0 η − 1 P , (48) gives the following for ∆ K ( A 0 ,B 0 ) :
[Equation]
By
[Equation]
(31), (49), and the Cauchy-Schwarz inequality provide
[Equation]
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
Next, note that E ( A, B ) ≤ κ (cid:63) , together with (29) and (30), implies that
[Equation]
Therefore
[Equation]
using (30)
[Equation]
and putting the above inequality together with
[Equation]
we get the ﬁrst desired result.
To establish the second result, it sufﬁces to let A 0 , B 0 be the one for which the above supremum over ϕ is achieved.
(cid:4)
Lemma 4 Consider a noiseless linear dynamical system with the stabilizable dynamics matrices
A, B .
That is, d X t =
( AX t + BU t ) d t , starting from X 0 = x .
Then, if we apply the linear feedback
U t = LX t , as long as λ ( A + BL ) < 0 , it holds that
[Equation]
Proof.
Denote D 1 =
A + B L ( A, B ) and D 2 =
A + BL .
So, the dynamics equation d X t = ( AX t + BLX t ) d t implies that X t =
e D 2 t x , which leads to
[Equation]
where
[Equation]
which yields to
[Equation]
Similar to (23), it holds
that D (cid:62) 1 K ( A, B ) + K ( A, B ) D 1 + Q + L
( A, B ) (cid:62) R L ( A, B ) .
So, subtracting the latter two equalities, we get
[Equation]
where
[Equation]
Because λ ( D 2 ) < 0 , solving (50) for P −K ( A, B ) , and using the fact D 2 − D 1 = B
[ L − L ( A, B )] , we have
[Equation]
where
[Equation]
Then, using B (cid:62) K ( A, B ) = − R L ( A, B ) , after doing some algebra we obtain
[Equation]
Thus, P − K ( A, B ) is
[Equation]
which implies the desired result.
[Equation]
G.5 Convergence of empirical covariance matrix of the state vectors
Lemma 5 Suppose that for t ≥ γ , the linear feedback L is applied to the system (1) such that λ ( D ) < 0 , where D =
A (cid:63) + B (cid:63) L .
Then, we have
[Equation]
Proof.
First, denote
[Equation]
Then, deﬁne the matrix Y t =
X t X (cid:62) t , and apply Ito’s Formula [4] to ﬁnd d Y t :
[Equation]
Plugging in for d X t from (1), we obtain
[Equation]
where we used the facts d t d t = 0 , d W t d t = 0 , and Ito Isometry d W t d W (cid:62) t
= d tI d W [4].
Thus, we have
[Equation]
where
[Equation]
This can equivalently be written as
[Equation]
Since λ ( D ) < 0 , the latter equality implies that V T is
[Equation]
Now, according to the following statements, the above leads to the desired result, because the terms corresponding to M γ,T , X γ , X γ + T vanish as T grows.
Clearly, it holds that lim T →∞
[Equation]
[Equation]
Since λ ( D ) < 0 , the expression
[Equation]
implies that lim T →∞ T − 1 / 2 ||
X γ + T
||
= 0 .
Putting λ
( D ) < 0 together with Doob’s Martingale Convergence Theorem [34, 4], we get
[Equation]
G.6 Manifolds of dynamical systems with equal optimal feedback matrices
Lemma 6 Consider the set of dynamics matrices A, B that share optimal feedback with A 0 , B 0 :
[Equation]
Then, M 0 is a manifold of dimension d 2 X .
Proof.
Suppose that for the matrix [ A, B ] =
[ A 0 , B 0 ]+ (cid:15) [ M, N ] , it holds that L ( A, B ) =
L ( A 0 , B 0 ) .
We ﬁnd the derivative of L ( A 0 , B 0 ) along the direction [ M, N ] .
First, using the expressions in (23) for A, B and for A 0 , B 0 , we get
[Equation]
where D
=
A 0 + B 0 L ( A 0 , B 0 ) .
Simplifying the above expressions and letting (cid:15) → 0 , for the matrix
[Equation]
we have
[Equation]
Thus, since according to Theorem 1, λ ( D 0 ) < 0 , it yields to
[Equation]
where
[Equation]
On the other hand, L ( A, B ) = − R − 1 B (cid:62) K ( A, B ) gives
[Equation]
So, M 0 is a manifold, and its tangent space consists of matrices M, N satisfying the above equation.
To ﬁnd the dimension, select a d X × d X matrix P arbitrarily, and let N be
[Equation]
Note that since λ min ( Q ) > 0 , the inverse K ( A 0 , B 0 ) − 1 exists.
Then, solve for M according to
M + N L ( A 0 , B 0 ) =
P .
Therefore, the matrices M, N satisfy in N (cid:62) K ( A 0 , B 0 ) + B (cid:62)
∆ = 0 , and so correspond to a member of M 0 .
Conversely, every matrices M, N in the tangent space of Mprovide a d X × d X matrix P
=
M + N L ( A 0 , B 0 ) such that N (cid:62) K
( A 0 , B 0 )
+ B (cid:62)
∆ = 0 .
Thus,
M 0 is of dimension d 2 X , which is the desired result.
[Equation]