Augmented RBMLE-UCB Approach for Adaptive Control of Linear Quadratic Systems
Anonymous Author(s)
AffiliationAddress email
Abstract
We consider the problem of controlling a stochastic linear system with quadraticcosts, when its system parameters are not known to the agent – called the adaptive LQ control problem.
We re-examine an approach called “Reward-Biased
Maximum Likelihood Estimate” (RBMLE) [1] that was proposed more than fortyyears ago, and which predates the “Upper Confidence Bound” (UCB) method aswell as the definition of “regret" [2].
It simply added a term favoring parameterswith larger rewards to the criterion for parameter estimation.
We show how the
RBMLE and UCB methods can be reconciled, and thereby propose an Augmented
RBMLE-UCB algorithm that combines the penalty of the RBMLE method withthe constraint of the UCB method [3], uniting the two approaches to optimism inthe face of uncertainty.
We establish that theoretically this method retains ˜ O ( √ T )regret, the best known so far.
We further compare the empirical performance of theproposed Augmented RBMLE-UCB and the standard RBMLE (without the augmentation) with UCB (OFU), Thompson Sampling (TS), Input Perturbation (IP),
Randomized Certainty Equivalence (RCE) and StabL on many real-world examples that include flight control of Boeing 747 and Unmanned Aerial Vehicle.
Weperform extensive simulation studies showing that the Augmented RBMLE consistently outperforms OFU, TS and StabL by a huge margin, while it is marginallybetter than IP and moderately better than RCE.
Introduction
Consider a linear stochastic system x t +
=
A ⋆ x t + B ⋆ u t + w t
+1 , where x t ∈ R n , u t ∈ 21 R m and w t are the state, control applied, and state noise, respectively, at time t .
Westudy the adaptive control/reinforcement learning problem [4, 5, 6, 7] where the controller’sgoal is to minimize the expected finite horizon quadratic cost E (cid:80) Tt =1 (cid:0) x
Tt Qx t + u tT Ru t (cid:1) bychoosing the control input based on observing the past states and controls, without knowing the “true parameter" θ ⋆ :=
( A ⋆ , B ⋆ ) T .
The deviation from what would have beenoptimally possible had the true parameter been known is measured by the expected valueof the “regret," [2], R ( T ) :
= (cid:80) Tt =1 (cid:0) x
Tt Qx t + u Tt Ru t (cid:1) − T J
⋆
θ ⋆ ) , where J ⋆ ( θ ⋆ ) :
=
lim sup T →∞ 1 T E (cid:80) Tt =1 (cid:0) x
Tt Qx t + u Tt Ru t (cid:1) is the optimal average cost achievable when ( A ⋆ , B ⋆ )is known.
For this adaptive linear-quadratic (LQ) control problem, there are broadly four classical approaches,the Reward-Biased Maximum Likelihood Estimate (RBMLE) approach [1], the Diminishing Excitation (DE) approach [8], the Upper Confidence Bound (UCB) approach [2], and the Thompson
Sampling (TS) [9] approach based on sampling from the posterior distribution.
The RBMLE and
UCB approaches are “certainty equivalent” (CE) in the sense that they make an estimate ˆ θ t of the unknown true parameter, and then take an action u t =
K (ˆ θ t ) x
t that would be optimal if the estimateˆ θ t were indeed the true parameter.
They only differ in what parameter estimate they choose.
The DEapproach applies u t =
K (ˆ θ t ) x t + v t , where v t is an added “excitation,” an independent noise, ofdiminishing variance and ˆ θ t is the maximum likelihood estimate.
The recent scheme Randomized
Certainty Equivalence (RCE) adds the excitation to the parameter estimate.
1.1 The Contributions
We unite
the
two approaches to “optimism under uncertainty,” RBMLE [ and UCB [
by showing
that the RBMLE method is a penalty version of the constrained optimizationproblem of UCB
for
the case of linear quadratic systems [10, 11, 12, 13, 3]
Based
on thiswe propose an Augmented RBMLE (ARBMLE) method that combines the penalty andconstrained versions so that on the one hand it retains the analytical tractability of UCBand on the other hand provides the performance of RBMLE.
We determine how to choose the biasing factor α ( t ) for ARBMLE, and establish a finite
time regret bound ˜ O ( √ T ) , the same as the OFU algorithm of [3], the best order available
to date.
We perform extensive comparative simulation studies of the performance of:
(a) ARBMLE and the standard RBMLE.
(b) OFU [3], which is the UCB-approach adapted to the LQ problem.
(c) TS [14] which is the Thompson sampling approach adapted to the LQ problem
(d) Input Perturbation (IP) [15] which is a recent reincarnation of DE for the adaptive LQ
problem that additionally assumes a piori knowlege of a stabilizing controller.
(e) Stabl [16], a modified OFU that adds “excitation” to the input for initialization.
(f) RCE [17], which adds excitation to the Least Squares Estimate (LSE).
The examples used for our simulation study have been used in many recent papers[18, 19, 16], namely (a) the longitudinal flight control of Boeing 747 with linearized dynamics [16],(b) Unmanned Aerial Vehicle (UAV) [20, 16] (c) unstable Laplacian dynamics[18], and (d) large transient dynamics [18].
Our simulation results show that ARBMLEoutperforms OFU, TS and StabL by a large margin, which is primarily due to lack ofstabilization experienced by OFU and TS [16].
RCE also exhibits a higher regret than
ARBMLE.
While the empirical performance of IP is marginally worse than ARBMLE.
The results show that the ARBMLE has the same performance as the original RBMLE[21], and they both outperform all the above other algorithms.
Notably the choices madeby ARBMLE and OFU within the confidence interval are very far apart, with ARBMLEoutperforming OFU by a large margin.
1.2 Previous Works
Prior work on RBMLE has concentrated on establishing long-term average optimality [22, 23, 1,10, 24, 25, 11, 12, 13, 26, 27, 28].
Recently, its regret performance has been addressed for MultiArmed Bandits (MAB)s [29], linear contextual bandits [30], and MDPs [31].
Forced explorationtechniques, somewhat similar in spirit to the ϵ t -greedy learning algorithm of [7, 32], are studied in[33, 34, 35] vis-a-vis ensuring that they do not suffer from the insufficient exploration.
An adaptive
LQG control based on the UCB approach, called OFU, is proposed in [3] which establishes a regretof ˜ O ( √ T ) .
A similar algorithm to address the adaptive LQG control problem is also designed in[36].
A computationally efficient algorithm called ROBUST with a regret of ˜ O
( T 0 .
67 ) is proposed in[18].
An alternative approach for designing learning algorithms is Thompson sampling [9], and [37,38] have established an expected regret of ˜ O ( √ T ) in a Bayesian context.
More recently, new DEbased algorithms, IP [15] and RCE [17], have been proposed that make an additional assumption, notmade in ARBMLE, OFULQ or TS, that one has access to a stabilizing controller for the unknownsystem, and establish ˜ O ( √ T ) regret.
1.3 The RBMLE Approach
The RBMLE approach, proposed four decades ago [1], was the first approach not resorting to forcedchoices as in [39].
We begin by giving an informal description of it in the context of the adaptive
LQ problem.
Since the true parameter θ ⋆ := ( A ⋆ , B ⋆ ) T is not known one can make a Least-Squares
Estimate (LSE) ˆ θ t
= ( ˆ A t , ˆ B t ) T of them:
[Equation]
Under the certainty-equivalent approach, the control input applied is u t =
K (ˆ θ t ) x t where K ( θ ) isthe optimal linear feedback gain for the LQ problem when the system is described by θ =
( A, B ) T .
Suppose now that these estimates were to converge as t → ∞ to ˆ θ ∞ =
( ˆ A ∞ , ˆ B ∞ ) T .
Then, asymptotically, the input applied is u t ≈ K
(ˆ θ ∞ ) x t .
The closed-loop system therefore settles down tobehaving according to
[Equation]
As it does so, one loses the ability to identify the matrices ( A ⋆ , B ⋆ ) , and asymptotically one canonly identify the closed-loop gain A ⋆ + B ⋆ K
(ˆ θ ∞ ) .
The problem is that as the parameter estimatesbegin to converge to ˆ θ ∞ , the control gain converges to K (ˆ θ ∞ ) and further exploration ceases, andone ends up only identifying the behavior of the system under the limiting gain K ( ˆ θ ∞ ) being appliedto the system.
Since the limiting policy u = K (ˆ θ ∞ ) x need not be optimal for the long-term averagecost for the true system ( A ⋆ , B ⋆ ) , the CE rule leads to a sub-optimal performance.
Indeed thisproblem goes by various names in different fields – the dual control problem [40, 41], the closedidentifiability problem [42, 43, 44], or the problem of exploration vs. exploitation [45].
This problem was resolved in [1] without resorting to forced exploration as in [39].
The key observation made there was that under CE the limiting parameter estimate ˆ θ ( ∞ ) has a one-sided bias.
Specifically, J ⋆ (ˆ θ ∞ ) ≥ J ⋆ ( θ ⋆ ) , i.e., the limiting parameter estimate has an optimal cost that islarger than the optimal cost of θ ⋆ .
To see this, denote by J ( K, θ ) the long-term cost of using thecontrol gain K when the parameter is θ .
Then the fact that the models ˆ θ ∞ and θ ⋆ have identical behavior under the gain K (ˆ θ ∞ ) implies that J ( K (ˆ θ ∞ ) , ˆ θ ∞ ) =
J ( K (ˆ θ ∞ ) , ˆ θ ⋆ ) .
Now note that
J ( K (ˆ θ ∞ ) , ˆ θ ∞ ) =
J ⋆ (ˆ θ ∞ ) since the gain K (ˆ θ ( ∞ )) is optimal for ˆ θ ∞ .
However the gain K (ˆ θ ∞ ) isnot necessarily optimal for θ ⋆ , and so J ( K (ˆ θ ∞ ) , ˆ θ ⋆ ) ≥ J ⋆ ( θ ⋆ ) .
Therefore
[Equation]
Following from this observation, it was reasoned in [1] that if one could slightly bias the MLE tofavor models with lower optimal costs J ⋆ ( θ ) so as to obtain J ⋆ (ˆ θ ∞ )
≤ J ⋆ ( θ ⋆ ) , then one wouldhave equality throughout (3), to obtain J ( K (ˆ θ ∞ ) , ˆ θ ⋆ ) =
J ⋆ ( θ ⋆ ) , yielding the desired result that thegain K (ˆ θ ∞ ) is optimal for θ ⋆ .
So motivated, [1] proposed RBMLE 1 which in the context of the LSEsuggests choosing the parameter estimate ˆ θ RBMLE ( t )
=
( ˆ A RBMLE , ˆ B RBMLE ) T as:
[Equation]
Generally α ( t ) =
o ( t ) with the precise growth rate of α ( t ) ↗ + ∞ dependent on the context.
1.4 The UCB Approach
The ‘Upper Confidence Bound" (UCB) approach was first proposed in the context of Multi-Armed
Bandit (MAB) Problems in [2].
In the context of Bernoulli bandits, it essentially consists of constructing, for each arm i at each time t , a confidence interval ( θ ( i, 1; t ) , θ ( i, 2; t )) of its payoffprobability with confidence (1 − δ ( t )) , for δ ( t ) =
o ( 1 t ) , and playing the arm i with the highest valueof θ ( i, 1; t ) .
This approach has been generalized to a variety of contexts including linear contextualbandits [46], Gaussian Processes [47], MDPs [48, 49], and LQ systems [3].
Particularized to LQ systems, the OFU algorithm of [3] that is based on the UCB approach suggestschoosing ˆ θ UCB ( t ) =
( ˆ A UCB , ˆ B UCB ) T as the minimizer of:
[Equation]
, where
[Equation] [Equation]
is a (1 − δ )
-high confidence set of parameters for an appropriate choice of γ T ( δ ) [3].
The Augmented RBMLE-UCB Method
The UCB approach has been called “Optimism in the Face of Uncertainty" (OFU), since it choosesan optimistic arm after calculating the confidence intervals.
The RBMLE version is also the same,though it does it in a different way by directly giving preference to parameters that can yield betterrewards.
It arrives at optimism in a very systematic way by noticing that closed-loop identificationleads to the chain of inequalities (3), which could then be made into equalities by ensuring that
J ⋆ ( ˆ θ ∞ ) ≤ J ⋆ ( θ ⋆ ) .
We now show how one may reconcile the two approaches to optimism in the face of uncertainty inthe context of the LQ problem, and then combine them to obtain a method that has experimentallysuperior empirical performance while also allowing a proof that it achieves the currently best knownorder of regret.
It can be seen that the RBMLE approach (4) can be considered an unconstrained penalty versionof the constrained optimization problem (5)-(6), with a penalty factor 1 α ( t ) for constraint violation.
This provides a justification for employing “optimism" in the UCB approach.
This raises the question of whether we can take advantage of these synergies to in fact fashion a superior algorithm, that is superior from the viewpoint of being able to provide a theoretical guaranteeof the best known order of regret to date, as well as superior from the point of view of providingthe best experimental performance of the algorithms to date.
One can draw inspiration here fromthe Augmented Lagrangian Method of [50, 51].
Given a constrained problem as in (5)-(6), it adds apenalty for constraint violation, and also adjoins the constraint through a Lagrange multiplier, thusinvolving the constraint twice.
In our case, we add the penalty and also retain the constraint.
Thisleads to the Augmented RBMLE-UCB method:
[Equation]
The advantage of retaining the constrained optimization is theoretical, in that we can make use of thebounds on parameter estimates from concentration inequalities, as we shall show in the sequel.
Wethereby prove in Section 5 that it also has ˜ O ( √ T ) order of regret, which is the best known so far [3].
Moreover, simulations reported in Section 6 show that its performance is also much improved.
In allcases, the simulation performance of ARBMLE is the same as RBMLE, raising the open problemof proving that the original RBMLE for the LQ problem also has the best known order of regret.
Problem Formulation
We make the following assumptions on θ ⋆ .
Assumption 1.
There exists a known positive constant c such that θ ⋆ ∈ S =
S
∩ S 1 where,
S
= { θ = ( A, B ) T | ( A, B ) is stabilizable and (cid:0) A, Q 1 / 2 (cid:1) is detectable } 2 [54], and S
= { θ =(
A, B ) T | trace ( θ T θ ) ≤ c } .
Assumption 2.
Let F t := σ (( x 0 , u 0 ) , ( x 1 , u 1 ) , · · · , ( x t , u t )) denote the history of states and inputsuntil time t .
The state noise { w t } is a martingale difference sequence with respect to {F t } , with 161 E [ w t +1 w Tt +1 |F t ]
= I n , and is element-wise sub-Gaussian, i.e., for any γ ∈ R , there exists L > 0with E [ exp ( γw t,j )
|F t ] ≤ exp (cid:16) γ 2 L 2 2 (cid:17) ∀ j and ∀ t .
We assume that Q ∈ R n × n and R ∈ R m × m are known positive semi-definite and positive definitematrices.
Define z
Tt :=
( x Tt , u Tt ) .
Then, the linear stochastic system can equivalently be written as
[Equation]
For a matrix M , we use ∥ M ∥ to denote its operator norm induced from the ℓ 2 -norm.
3.1 Controller Design for a Known LQG System
For a discrete time linear system, x t +
=
θ T z t + w t
+1 , where θ =
( A, B ) T ∈ S , there isan unique positive semidefinite matrix P ( θ ) that satisfies the Riccati equation (see [6]) P ( θ ) =
Q + A T P ( θ )
A − A T P ( θ ) B ( B T P ( θ ) B + R ) − 1 B T P ( θ ) A .
The optimal control law which minimizes the long-term average quadratic cost is u t =
K ( θ ) x t , where the “gain matrix" is K ( θ ) :
=− ( B T P ( θ ) B + R ) − 1 B T P ( θ ) A .
The optimal average cost, J ⋆ ( θ ) is equal to trace ( P ( θ )) .
As aconsequence of Assumption 1, the parameter set S is bounded, hence one can show that P ( θ ) isbounded as well:
[Equation]
Assumption 3.
[Equation] [Equation]
3.2 Construction of Confidence Interval
The squared fitting error of θ is defined as V t ( θ ) :
= (cid:80) t − 1 s =
||
( x s +1 − θ T z s || 22 .
The ℓ 2 -regularizedsquared fitting error with parameter λ > 0 is E t ( θ ) =
λ
||
θ
||
22 + V t ( θ ) .
Let ˆ θ t be the ℓ 2regularized least-squares estimate of θ ⋆ , i.e., ˆ θ t ∈ arg min θ ∈S E t ( θ ) .
Next, given the history
F t ,we construct a “high-probability confidence ball,” C t ( δ ) around ˆ θ t , i.e., a set of plausiblesystem parameters that contains the true parameter θ ⋆ with a high probability.
Let C t ( δ ) :
= 181 (cid:110) θ : trace (cid:16) ( θ − ˆ θ t ) T Z t ( θ − ˆ θ t ) (cid:17) ≤ β t ( δ ) (cid:111) , where Z t := λI n + m + (cid:80) ts =
z s z Ts , and
[Equation]
Define
E 1 ( t
[Equation]
[Equation]
⋆
[Equation]
C s ( δ
[Equation] [Equation] ∀ s [Equation]
[Equation]
2 , · · · , t }
[Equation]
E 2 ( t ) =
{|| x s || ≤ d t
[Equation] ∀ s ≤ t } [Equation] (13) where d t is defined in the Appendix.
Let E 1 :=
E 1 ( T ) and E 2 :=
E 2 ( T ) .
Then, as in [3], 184 P ( E 1 ∩ E 2 ) ≥ 1 − δ 2 .
Moreover, on the event E 2 ( t ) defined in (13), the following holds, [Equation] where,
[Equation]
Y t :
[Equation]
max
[Equation]
with c 1 and c 2 being problem dependant constants.
Initialize: t = 0 , Z 0 = λI n + m for k = 0 , 1 , · · · do if det ( Z t ) > 2 det ( Z t k − 1 ) then solve following optimization to obtain θ t k , θ t ∈ arg min θ ∈S∩C tk (
[Equation]
[Equation] J ⋆ ( θ
[Equation]
, else θ t = θ t − 1 end if u t = K ( θ t ) x t
Z t +1 =
Z t + z s z
Ts end for
The Augmented RBMLE-UCB Algorithm
We employ a version of the Augmented RBMLE-UCB (ARBMLE) algorithm that proceeds in anepisodic manner.
Let t k denote the starting time of the k -th episode.
Then during episode k , itimplements the the control policy, u t
= K ( θ t k ) x t ∀ t ∈ [ t k , t k +1 ) .
where, θ t k is given by solvingthe following optimization problem.
[Equation]
(15) where the bias-term, α ( t ) =
α 0 √ T, ∀ t , for α 0 > 0 .
Regret Analysis
Lemma 5.1.
The regret of the Augmented RBMLE-UCB learning algorithm can be decomposedas R ( T )
=
R 1 + R 2 + R 3 + R 4 , where
R 1 :=
T (cid:88) t [Equation] (cid:26)
[Equation]
R
[Equation]
[Equation]
t
[Equation]
R
[Equation]
[Equation]
T (cid:88) t = (cid:26) ( A ⋆ x t + B ⋆ u t ) T P ( θ t ) ( A ⋆ x t + B ⋆ u t ) − ( A t x t + B t u t ) T P ( θ t ) ( A t x t + B t u t ) (cid:27) ,
R 4 :=
T (cid:88) t =
( J ( θ t ) − J ( θ ⋆ )) .
We bound R 1 , R 2 , R 3 and R 4 on the event E 1 ∩ E 2 .
Lemma 5.2.
On the event E 1 ∩ E 2 , R 1 ≤ 2 DW 2 (cid:113) 2 T log 8 δ + n √ B δ , where B δ :=
b log (cid:16) 4 n √ b vδ (cid:17) ,b := v + T ( cDX T ) 2 (1 + c 20 ) , W = nL (cid:113) 2 n log (cid:0)
nTδ (cid:1) and D is as in (9) .
Lemma 5.3.
On the event E 1 ∩ E 2 , R 2 ≤ 2 DX 2 T log 2 T , where X T is defined in (14) .
Lemma 5.4.
On the event E 1 ∩ E 2 , R 3 ≤ 8 cX 2 T D (1+ c 20 ) √ λ (cid:113)
Tβ T (cid:0) δ 8 (cid:1) log det ( Z T ) det ( λI ) , where Z T :=max 0 ≤ t ≤ T ∥
z t ∥ , and β T (cid:0) δ 8 (cid:1) is defined in (12) .
Lemma 5.5.
On the event E 1 ∩ E 2 , R 4 ≤ 1 α 0 (cid:0) β T (cid:0) δ 4 (cid:1) + λc 2 (cid:1) √ T .
We now use these bounds in conjunction with Lemma 5.1 to bound the regret of RBMLE. Ex
[Table]
Theorem 5.6.
For any δ ∈ (0 , 1) and T > 0 , the regret of the RBMLE Algorithm is upper-boundedby R ( T ) =
˜ O (cid:16)(cid:113) T log 1 δ (cid:17) with a probability at least (1 − δ ) .
Empirical Performance
We evaluate the empirical performance of ARBMLE as well as standard (unaugmented) RBMLE.
We compare these algorithms with OFU [3], Thompson Sampling (TS) [14], Input Perturbations(IE)
[15], Randomized Certainty Equivalence (RCE) [17], and Stabl [16].
The results shown hereare for the following examples of linear system that have appeared in the recent literature on adaptivecontrol of linear systems:
Longitudinal Flight Control of Boeing 747 [16].
Unmanned Aerial Vehicle (UAV) [20, 16]
Unstable Laplacian dynamics [18, 16, 19].
Large transient dynamics [18].
The details of these examples are provided in the Appendix.
Each simulation experiment is performed for a time horizon of 500, and repeated 100 times.
Thereported results are the averaged values over the 100 runs.
In Figure 1, we compare the log 10 ofthe averaged regret of ARBMLE, OFU, TS and Stabl.
In Figure 2, we plot the averaged regret of
RBMLE, ARBMLE, IP and RCE.
We summarize the results using the regret values at T = 500 ,averaged over the 100 runs,in Table 1.
More results for more examples are provided in the Appendix.
The details of the implementations can be found in the Appendix.
We highlight the key observations from above experiments:
• ARBMLE and RBMLE were always found to have the same empirical performance.
More specifically, both these algorithms choose the same estimate θ t , and this suggests that 1 α ( t ) needs to begreater than the Lagrange multiplier for the ball constraint θ ∈ C t k ( δ ) .
This also motivates future
study of the regret of the (unaugmented) RBMLE.
• As can be seen in Figure 1, ARBMLE/RBMLE outperform OFU and TS by a huge margin.
We
conjecture that this is due to temporary instability when OFU, TS are employed [16].
ARBMLE/RBMLE also has signficantly lower regret compared to StabL.
• Figure 2 shows that ARBMLE/RBMLE also outperforms RCE moderately and IP marginally.
Remark 1.
Our simulations for ARBMLE, OFU and TS are based on the confidence interval β t ( δ )as defined in (12) .
Instead, recent works [18, 16] use β t ( δ ) := trace (cid:16) ( θ ⋆ − ˆ θ t ) T Z t ( θ ⋆ − ˆ θ t ) (cid:17) .
However, one may note that θ ⋆ in β t ( δ ) is not known to the learning agent, and so such a definitionof β t ( δ ) is not a viable implementation.
The effect of the choice of β t ( δ ) on the regret performanceis shown in the appendix.
Concluding Remarks
We reconcile the RBMLE and UCB approaches by showing that RBMLE is an unconstrainedpenalty version of the constrained optimization problem of UCB.
Showing that OFU is an constrained version of RBMLE also explains why the optimism embodied in UCB-based schemes is
[-1 × 
-]
[Figure]
[-9 × 10  3 . 6 × 
-]
justified.
In particular, it is justified by the goal of nullifying the one-sided bias that results from theclosed-loop identification of dynamics.
Building on this, we propose an Augmented RBMLE-UCB method that not only matches the bestknown order of regret, ˜ O ( √ T ) , to date – that of the OFU algorithm – but also outperforms OFU,
TS and Stabl by a significant margin in simulation experiments.
In fact, for the UAV experiment,the regret of OFU is ≈ 10 4 , TS ≈ 10 [-
-]16 and Stabl ≈ 10 5 times that of RBMLE.
It outperforms RCEalso, but the gains are moderate, while with respect to IP the gains are smaller.
The simulationswere carried out on real-world examples such as Boeing 747, UAV, Unstable Laplacian and Large
Transient Dynamics which were taken from recent works [18, 55, 19, 16, 20] on adaptive LQGcontrol.
This work furtjer extends recent studies of RBMLE for MDPs [31], stochastic Multi-Armed Bandits [29] and Contextual bandits [30] establishing state-of-art regret results as well as empiricallygood performance.
There remain several open questions.
Currently we study only an augmented form of RBMLE, i.e. ARBMLE, in which the agent searchesfor a parameter value that optimizes a certain reward-biased maximum likelihood objective functionwithin a “high-confidence ball.”
However, (unaugmented) RBMLE optimizes this over the largerset S (Assumption 1) that is known to contain the true parameter with high probability.
The reasonfor this augmentation is simply that currently we are unable to prove regret bounds for RBMLEwithout it.
By including it, we can capitalize on the nice technical results in [3] in the analysis of
OFU.
Simulations show that the constraint is loose, and performance is the same with/without theconstraint.
In fact, as shown by the simulations, the choices of θ t k made by the standard RBMLE,and the Augmented RBMLE are the same.
It remains to be seen whether similar regret guaranteescan be obtained by the (unaugmented) RBMLE, which remains an open problem.
One may note that ARBMLE and OFU differ in their choices of decisions.
Denoting their estimatesof the unknown parameter at a given time by θ ARBMLE and θ OFU respectively, θ OFU always lies [Figure] at the boundary of the UCB-ball constraint.
This is easily seen to be true for bandits and MDPs, andempirically observed to be so for LQ systems.
In contrast, θ ARBMLE is most often in the interiorof the UCB-ball.
Moreover, while OFU treats all models within the UCB-ball equally and onlyassesses them by their cost, ARBMLE prefers models that are closer to the Least Squares Estimate( θ LSE ).
For example, if J ⋆ ( θ 1 ) =
J ⋆ ( θ 2 ) , then ARBMLE prefers the θ i that is closer to θ LSE .
Weconjecture that this is the reason ARBMLE has significantly better performance than OFU – but weare unable to prove it.
It has been shown in previous works on adaptive LQG control [10, 11] that under appropriate formulations RBMLE is guaranteed to stabilize an unknown LQG system over an infinite horizon, i.e.,lim sup T →∞ 1 T (cid:80)
Tt =1 ( ∥ x ( t ) ∥ 2 + ∥ u ( t ) ∥ 2 ) <
∞ a.s.. Morover, the sample path performance costlim sup T →∞ 1 T (cid:80) T t =1 ( x T ( t )
Qx ( t ) + u T ( t )
Ru ( t )) is (a.s.) equal to the optimal performance thatcould be attained if the system parameters ( A ⋆ , B ⋆ ) were known.
In fact, stability is a prerequisite before one can establish the latter type of result, as the decades of work on stochastic adaptivecontrol in the nineteen seventies to the nineties has shown.
While recent work has tended to studythe finer performance measure of regret, it is usually over a finite horizon, and more attention to thestability of the learning process over an infinite horizon appears well deserved.
Similarly, robustnesswhich also was rigorously formulated and addressed in earlier decades earlier needs to re-examined.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data
you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifi-
able information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]