Augmented RBMLE-UCB Approach for Adaptive Control of Linear Quadratic Systems
Abstract
We consider the problem of controlling an unknown stochastic linear system with quadratic costs – called the adaptive LQ control problem.
We re-examine an approach called “Reward-Biased Maximum Likelihood Estimate” (RBMLE) [1] that was proposed more than forty years ago, and which predates the “Upper Confidence Bound” (UCB) method as well as the definition of “regret” for bandit problems [2].
It simply added a term favoring parameters with larger rewards to the criterion for parameter estimation.
We show how the RBMLE and UCB methods can be reconciled, and thereby propose an Augmented RBMLE-UCB algorithm that combines the penalty of the RBMLE method with the constraints of the UCB method [3], uniting the two approaches to optimism in the face of uncertainty.
We establish that theoretically, this method retains ˜ O ( √ T ) regret, the best known so far.
We further compare the empirical performance of the proposed Augmented
RBMLE-UCB and the standard RBMLE (without the augmentation) with UCB,
Thompson Sampling, Input Perturbation, Randomized Certainty Equivalence and
StabL on many real-world examples including flight control of Boeing 747 and
Unmanned Aerial Vehicle.
We perform extensive simulation studies showing that the Augmented RBMLE consistently outperforms UCB, Thompson Sampling and
StabL by a huge margin, while it is marginally better than Input Perturbation and moderately better than Randomized Certainty Equivalence.
Introduction
Consider a linear stochastic system x t +
=
A ⋆ x t + B ⋆ u t + w t
+1 , where x t ∈ R n , u t ∈ R m and w t ∈ R n are the state, control applied and state noise, respectively, at time t .
We study the adaptive control/reinforcement learning problem [4, 5, 6, 7] where the controller’s goal is to minimize the expected finite horizon quadratic cost E (cid:80)
Tt =1 ( x ⊺ t Qx t + u ⊺ t Ru t ) by choosing the control input based on observing the past states and controls, without the knowledge of “true parameter” θ ⋆ :=
( A ⋆ , B ⋆ ) ⊺ .
The deviation from what would have been optimally possible had the true parameter been known is measured by the “regret,” [2], R ( T ) :
= (cid:80) Tt =1 ( x ⊺ t
Qx t + u ⊺ t Ru t ) − T J ⋆ ( θ ⋆ ) , where
J ⋆ ( θ ⋆ ) :
=
lim sup T →∞ 1 T E (cid:80) Tt =1 ( x ⊺ t Qx t + u ⊺ t Ru t ) is the optimal average cost achievable when ( A ⋆ , B ⋆ ) are known.
For this adaptive linear-quadratic (LQ) control problem, there are broadly four classical approaches: the Reward-Biased Maximum Likelihood Estimate (RBMLE) approach [1], the Diminishing Excitation (DE) approach [8], the Upper Confidence Bound (UCB) approach [2], and the Thompson
Sampling (TS) [9] approach based on sampling from the posterior distribution.
The RBMLE and
UCB approaches are “certainty equivalent” (CE) in the sense that they make an estimate ˆ θ t of the unknown true parameter, and then take an action u t =
K (ˆ θ t ) x
t that would be optimal if the estimate ˆ θ t were indeed the true parameter.
They only differ in what parameter estimate they choose.
The
DE approach applies u t =
K (ˆ θ t ) x t + v t , where v t is an added “excitation,” an independent noise, of diminishing variance and ˆ θ t is the maximum likelihood estimate.
The Randomized Certainty
Equivalence (RCE) [10] adds excitation to the parameter estimate.
1.1 The Contributions
We unite the two approaches to “optimism under uncertainty”, RBMLE [1] and UCB [2], by showing that the RBMLE method is a penalty version of the constrained optimization problem of UCB for the case of linear quadratic systems [11, 12, 13, 14, 3].
Based on this we propose an Augmented RBMLE (ARBMLE) method that combines the penalty and constrained versions so that on the one hand it retains the analytical tractability of UCB and on the other hand provides the performance of RBMLE.
We determine how to choose the biasing factor α ( t ) for ARBMLE, and establish a finite time regret bound ˜ O ( √ T ) , the same as the OFULQ algorithm of [3], the best order available to date.
We perform extensive comparative simulation studies of the performance of: (a) ARBMLE and the standard RBMLE.
(b) OFULQ [3], which is the UCB-approach adapted to the LQ problem.
(c) TS [15] which is the Thompson sampling approach adapted to the LQ problem (d) Input Perturbation (IP) [16] which is a recent reincarnation of DE for the adaptive LQ problem that additionally assumes apriori knowledge of a stabilizing controller.
(e) Stabl [17], a modified OFULQ that adds “excitation” to the input for initialization.
(f) RCE [10], which adds excitation to the Least Squares Estimate (LSE).
The examples used for our simulation study have been used in many recent papers [18, 19, 17], namely (a) the longitudinal flight control of Boeing 747 with linearized dynamics [17],(b) Unmanned Aerial Vehicle (UAV) [20, 17] (c) unstable Laplacian dynamics [18], and (d) large transient dynamics [18].
Our simulation results show that ARBMLE outperforms OFULQ, TS and StabL by a large margin, which is primarily due to lack of stabilization experienced by OFULQ and TS [17].
RCE also exhibits a higher regret than
ARBMLE.
While the empirical performance of IP is marginally worse than ARBMLE.
The results show that the ARBMLE has the same performance as the original RBMLE [21], and they both outperform all the above other algorithms.
Notably the choices made by ARBMLE and OFULQ within the confidence interval are very far apart, with ARBMLE outperforming OFULQ by a large margin.
1.2 Previous Works
Prior work on RBMLE has concentrated on establishing long-term average optimality [22, 23, 1, 11, 24, 25, 12, 13, 14, 26, 27, 28].
Recently, its regret performance has been addressed for MultiArmed Bandits (MABs) [29], linear contextual bandits [30], and MDPs [31].
Forced exploration techniques, somewhat similar in spirit to the ϵ t -greedy learning algorithm of [7, 32], are studied in [33, 34, 35] vis-a-vis ensuring that they do not suffer from the insufficient exploration.
An adaptive
LQG control based on the UCB approach, called OFULQ, is proposed in [3] which establishes a regret of ˜ O ( √ T ) .
A similar algorithm to address the adaptive LQG control problem is also designed in [36].
A computationally efficient algorithm called ROBUST with a regret of ˜ O
( T 23 ) is proposed in [18].
An alternative approach for designing learning algorithms is Thompson sampling [9], and [37, 38] have established an expected regret of ˜ O ( √ T ) in a Bayesian context.
More recently, new DEbased algorithms, IP [16] and RCE [10], have been proposed that make an additional assumption, not made in ARBMLE, OFULQ or TS, that one has access to a stabilizing controller for the unknown system, and establish ˜ O ( √ T ) regret.
1.3 The RBMLE Approach
The RBMLE approach, proposed four decades ago [1], was the first approach not resorting to forced choices as in [39].
We begin by giving an informal description of it in the context of the adaptive LQ problem.
Since the true parameter θ ⋆ := ( A ⋆ , B ⋆ ) T is not known, one can make a Least-Squares
Estimate (LSE) ˆ θ t
= ( ˆ A t , ˆ B t ) ⊺ of them:
[Equation]
Under the certainty-equivalence approach, the control input applied is u t =
K (ˆ θ t ) x t where K ( θ ) is the optimal linear feedback gain for the LQ problem when the system is described by θ =
( A, B ) ⊺ .
Suppose now that these estimates were to converge as t → ∞ to ˆ θ ∞ =
( ˆ A ∞ , ˆ B ∞ ) ⊺ .
Then, asymptotically, the input applied is u t ≈ K
( ˆ θ ∞ ) x t .
The closed-loop system therefore settles down to behaving according to
[Equation]
As it does so, one loses the ability to identify the matrices ( A ⋆ , B ⋆ ) , and asymptotically one can only identify the closed-loop gain A ⋆ + B ⋆ K
(ˆ θ ∞ ) .
The problem is that as the parameter estimates begin to converge to ˆ θ ∞ , the control gain converges to K (ˆ θ ∞ ) and further exploration ceases, and one ends up only identifying the behavior of the system under the limiting gain K (ˆ θ ∞ ) being applied to the system.
Since the limiting policy u t = K (ˆ θ ∞ ) x t need not be optimal for the long-term average cost for the true system ( A ⋆ , B ⋆ ) , the CE rule leads to a sub-optimal performance.
Indeed, this problem goes by various names in different fields – the dual control problem [40, 41], the closedidentifiability problem [42, 43, 44], or exploration vs. exploitation dilemma [45].
This problem was resolved in [1] without resorting to forced exploration as in [39].
The key observation made there was that under CE the limiting parameter estimate ˆ θ ( ∞ ) has a one-sided bias.
Specifically, J ⋆ (ˆ θ ∞ ) ≥ J ⋆ ( θ ⋆ ) , i.e., the limiting parameter estimate has an optimal cost that is larger than the optimal cost of θ ⋆ .
To see this, denote by J ( K, θ ) the long-term cost of using the control gain K when the parameter is θ .
Then the fact that the models ˆ θ ∞ and θ ⋆ have identical behavior under the gain K (ˆ θ ∞ ) implies that J ( K (ˆ θ ∞ ) , ˆ θ ∞ ) =
J ( K (ˆ θ ∞ ) , ˆ θ ⋆ ) .
Now note that
J ( K (ˆ θ ∞ ) , ˆ θ ∞ ) =
J ⋆ (ˆ θ ∞ ) since the gain K (ˆ θ ( ∞ )) is optimal for ˆ θ ∞ .
However the gain K (ˆ θ ∞ ) is not necessarily optimal for θ ⋆ , and so J ( K (ˆ θ ∞ ) , ˆ θ ⋆ ) ≥ J ⋆ ( θ ⋆ ) .
Therefore,
[Equation]
Following from this observation, it was reasoned in [1] that if one could slightly bias the MLE to favor models with lower optimal costs J ⋆ ( θ ) so as to obtain J ⋆ (ˆ θ ∞ ) ≤ J ⋆ ( θ ⋆ ) , then one would have equality throughout (3), to obtain J ( K (ˆ θ ∞ ) , ˆ θ ⋆ ) =
J ⋆ ( θ ⋆ ) , yielding the desired result that the gain K (ˆ θ ∞ ) is optimal for θ ⋆ .
So motivated, [1] proposed RBMLE 1 which in the context of the LSE suggests choosing the parameter estimate
[Equation]
Generally α ( t ) =
o ( t ) with the precise growth rate of α ( t ) ↗ + ∞ dependent on the context.
1.4 The UCB Approach
The “Upper Confidence Bound” (UCB) approach was first proposed in the context of Multi-Armed
Bandit (MAB) Problems in [2].
In the context of Bernoulli bandits, it essentially consists of constructing, for each arm i at each time t , a confidence interval ( θ ( i, 1; t ) , θ ( i, 2; t )) of its payoff probability with confidence (1 − δ ( t )) , for δ ( t ) =
o ( 1 t ) , and playing the arm i with the highest value of θ ( i, 1; t ) .
This approach has been generalized to a variety of contexts including linear contextual bandits [46], Gaussian Processes [47], MDPs [48, 49], and LQ systems [3].
Particularized to LQ systems, the OFULQ algorithm of [3] that is based on the UCB approach suggests choosing ˆ θ OFLUQ ( t ) =
( ˆ A OFULQ , ˆ B OFULQ ) ⊺ as the minimizer of: min θ ∈C t ( δ ) J ⋆ ( θ ) , where
[Equation] [Equation]
is a (1 − δ )
-high confidence set of parameters for an appropriate choice of γ T ( δ ) [3].
The Augmented RBMLE-UCB Method
The UCB approach has been called “Optimism in the Face of Uncertainty” (OFU), since it chooses an optimistic arm after calculating the confidence intervals.
The RBMLE version is also the same, though it does it in a different way by directly giving preference to parameters that can yield better rewards.
It arrives at optimism in a very systematic way by noticing that closed-loop identification leads to the chain of inequalities (3), which could then be made into equalities by ensuring that
J ⋆ (ˆ θ ∞ ) ≤ J ⋆ ( θ ⋆ ) .
We now show how one may reconcile the two approaches to optimism in the face of uncertainty in the context of the LQ problem, and then combine them to obtain a method that has experimentally superior empirical performance while also allowing a proof that it achieves the currently best known order of regret.
It can be seen that the RBMLE approach (4) can be considered an unconstrained penalty version of the constrained optimization problem (5)-(6), with a penalty factor 1 α ( t ) for constraint violation.
This provides a justification for employing “optimism” in the UCB approach.
This raises the question of whether we can take advantage of these synergies to fashion a superior algorithm, that is superior from the viewpoint of being able to provide a theoretical guarantee of the best known order of regret to date, as well as superior from the point of view of providing the best experimental performance of the algorithms to date.
One can draw inspiration here from the
Augmented Lagrangian Method of [50, 51].
Given a constrained problem as in (5)-(6), it adds a penalty for constraint violation, and also adjoins the constraint through a Lagrange multiplier, thus involving the constraint twice.
In our case, we add the penalty and also retain the constraint.
This leads to the Augmented RBMLE-UCB method:
[Equation]
The advantage of retaining the constrained optimization is in enabling theoretical analysis, in that we can make use of the bounds on parameter estimates from concentration inequalities, as we shall show in the sequel.
We thereby prove in Section 4 that it also has ˜ O ( √ T ) order of regret, which is the best known so far [3].
Moreover, simulations reported in Section 5 show that its performance is also much improved.
In all cases, the simulation performance of ARBMLE is the same as RBMLE, raising the open problem of proving that the original RBMLE for the LQ problem also has the best known order of regret.
Problem Formulation
As introduced in section 1, we consider the following linear system,
[Equation]
where, A ⋆ ∈ R n × n and B ⋆ ∈ R n × m are unknown system parameters.
Define z ⊤ t :=
( x ⊤ t , u ⊤ t ) .
Then, the linear stochastic system can equivalently be written as
[Equation]
The system incurs a cost at time t given by, x ⊺ t Qx t + u ⊺ t Ru t .
We assume that Q ∈ R n × n and
R ∈ R m × m are known positive semi-definite and positive definite matrices respectively.
We make the following assumptions on θ ⋆ =
( A ⋆ , B ⋆ ) ⊺ .
Assumption 1.
There exists a known positive constant c such that θ ⋆ ∈ S =
S
∩ S 1 where,
S
= { θ = ( A, B ) ⊺ | ( A, B ) is stabilizable and (cid:0) A, Q 1 / 2 (cid:1) is detectable } 2 [54], and S
= { θ = ( A, B ) ⊺ | trace ( θ ⊤ θ ) ≤ c } .
Assumption 2.
Let F t := σ (( x 0 , u 0 ) , ( x 1 , u 1 ) , · · · , ( x t , u t )) denote the history of states and inputs until time t .
The state noise { w t } is a martingale difference sequence with respect to {F t } , with E [ w t +1 w ⊺ t
+1 |F t ] = I n , and is element-wise sub-Gaussian, i.e., for any γ ∈ R , there exists L >with E [ exp ( γw t,j )
|F t ] ≤ exp (cid:16) γ 2 L 2 2 (cid:17) ∀ j and ∀ t .
For a matrix M , we use ∥ M ∥ to denote its operator norm induced from the ℓ 2 -norm.
3.1 Controller Design for a Known LQG System
For a discrete time linear system, x t +
= θ ⊺ z t + w t
+1 , where θ =
( A, B ) ⊺ ∈ S , there is an unique positive semidefinite matrix P ( θ ) that satisfies the Riccati equation (see [
[Equation]
The optimal control law which minimizes the long-term average quadratic cost is u t =
K ( θ ) x t , where the “gain matrix” is K ( θ ) :
= − ( B ⊺ P ( θ ) B + R ) − 1 B ⊺ P ( θ ) A .
The optimal average cost,
J ⋆ ( θ ) is equal to trace ( P ( θ )) .
As a consequence of Assumption 1, the parameter set S is bounded, hence one can show that P ( θ ) is bounded as well:
[Equation]
The following assumption is commonly made in online LQR learning problem in which the knowledge of a stabilizing controller is not made [3, 38].
Assumption 3.
[Equation]
Note that there are recent works which relax (11) slightly to sup θ ∈S ρ ( A + BK ( θ )) < 1 , however they assume that the learner has access to a stabilizing controller [18, 55, 56].
3.2 Construction of Confidence Interval
The ℓ 2 -regularized squared fitting error with parameter λ > 0 is given by:
[Equation]
Let ˆ θ t be the ℓ 2 regularized least-squares estimate of θ ⋆ , i.e., ˆ θ t ∈ arg min θ ∈S V t ( θ ) .
Next, given the history F t , we construct a “high-probability confidence ball,” C t ( δ ) around ˆ θ t , i.e., a set of plausible system parameters that contains the true parameter θ ⋆ with a high probability.
Let
[Equation]
where Z t := λI n + m + (cid:80) ts =
z s z
⊺
s , and
[Equation]
Define
[Equation]
where d t is defined in the Appendix.
Let E 1 :=
E 1 ( T ) and E 2 :=
E 2 ( T ) .
Then, as in [3], P ( E 1 ∩ E 2 ) ≥ 1 − δ 2 .
Moreover, on the event E 2 ( t ) defined in (16), the following holds,
[Equation]
, ∀ t = 1 , 2 , . . .
, where,
[Equation] [Equation]
with c 1 and c 2 being problem dependent constants.
The Augmented RBMLE-UCB Algorithm
We employ a version of the Augmented RBMLE-UCB (ARBMLE) algorithm that proceeds in an episodic manner.
Let t k denote the starting time of the k -th episode.
Then, during episode k , it implements the control policy, u t
=
K
( θ t k ) x t , ∀ t ∈ { t k , t k + 1 , . . .
, t k +1 − 1 } .
where, θ t k is obtained by solving the following optimization problem,
[Equation] [Equation]
where the bias-term, α ( t ) = α 0 √ T, ∀ t , for α 0 > 0 .
Algorithm
Initialize: t = 0 , Z 0 = λI n + m for k = 0 , 1 , · · · do if det ( Z t ) > 2 det ( Z t k − 1 ) then solve the following optimization to obtain θ t k ,
[Equation]
else θ t = θ t − 1 end if
[Equation]
t → t +end for
In Theorem 4.1, we show that regret for ARBMLE is upper bounded by ˜ O (cid:16)(cid:113) T log 1 δ (cid:17) which is same order as OFULQ [3].
Theorem 4.1.
For any δ ∈ (0 , 1) and T > 0 , with a probability at least (1 − δ ) , the regret of the
ARBMLE Algorithm is upper-bounded by R ( T ) ≤ ˜ O (cid:16)(cid:113) T log 1 δ (cid:17) .
Proof.
Appendix A.1.
[Table]
Empirical Performance
We evaluate the empirical performance of ARBMLE as well as standard (unaugmented) RBMLE.
We compare these algorithms with OFULQ [3], Thompson Sampling (TS) [15], Input Perturbations (IE) [16], Randomized Certainty Equivalence (RCE) [10], and Stabl [17].
The results shown here are for the following examples of linear systems that have appeared in the recent literature on adaptive control of linear systems:
Unstable Laplacian dynamics [18, 17, 19].
Large transient dynamics [18].
Unmanned Aerial Vehicle (UAV) [20, 17].
Longitudinal Flight Control of Boeing 747 [17].
The details of these examples are provided in the Appendix.
Each simulation experiment is performed for a time horizon of 500 steps, and repeated 50 times.
The reported results are the averaged values over the 100 runs.
In Figure 1, we compare the logof the averaged regret of ARBMLE, OFULQ, TS and Stabl.
In Figure 2, we plot the averaged regret of RBMLE, ARBMLE, IP and RCE.
We summarize the results using the regret values at
T = 500 , averaged over the 50 runs, in Table 1.
Results for more examples are provided in the
Appendix.
Details of the implementations can be found in the Appendix.
We highlight the key observations from above experiments:
• ARBMLE and RBMLE were always found to have the same empirical performance in most cases.
More specifically, both these algorithms choose the same estimate θ t , and this suggests that 1 α ( t ) needs to be greater than the Lagrange multiplier for the ball constraint θ ∈ C t k ( δ ) .
This also motivates future study of the regret of the (unaugmented) RBMLE.
• As can be seen in Figure 1, ARBMLE/RBMLE outperform OFULQ and TS by a huge margin.
We conjecture that this is due to temporary instability when OFULQ, TS are employed [17].
ARBMLE/RBMLE also has a signficantly lower regret as compared with StabL. • Figure 2 shows that ARBMLE/RBMLE also outperforms RCE moderately, and IP marginally
Remark 1.
Our simulations for ARBMLE, OFULQ and TS are based on the confidence interval β t ( δ ) as defined in (15) .
Instead, recent works [17, 18] use β t ( δ ) := trace (cid:16) ( θ ⋆ − ˆ θ t ) ⊺ Z t ( θ ⋆ − ˆ θ t ) (cid:17) .
However, one may note that θ ⋆ in β t ( δ ) is not known to the learning agent, and so such a definition of β t ( δ ) is not a viable implementation.
The effect of the choice of β t ( δ ) on the regret performance is shown in the appendix.
Concluding Remarks
We reconcile the RBMLE and UCB approaches by showing that RBMLE is an unconstrained penalty version of the constrained optimization problem of UCB.
Showing that UCB is a constrained version of RBMLE also explains why the optimism embodied in UCB-based schemes is justified.
In particular, it is justified by the goal of nullifying the one-sided bias that results from the closed-loop identification of dynamics.
[Figure]
Building on this
[Figure]
we propose an Augmented RBMLE-UCB method that not only matches the best known order of regret, ˜ O ( √ T ) , to date – that of the OFULQ algorithm – but also outperforms
OFULQ, TS and Stabl by a significant margin in simulation experiments.
In fact, for the UAV experiment, the regret of OFULQ is ≈ 10 8 , TS ≈ 10
and StabL ≈
10 5 times that of RBMLE.
It outperforms RCE also, but the gains are moderate, while with respect to IP the gains are smaller.
The simulations were carried out on real-world examples such as Boeing 747, UAV, Unstable Laplacian and Large Transient Dynamics which were taken from recent works [18, 55, 19, 17, 20] on adaptive
LQG control.
This work further extends recent studies of RBMLE for MDPs [31], stochastic Multi-Armed Bandits [29] and Contextual bandits [30] establishing state-of-art regret results as well as empirically good performance.
There remain several open questions.
Currently we study only an augmented form of RBMLE, i.e. ARBMLE, in which the agent searches for a parameter value that optimizes a certain reward-biased maximum likelihood objective function within a “high-confidence ball.”
However, (unaugmented) RBMLE optimizes this over the larger set S (Assumption 1) that is known to contain the true parameter.
The reason for this augmentation is simply that currently we are unable to prove regret bounds for RBMLE without it.
By including it, we can capitalize on the nice technical results in [3] in the analysis of OFULQ.
Simulations show that the constraint is loose, and performance is the same with/without the constraint.
In fact, as shown by the simulations, the choices of θ t k made by the standard RBMLE, and the Augmented
RBMLE are the same.
It remains to be seen whether similar regret guarantees can be obtained by the (unaugmented) RBMLE, which remains an open problem.
One may note that ARBMLE and OFULQ differ in their choices of decisions.
Denoting their estimates of the unknown parameter at a given time by θ ARBMLE and θ OFULQ respectively, θ OFULQ always lies at the boundary of the UCB-ball constraint.
This is easily seen to be true for bandits and MDPs, and empirically observed to be so for LQ systems.
In contrast, θ ARBMLE is most often in the interior of the UCB-ball.
Moreover, while OFULQ treats all models within the UCB-ball equally and only
[Figure]
assesses them by their cost, ARBMLE prefers models that are closer to the Least Squares Estimate ( θ LSE ).
For example, if J ⋆ ( θ 1 ) =
J ⋆ ( θ 2 ) , then ARBMLE prefers the θ i that is closer to θ LSE .
We conjecture that this is the reason why ARBMLE has a significantly better performance than OFULQ – but we are unable to prove it
[Figure]
It has been shown in previous works on adaptive LQG control [11, 12] that under appropriate formulations RBMLE is guaranteed to stabilize an unknown LQG system over an infinite horizon, i
[Figure]
e., lim sup T →∞ 1 T (cid:80) Tt =1 ( ∥ x ( t ) ∥ 2 + ∥ u ( t ) ∥ 2 ) <
∞ a.s.. Morover, the sample path performance cost lim sup T →∞ 1 T (cid:80) Tt =1 ( x ⊺ ( t )
Qx ( t ) + u ⊺ ( t )
Ru ( t )) is (a.s.) equal to the optimal performance that could be attained if the system parameters ( A ⋆ , B ⋆ ) were known.
In fact, stability is a prerequisite before one can establish the latter type of result, as the decades of work on stochastic adaptive control in the nineteen seventies to the nineties has shown.
While recent work has tended to study the finer performance measure of regret, it is usually over a finite horizon, and more attention to the stability of the learning process over an infinite horizon appears well deserved.
Similarly, robustness which also was rigorously formulated and addressed in earlier decades needs to be re-examined [57].
Acknoweldgments
This material is based upon work partially supported by the US Army Contracting Command under W911NF-22-1-0151, US Office of Naval Research under N00014-21-1-2385; 4/21-22 DARES:
Army Research Office W911NF-21-20064 US National Science Foundation under CMMI-2038625,
The views expressed herein and conclusions contained in this document are those of the authors and should not be interpreted as representing the views or official policies, either expressed or implied, of the U.S. Army Contracting Command, ONR, ARO, NSF, or the United States Government.
The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
The work of Rahul Singh was partially supported by the SERB Grant SRG/2021/002308, and PC 39010B.
[-Note that w t +
-]
[-b log (cid:16) 4 n √ b vδ (cid:17) , b := v + T ( cDX T ) 2 (1 + c 20 ) , W = nL (cid:113) 2 n log (cid:0) 
-]