P ROVABLY E FFICIENT C ONTEXTUAL L INEAR Q UADRATIC R EGULATOR
Anonymous authors
Paper under double-blind review
A BSTRACT
A fundamental challenge in artiﬁcially intelligence is to build an agent that generalizes and adapts to unseen environments.
A common strategy is to build a decoder that takes a context of the unseen new environment and generates a policy.
The current paper studies how to build a decoder for the fundamental continuous control environment, linear quadratic regulator (LQR), which can model a wide range of real world physical environments.
We present a simple algorithm for this problem, which uses upper conﬁdence bound (UCB) to reﬁne the estimate of the decoder and balance the exploration-exploitation trade-off.
Theoretically, our algorithm enjoys a (cid:101) O (cid:16) √ T (cid:17) regret bound in the online setting where T is the number of environments the agent played.
This also implies after playing (cid:101) O (cid:0) 1 /(cid:15) 2 (cid:1) environments, the agent is able to transfer the learned knowledge to obtain an (cid:15) -suboptimal
policy for an unseen environment.
To our knowledge, this is ﬁrst provably efﬁcient algorithm to build a decoder in the continuous control setting.
While our main focus is theoretical, we also present experiments that demonstrate the effectiveness of our algorithm.
I NTRODUCTION
Humans are able to solve a new task without any training based on previous experience in similar tasks.
Our intelligent agent should be able do the same, learning from previous experience, adapting to the new ones and improving the performance as the agent gains more experience.
This is a challenging problem as we need to design an adaptation mechanism which is fundamentally different from classical supervised learning methods.
A common approach is to build a decoder so that once the agent sees a description of new task, i.e., the context of the new task, the decoder turns the context into a succinct representation of the new task, based on which the agent is able to design a policy to solve the task.
Note this procedure resembles how a human solves a new task.
For example, if a human wants to push an object on a table, the human ﬁrst sees the object and the table (context).
Then, in his/her mind, the context becomes a representation of this task, e.g., a sense of weight of the object.
Based on this representation, the human can easily reason about how much force to exert on the object.
This general approach has been applied in practice.
For example, Wu et al.
(2018) studied the visual navigation task and built a Bayesian model that takes the context of new environments and outputs the policy that enables the agent to navigate.
Killian et al.
(2016) used this approach to develop personalized medicine policies for HIV treatment.
While this is a promising approach, currently we only have limited theoretical understanding.
The approach can be formulated in Contextual Markov Decision Process (CMDP) framework (Hallak et al., 2015).
Recently, there is a line of work gave provable guarantees for CMDP (Abbasi-Yadkori & Neu, 2014; Hallak et al., 2015; Dann et al., 2018; Modi et al., 2018; Modi & Tewari, 2019).
These work all studied tabular MDPs, and use function approximation, e.g., linear functions, generalized linear models, etc, to model the mapping from the context to the probability transition matrix.
A major drawback of these work is that they are restricted to the tabular setting and thus can only deal with discrete environments.
Therefore, they can hardly model real-world continuous control tasks, like the task of pushsing an object as we described above.
A natural question arises:
Can we design a provably efﬁcient decoder for continuous control problems?
In this paper, we make an important step towards answering this question.
We study the fundamental setting in continuous control, linear quadratic regulator (LQR).
LQR is arguably the most widely used framework in continuous control, as LQR easily models real world physical phenomena, e.g., the pushing object task we described earlier.
We propose a new algorithm that builds a decoder, so that for a new LQR task, the decoder takes LQR’s context and outputs a representation based on which the agent can easily infer a near-optimal policy for new continuous control tasks.
In the training phase, we build the decoder via a sequence of LQRs (in an online fashion) with unknown parameters.
For each new task, we ﬁrst use the current decoder to build the representation of this task, infer a policy based on this representation and use this policy to do control for this episode.
There are two crucial components in our algorithm.
First, after each episode, we will reﬁne the estimate of the decoder based on the observations from this episode.
Second, it is crucial to use a upper conﬁdence bound (UCB) estimator of the decoder to build the representation so that the agent can perform a near-optimal trade-off between exploration and exploitation.
In this way, we provably show the decoder improves the performance as it experiences more training tasks.
Formally, we show our algorithm enjoys (cid:101) O (cid:16) √ T (cid:17) regret (the difference between the cumulative rewards of our algorithm and the unknown optimal policy on every seen environment) bound in the online setting.
Moreover, the algorithm is able to obtain an (cid:15) -suboptimal policy for an unseen LQR environment after playing (cid:101) O (cid:0) (cid:15) − 2 (cid:1) environments.
To our knowledge, this is the ﬁrst provably efﬁcient algorithm that builds a decoder for continuous control environments.
Empirically, we simulate several physical environments to illustrate the effectiveness of our algorithm.
Organization This paper is organized as follows.
In Section 2, we discuss related work.
In
Section 3, we formally describe the problem setup.
In Section 4, we present our algorithm and its theoretical guarantees.
In Section 5, we use simulation on physical environments to demonstrate the effectiveness of our approach.
We conclude in Section 6 and defer most technical proofs to the appendix.
R ELATED W ORK
Recently there is a large body of literature focusing on learning for control in LQR systems.
The ﬁrst work we are aware of is Fiechter (1997) which studies the sample complexity of LQR in the ofﬂine setting.
For the online setting, where the agent can only obtain the next state starting from the present state, the ﬁrst near-optimal regret bound ( (cid:101) O ( √ T ) ) is due to Abbasi-Yadkori & Szepesv ´ ari (2011), which studies the learning problem in the inﬁnite-horizon average-case cost setting.
Later on, a sequence of papers (Tu & Recht, 2017; Dean et al., 2017; 2018;
Tu & Recht, 2018; AbbasiYadkori et al., 2018; Cohen et al., 2019) studied this problem in similar settings, improved efﬁciency of the algorithms and characterized the gap between model-free and model-based approaches.
Building an agent that quickly adapts to new environment has received increasing interest in the machine learning community.
Taylor & Stone (2009) gave a summary for the literature status before 2009.
More recently, a sequence of theory papers Lehnert & Littman (2018); Spector & Belongie (2018);
Abel et al.
; Lehnert et al.
(2019) studied the transferability of reward knowledge, state-abstraction, and model features for Markov decision processes.
Please also refer to references in paper cited above for more details.
There are also some experimental works, e.g., Santara et al.
(2019); Yu et al.
; Wu et al.
(2018); Gamrian & Goldberg (2018), studying how to transfer knowledge from seen tasks to unseen tasks.
Nevertheless, we are not aware of any study on how to provably perform continuous control with contexts.
P RELIMINARIES
Notations.
We begin by introducing necessary notations.
We write [ h ] to denote the set { 1 , . . .
, h } .
We use I d ∈ R d × d to denote the d -dimensional identity matrix.
We use 0 d × d (cid:48) to represent the allzero matrix in R d × d (cid:48) .
If it is clear from the context, we omit the subscript d × d (cid:48) .
Let (cid:107)·(cid:107) 2 denote the Euclidean norm of a vector in R d .
For a symmetric matrix A , let (cid:107) A (cid:107) op denote its operator
Finite Horizon Linear Quadratic Regulator.
We now formally deﬁne the ﬁnite horizon Linear
Quadratic Regulator (LQR) problem.
In the LQR problem, there is a state space X ⊂ R d and a closed action space U ⊂ R d (cid:48) .
Suppose we always start from the initial state x 1 = x init ∈ X and play for H steps.
Then at a state x h ∈ X , if an action u h ∈ U is played, the next state is given by
[Equation]
where A, B are matrices of proper dimension and w h
+1 is a zero-mean random vector.
Here A, B can be viewed as the succinct representation of this LQR because as will be explained below, given
A, B , we can easily infer the optimal policy for this LQR.
For simplicity, we additionally denote
[Equation]
[Equation]
y
[Equation]
Now the state transition can be rewritten as x h +
=
My h + w h
+1 .
For the ease of presentation, we assume that the covariance matrix of noise vector w h
+
is E ( w h +1 w (cid:62) h +1 )
=
I d .
Our analysis follows similarly if the covariance matrix is not I d (see e.g. Remark 3 of Abbasi-Yadkori & Szepesv´ari (2011)).
After each step, the player receives an immediate cost x (cid:62) h Q h x h + u (cid:62) h R h u h , where Q h , R h are positive deﬁnite (PD) matrices of proper dimensions.
Throughout the paper, we assume the agent knows Q h and R h for all h .
At a terminal state x H , there is no action to be played, and the player receives a terminal cost x (cid:62) H Q H x H , where Q H is a PD matrix of proper dimension.
The goal of the player is to ﬁnd a policy π : ( X × U ) ∗ × X → U , which is a function that maps the trajectory { ( x i , u i ) } h − 1 i =1 ∪ { x h } to the next action u h , such that the following objectives are minimized:
[Equation]
where the action u h is given by u h
= π
[( x 1 , u 1 ) , ( x 2 , u 2 ) , . .
, ( x h − 1 , u h − 1 ) , x h ] , and the expectation is over the randomness of w h and π .
It is well-known that the optimal policy π ∗ is Markovian Puterman (2014), i.e., it only depends on the present state.
For an unconstrained action space U , we have
[Equation]
where M =
[ A, B ] and K h ( M ) is a matrix that will be deﬁned shortly.
It is also known (see e.g.
Bertsekas (1996))
that the optimal cost function J ∗ h ( x ) :=
J π ∗ h ( x ) is given by
[Equation]
where
[Equation]
P h +1 ( M ) A − A (cid:62) P h +1 B ( R h + B (cid:62) P h +1 ( M ) B ) − 1 B (cid:62) P
h +1 ( M )
A h < H Q H h = H (3) and
C h ( M ) =
(cid:26) C
[Equation]
+ [Equation] [Equation] (cid:62) h +
[Equation]
+
[Equation]
+ [Equation] <
[Equation]
h
[Equation]
We now deﬁne K h ( M ) as
[Equation]
Note that the optimal value Equation ( 2 ) satisﬁes Bellman equations,
∀ h ∈ [ H − 1] : [Equation] and ∀ h ∈
[ H − 1] :
[Equation]
Now we have shown that if we are given A and B , then we can obtain the optimal policy directly.
In this paper, we deal with setting where A and B are unknown and we need to use decoder to decode
A and B from the contexts of the current LQR, as speciﬁed below.
Learning to Control LQR with Contexts In the continuous control with contexts setting, in each episode we observe a context
[Equation]
where µ is a distribution on R p ×
d × R p (cid:48)
× d (cid:48) .
The context [ C, D ] encodes the information of the environment.
Formally, the representation ( [ A, B ] ) of this environment can be decoded from the context via a decoding matrix Θ ∗ ∈
R
d × ( p
[Equation] [Equation]
From now on, to emphasize that the representation of LQR can be decoded from Θ ∗ , we write
[Equation]
If it is clear from the context, we ignore [ C, D ] for notational simplicity.
Note the optimal decoder Θ ∗ is unknown to the agent and the goal is to learn Θ ∗ from contexts and interactions with the environment.
Below we formally deﬁne the problem that we study.
Deﬁnition 3.1 (Contextual Transfer Learning Problem) .
Build an agent that plays on K LQR games (one trajectory per game) with context pairs { ( C (1) , D (1) ) , ( C (2) , D (2) ) , . .
, ( C ( K ) , D ( K ) ) } ∼ µ , for some integer K ≥ 0 such that for another new context pair ( C, D ) ∼ µ , the agent outputs a policy π based on ( C, D ) which satisﬁes
[Equation]
for some given target accuracy (cid:15) > 0 .
Here K is the sample complexity which ideally scales polynomially with (cid:15) and problem-dependent parameters.
The performance of the agent can also be measured by regret, as deﬁned below.
[Equation]
where (cid:101) π ( k ) is the policy played at episode k by the agent.
This quantity measurse the sub-optimality of policies the agent played in the ﬁrst K episodes.
Remark 3.1.
We consider matrix-type linear maps from context to the representation only for sake of presentation.
Our algorithm and analysis can be readily extended to other linear maps, e.g., [ A ∗ ( C ) , B ∗ ( D )] :
= f ( C, D ) for some unknown linear function f .
M AIN A LGORITHM
In this section, we ﬁrst describe the algorithm and then present its sample complexity guarantees.
Since the decoder is linear, a straightforward algorithm is ﬁrst to estimate ( A, B ) using the trajectory from the single episode with system identiﬁcation techniques (Mehra, 1974), and then to use linear regression to estimate the mapping from ( C, D ) to ( A, B ) .
However, this naive algorithm has two drawbacks.
First, estimating ( A, B ) accurately requires a long horizon.
However, in our setup, we do not have any restrictions on the horizon.
To ﬁx this, we stack contexts and observations to construct a more direct estimate on the decoder (cf.
Equation equation 8). Second, in order to achieve √ T -type regret guarantee, one needs to balance exploration and exploitation carefully, but the naive algorithm does not have such an component. Our algorithm uses UCB to construct a conﬁdence set which helps balance exploration and exploitation. Algorithm We describe the high-level idea of the algorithm below.
The agent maintains a decoder that maps the context ( C, D ) to the representation ( A, B ) .
We denote Θ ( k ) the decoder at the k -th episode.
Initially, we know nothing about Θ ∗ , so we initialize our decoder by setting Θ (1) =
∈ R
d × p .
At the k -th episode, the agent plays policy π ( k ) and in each time step h ∈
[ H − 1] , it collects data
[Equation]
1: Input Total number of episodes K ; 2: Initialize Θ (1) ← 0 ∈ R d × 2 p , V (1) ← I 2 p, 2 p , W (1) ← 0 ∈ R 2 p × d ; 3: for episode k = 1 , 2 , . .
, K do 4:
Let x ( k ) 1 ← x init , V ( k +1) ← V ( k ) , W ( k +1) ← W ( k ) ; 5:
Obtain context [ C ( k ) , D ( k ) ] ∼ µ ; 6:
Solve for the present policy:
[Equation]
(10) 7: where J ∗ 1 is given by Equation 2, and C ( k ) is deﬁned in Equation 11; 8: for stage h = 1 , 2 , . .
, H − 1 do 9:
Let the current state be x ( k ) h ; 10:
Play action u ( k ) h ← K h (cid:0) M Θ ( k ) ,C ( k ) ,D ( k ) (cid:1) · x ( k ) h , where K h is deﬁned in Equation 4; 11:
Obtain the next state x ( k ) h +1 ; 12:
Let z ( k ) h ← (cid:20) C ( k ) x
( k ) h
D ( k ) u ( k ) h (cid:21) ; 13:
Update: V ( k +1)
← V ( k +1)
+ z ( k )
h z ( k ) (cid:62) h ; 14:
Update: W ( k +1) ← W
( k +1)
+ z ( k ) h (cid:16) x ( k ) h +1 (cid:17) (cid:62) ; 15:
Compute Θ ( k +1) (cid:62) ← (cid:16) V ( k +1) (cid:17) − 1 W where z ( k ) h can be viewed as the context regularized observation. We now describe how to obtain policy π ( k ) . We ﬁrst solve the following optimization problem, [Equation]
where J ∗ 1 is given by Equation (2), and the conﬁdence set C ( k ) will be deﬁned shortly. C ( k ) represents our conﬁdence region on Θ ∗ . Since we choose the one that minimizes the cost, this represents the principle “optimism in the face of uncertainty” and it is the key to balance exploration and exploitation which will be more clear in the proof. Notice that the above optimization problem is a polynomial optimization problem.
Then the policy is given by
[Equation]
M
[Equation] k ) [Equation] · x where M ( k ) =
M Θ ( k ) ,C ( k )
[Equation]
D ( k ) :
=
Θ ( k ) · (cid:20) C ( k )
D ( k ) (cid:21) , and K h is given by Equation (4).
After episode k ∈ [ K ] , we use the following ridge regression formulation to update decoder
[Equation]
(8) where V
[Equation]
and W ( k +
[Equation]
=
k
[Equation]
After playing K episodes, the algorithm outputs a (cid:101) Θ by picking one from { (cid:101) Θ ( k ) } k ∈
[ K ] uniformly at random.
Now for a new task with its context, our learned policy map is given by:
[Equation]
The formal algorithm is presented in Algorithm 1.
4. A LGORITHM A NALYSIS
To present the analysis of the algorithm, we ﬁrst introduce some assumptions.
Assumption 4.1.
The contexts and LQR satisfy the following properties.
• ∀ h ∈
[ H ] , (cid:107) P h ( M ) (cid:107) 2 ≤ c q for some parameter c q > 0 .
• (cid:107) Θ ∗ (cid:107) F ≤ c Θ ; • ∀ h ∈
[2 , H ] , i ∈
[ d ] : (cid:107) w h (cid:107) 2 < ∞ and ∀ γ > 0 , E
[ γw h,i ] ≤ exp( γ 2 c 2 w / 2) ; • ∀ x ∈ X , u ∈ U , ( C, D ) ∈ supp( µ ) : [-(cid:107) Cx (cid:107) 
-][Equation]
x
[Equation]
• ∀ ( C, D ) ∈ supp( µ ) , x ∈ X , h ∈
[ H ] : K h ( M Θ ∗ ,C,D ) · x ∈ U .
where c Θ , c w , c x are some positive parameters.
The ﬁrst assumption is standard to ensure controllability.
The second is a regularity condition on the optimal decoder Θ ∗ .
The third assumption imposes almost sure boundedness of the noise w .
The fourth assumption is a regularity condition on the observation.
The last assumption guarantees the optimal controller for the unconstrained LQR problem is realizable in our control set U .
Given these assumptions, We are now ready to deﬁne conﬁdence set C ( k ) as follows. [Equation] and
[Equation]
D
[Equation]
where P h is given by Equation (3) and β ( k ) is deﬁned as follows,
[Equation]
We remark that C ( k ) is changing at every episode because we update Θ ( k ) and V ( k ) at every episode.
The size of C ( k ) is decreasing because V ( k ) is strictly increasing at every episode.
With the above assumptions, the guarantee of Algorithm 1 is formally presented in the next theorem.
Theorem 4.1.
Suppose we run Algorithm 1 for
[Equation]
episodes, for some parameter c (cid:48)
H,c q ,c x ,c Θ ,c w depending polynomially on H , c q , c x , c Θ , c w , Then with probability at least 1 − δ , we have for (cid:101) π C,D be deﬁned in Equation 9.
[Equation]
[Equation] Theorem 13 states after playing polynomial number of episodes, our agent can learn a decoder (cid:101) Θ such that given a new LQR with contexts ( C, D ) , this decoder can turns the contexts into a nearoptimal policy (cid:101) π C,D without any training on the new LQR.
Note this is the desired agent we want to build as described in the introduction.
We emphasize again that this is the ﬁrst provably efﬁcient algorithm that builds a decoder for continuous control environments.
Remark 4.1.
Via similar analysis, it is easy to show that if the output (cid:101) Θ is picked uniformly at random from { Θ ( k ) } k ∈ [ K ] , the policy achieves similar accuracy.
In fact, Theorem 4.1 is implies by the following regret bound of our algorithm.
Proposition 4.1.
With probability at least 1 − δ ,
[Equation]
where c (cid:48)
H is a constant depending only polynomially on H , c q , c x , c M , c w .
By the deﬁnition of regret, this proposition justiﬁes that the performance of the agent actually improves as it sees more environment.
E XPERIMENTS
In this section, we validate the effectiveness of our algorithm via numerical simulations.
We perform experiments on a path-following task.
In this task, we are given a trajectory z ∗ 1 , z ∗ 2 , . . .
, z ∗ H ∈ R 2 .
Our goal is to exert forces u 1 , u 2 , . . .
, u m ∈ R 2 on objects with different (measurable) masses to minimize the total squared distance (cid:80)
Hh =1 (cid:107) z
h − z ∗ h (cid:107)
+ (cid:107) u
i (cid:107) 22 .
Each state x h =
[ z h ; v h ] ∈ R 4 is a vector whose ﬁrst two dimensions represent the current position and the last two dimension represent the current velocity.
In each stage h , we may exert a force u h ∈ R 2 on the object, which produces an accelerations u h m ∈ R 2 .
The dynamics of the system can be described as
[Equation]
(14) where 0 < k ≤ 1 is the decay rate of velocity induced by resistance.
In our setting, the decay rate of velocity k is ﬁxed (encoded in Θ ∗ ), where the mass of the object m is drawn from the uniform distribution over [0 . 1 , 10] .
In our experiments, we set the noise vector w h in the dynamics of the LQR system (cf.
Equation 1) to be a Gaussian random vector with zero mean and covariance 10 − 4 · I .
In each episode, we receive an object with mass m where m is draw from the uniform distribution over [0 . 1 , 10] , train one trajectory using that object, and the goal is to recover the physical law described in Equation 14 so that our model can deal with objects with unseen mass m .
Please see Appendix B for the concrete value of Θ ∗ , Q and R and the distribution of C and D .
In our experiments, we use 100 different masses as training masses (ﬁxed among all experiments), and use 100 different masses as test masses (again ﬁxed among all experiments).
All the training masses and test masses are drawn from the uniform distribution over [0 . 1 , 10] .
We implement a practical version of Algorithm 1.
In particular, instead of solving the optimization problem in Equation 10 exactly, we sample 100 different Θ from C ( k ) uniformly at random, and choose the Θ which minimizes the objective function.
Moreover, instead of using the theoretical bound for β ( k ) in Equation 12, we treat β ( k ) as a tunable parameter and set β ( k ) =
10 4 in our experiments to encourage exploration at early stage of the algorithm.
We use two different metrics to measure the accuracy of the learned model.
First, we use (cid:107) Θ k − Θ ∗ (cid:107) F where Θ k is calculated in Line 15 to measure the accuracy of the learned Θ .
Moreover, using the learned Θ , we test on 100 objects whose masses are the 100 test masses to calculate the control cost (cid:80)
Hh =1 (cid:107) z
h − z ∗ h (cid:107)
+ (cid:107) u
i (cid:107) 22 .
We compare the control cost of the learned Θ and the optimal control cost, and use the mean value of the differences (named mean control error) to measure the accuracy.
In all experiments we ﬁx H = 20 .
We use three different types of trajectories: unit circle, parabola y =
x 2 with x ∈ [0 , 1] and Lemniscate of Bernoulli with a = 1 1 .
For all three types of trajectories we use their parametric equation x = x ( t ) and y
=
y ( t ) , divide the interval [0 , 1] evenly into H parts, and set t to be the endpoints of these parts.
We use these t values to deﬁne the trajectory z ∗ 1 , z ∗ 2 , . . .
, z ∗ H ∈ R 2 .
We set the decay ratio k to be k = 1 or k = 0 .
7 in our experiments.
We plot the accuracy of the learned model in Figure 1.
Here we vary the number of training episodes (the number of training masses) and observe its effect on the accuracy.
It can be observed that our algorithm achieves an satisfactory accuracy using only 5 episodes.
We also illustrate trajectories obtained by our resulting controllers in Figure 2.
From Figure 2, it is clear that as the agent plays more environments, it can enjoy better performance.
C ONCLUSION
In this paper, we give a provably efﬁcient algorithm for learning LQR with contexts.
Our result bridges two major ﬁelds, learning with contexts and continuous control from a theoreticallyprincipled view.
For future work, it is interesting to study more complex settings, include non-linear control.
Another interesting direction is to design provable algorithm in our setting with safety guarantees (Dann et al., 2018). [Figure] [Figure]
A P ROOF OF M AIN R ESULTS
This sections devotes to proving the main results.
Before we prove Proposition 4.1, let us use it to prove Theorem 4.1.
Proof of Theorem 4.1.
We rewrite the Equation equation 13 as follows.
[Equation]
where
[Equation]
and
[Equation]
Let F k be the ﬁltration of ﬁxing all randomness before episode k .
We have R 1 and R 2 are Martingale difference sum.
Note that the magnitude of each summand in R 1 or R 2 is upper bounded by (proved in Lemma A.3 and A.4),
[Equation]
almost surely.
Therefore, by Azuma’s inequality (Theorem A.1), we have, with probability greater than 1 − δ/ 2 ,
[Equation]
Moreover, by Proposition 4.1, we have with probability greater than 1 − δ/ 2 ,
[Equation]
K
[Equation]
where c is constant depending only polynomially on H , c q , c x , c M , and c w .
Combining the above two inequalities, and setting K appropriately, we complete the proof of Theorem 4.1.
A. U SEFUL C ONCENTRATION B OUNDS
Before we prove the main proposition, we ﬁrst recall some useful concentration bounds.
Theorem A.1 (Azuma’s inequality) .
Assume that { X s } s ≥ 0 is a martingale and | X s − X s − 1 | ≤ c s almost surely.
Then for all t > 0 and all (cid:15) > 0 ,
[Equation]
Theorem A.2 (Martingale Concentration, Theorem 16 of Abbasi-Yadkori & Szepesv´ari (2011)) .
Let
F t ; t ≥ 0 be a ﬁltration, ( z t ; t ≥ 0) be an R d -valued stochastic process adapted to ( F t ) .
Let ( η t ; t ≥ 1) be a real-valued martingale difference process adapted to
F t .
Assume that η t is conditionally sub-Gaussian with constant L , i.e.,
[Equation]
Consider the following martingale
[Equation]
and the matrix-valued processes
[Equation]
Then for any δ ∈ (0 , 1) , with probability at least 1 − δ ,
[Equation]
[Equation]
where (cid:107) S t (cid:107) 2 V − 1 t := S (cid:62) t V − 1 t S t .
A. P ROOF OF P ROPOSITION 4.
In this section, we prove the main proposition.
We ﬁrst bound det( V ( k ) ) for any k .
Lemma A.1.
For all k ∈ [ K ] ,
[Equation]
Proof.
Since V ( k ) is PD, we have,
[Equation]
(cid:13)(cid:13)
[Equation]
p
[Equation]
p .
By Assumption 4.1, we have (cid:107) z
( k (cid:48) )
h (cid:107) 22 ≤ c 2 x .
This completes the proof.
Let us then deﬁne an event E k as follows.
Deﬁnition A.1 (Good Event) .
We deﬁne event E k as {∀ k (cid:48)
≤ k : Θ ∗ ∈ C ( k (cid:48) ) } .
We then show that the event E k happens with high probability.
Lemma A.2.
For all k ∈ [ K ] , we have Pr[ E k ] ≥ 1 − δ .
Proof.
Now we consider Θ ∗ − Θ ( k ) .
We immediately have
[Equation]
Next, we have
[Equation]
=
Θ ∗
[Equation] [Equation]
[Table]
[Equation]
=
I − ( V ( k ) ) − 1 .
Hence we have
[Equation]
where (cid:107) X (cid:107) 2 V := tr (cid:0) X (cid:62) V X (cid:1) and the last inequality uses Cauchy-Schwartz inequality.
Notice that
[Equation]
Moreover
[Equation]
we have
[Equation]
h = w ( k (cid:48) )
h +1 ,j z ( k (cid:48) )
h (cid:13)(cid:13)(cid:13) 2 ( V ( k ) ) −
By Theorem A.2, we have, for every j ∈ [ d ] , with probability at least 1 − δ/d , we have, (cid:13)(cid:13) (cid:13)
k (cid:88) k (cid:48)
=
H − 1 (cid:88) h
= w ( k (cid:48) )
h +1 ,j z ( k (cid:48) )
h (cid:13)(cid:13) (cid:13)
( V ( k ) ) − 1 ≤ 2 c 2 w log( d det( V ( k ) ) 1 / 2 /δ ) .
By an union bound, we have, with probability at least 1 − δ ,
[Equation]
h = w ( k
[Equation]
h
[Equation]
z
[Equation]
k
[Equation]
h (cid:13)(cid:13)(cid:13) 2 ( V ( k ) ) − 1 ≤ 2 dc 2 w log( d det( V ( k ) ) 1 / 2 /δ ) .
Plugging to tr (cid:2) (Θ ∗ − Θ ( k ) )
V ( k ) (Θ ∗ − Θ ( k ) ) (cid:62) (cid:3) , we have, with probability at least 1 − δ , tr (cid:2) (Θ ∗ − Θ ( k ) )
V ( k ) (Θ ∗ − Θ ( k ) ) (cid:62) (cid:3) ≤ (cid:16) c
Θ + c w (cid:113) 2 d log( d det( V ( k ) ) 1 / 2 /δ )
(cid:17)≤ (cid:16) c
Θ + c w (cid:113) 2 d (cid:0) log d + p log(1 + kHc 2 x /p ) / 2 + log δ − 1 (cid:1)(cid:17) 2 .
This completes the proof.
We deﬁne I E K as the indicator for E K happens.
We denote
M ( k ) ∗ =
M Θ ∗ ,C ( k ) ,D ( k ) ,
M ( k ) =
M Θ ( k ) ,C ( k ) ,D ( k ) , and y ( k ) h
=
[ x ( k ) (cid:62) h , u ( k ) (cid:62) h ] (cid:62) .
On E k , we have
[Equation]
We denote ∆ ( k ) :
=
J π k h ( M ( k ) ∗ , x 1 ) − J ∗ h ( M ( k ) ∗ , x 1 ) .
We can rewrite equation 7 as
[Equation]
where the second term is non-zero with probability less than δ .
For the ﬁrst term, we have
[Equation]
where
[Equation]
Let us consider (cid:101) ∆ ( k ) h .
We denote ﬁltration F k,h as ﬁxing the trajectory up to time ( k, h ) and all { C ( k (cid:48) ) , D ( k (cid:48) )
} k (cid:48) ≤ k .
We have
[Equation]
where
[Equation] [Equation]
By induction
[Equation]
we have
[Equation]
[Table]
Notice that δ ( k ) h and δ ( k ) h are Martingale difference adapted to F k,h .
We can well bound the sum of them via Azuma’s inequality.
Lemma A.3.
For all h
[Equation] [Equation]
Proof.
Prove by induction on h .
The base case J π
[Equation]
H
[Equation]
M
[Equation]
holds straightforwardly.
Consider an arbitrary h < H , we have
[Equation]
as desired.
Lemma A.4.
For all x ∈ X , we have
[Equation]
Proof.
Follows from Assumption 4.1.
We are now ready to prove Proposition 4.1.
Proof of Proposition 4.1.
Thus by Azuma’s inequality, we have, with probability at least 1 − δ , (cid:12)(cid:12)(cid:12) k (cid:88)
[Equation]
And, with probability at least 1 − δ , (cid:12)(cid:12)(cid:12) k (cid:88) k (cid:48)
=
H − 1 (cid:88) h
= δ (cid:48)
( k ) h (cid:12)(cid:12)(cid:12)
[Table]
[Equation]
[Table]
For
[Table]
, we bound it here
[Table]
( k ) h
[Equation]
(cid:107)
[Equation]
c q c x c Θ and (cid:107) P
h +1 ( (cid:102) M ( k ) )
[Equation]
(cid:0) M ∗ y ( k ) h (cid:1) (cid:107)
[Equation]
c q c
[Equation]
Hence (cid:104) k (cid:88)
[Equation]
Moreover, by triangle inequality
[Equation]
we have
[Equation] [Equation]
By Assumption 2, we also have (cid:107) ( V ( k ) ) − 1 / 2 y
( k ) h (cid:107) 2 ≤ (cid:107) ( y ( k ) h (cid:107) 2 ≤ √ c x .
Hence,
[Equation]
Combining the above equations
[Equation]
we have,
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Lastly, by Lemma 8 of Yang & Wang (2019), we have
[Equation]
Together with Lemma A
[Equation]
1, we have
[Equation]
Overall, we have,
[Equation]
Putting everything together, with probability at least 1 − 2 δ , we have
[Equation]
where c H is a constant depending on H, c q , c x , c Θ and c w .
B C ONCRETE C HOICE OF THE P ARAMETERS
We further augment the state so that the ﬁrst coordinate is a constant with value 1 .
More speciﬁcally, we set the state x h
= [1; z h ; v h ] ∈ R 5 .
We set
[Equation] [Equation]


so that for any state x h , x Th Q
h
x
H
=
(cid:107)
z h − z ∗ h (cid:107) 22 .
We set R h
= I with size 2 × 2 .
We set Θ ∗ =   1 0  00 1
[Equation]
[Equation]
k
[Equation]
k
[Equation]


[Equation]
C to be the 5 × 5 identity matrix and D to be I/m with size 2 × 2 where m is sampled from the uniform distribution over [0 . 1 , 10] , to represent the physical law in Equation 14.