P ROVABLY E FFICIENT L IFELONG R EINFORCEMENT L EARNING WITH L INEAR R EPRESENTATION
A BSTRACT
We theoretically study lifelong reinforcement learning (RL) with linear representation in a regret minimization setting.
The goal of the agent is to learn a multi-task policy based on a linear representation while solving a sequence of tasks that may be adaptively chosen based on the agent’s past behaviors.
We frame the problem as a linearly parameterized contextual Markov decision process (MDP), where each task is speciﬁed by a context and the transition dynamics is context-independent, and we introduce a new completeness-style assumption on the representation which is sufﬁcient to ensure the optimal multi-task policy is realizable under the linear representation.
Under this assumption, we propose an algorithm, called UCB Lifelong Value Distillation (UCBlvd), that provably achieves sublinear regret for any sequence of tasks while using only sublinear planning calls.
Speciﬁcally, for K task episodes of horizon H , our algorithm has a regret bound ˜ O
( (cid:112) (
d 3 + d (cid:48) d ) H 4 K ) based on O ( dH log( K )) number of planning calls, where d and d (cid:48) are the feature dimensions of the dynamics and rewards, respectively.
This theoretical guarantee implies that our algorithm can enable a lifelong learning agent to learn to internalize experiences into a multi-task policy and rapidly solve new tasks.
I NTRODUCTION
Recently, there has been a surging interest in designing lifelong learning agents that can continuously learn to solve multiple sequential decision making problems in their lifetimes (Thrun & Mitchell, 1995; Khetarpal et al., 2020; Silver et al., 2013; Xie & Finn, 2021).
This scenario is in particular motivated by building multi-purpose embodied intelligence, such as robots working in a weakly structured environment (Roy et al., 2021).
Typically, curating all tasks beforehand for such problems is nearly infeasible, and the problems the agent is tasked with may be adaptively selected based on the agent’s past behaviors.
Consider a household robot as an example.
Since each household is unique, it is difﬁcult to anticipate upfront all scenarios the robot would encounter.
Moreover, the tasks the robot faces are not independent and identically distributed (i.i.d.).
Instead, what the robot has done before can affect the next task and its starting state; e.g., if the robot fails to bring a glass of water and breaks it, then the user is likely to command the robot to clean up the mess.
Thus, it is critical that the agent continuously improves and generalizes learned abilities to different tasks, regardless of their order.
In this work, we theoretically study lifelong reinforcement learning (RL) in a regret minimization setting (Thrun & Mitchell, 1995; Ammar et al., 2015), where the agent needs to solve a sequence of tasks using rewards in an unknown environment while balancing exploration and exploitation.
Motivated by the embodied intelligence scenario, we suppose that tasks differ in rewards, but share the same state and action spaces and transition dynamics (Xie & Finn, 2021).To
be realistic, we make no assumptions on how the tasks and initial states are selected 1 ; generally we allow them to be chosen from a continuous set by an adversary based on the agent’s past behaviors.
Once a task is speciﬁed and revealed, the agent has one chance (i.e., executing one rollout from its current state) to complete the task and then it moves to the next task.
The agent’s goal is to perform near optimally for the tasks it faces, despite the online nature of the problem.
This means that the accumulated regret of the learner compared with the best policy for each task should be sublinear in its lifetime.
We assume that there is no memory constraint; this is usually the case for robotics applications where real-world interactions are the main bottleneck (Xie & Finn, 2021).
Nonetheless, we require that the agent eventually learns to make decisions without frequent deliberate planning, because planning is time consuming and creates undesirable wait time for user-interactive scenarios.
In other words, the agent needs to learn a multi-task policy, generalizing from not only past samples but also past computation, to solve new tasks.
Formally, we consider an episodic setup based on the framework of contextual Markov decision process (CMDP) (Abbasi-Yadkori & Neu, 2014; Hallak et al., 2015).
It repeats the following steps:
1) At the beginning of an episode, the agent is set to an initial state and receives a context specifying the task reward, both of which can be arbitrarily chosen.
2) When needed, the agent uses its past experiences to plan for the current task.
The agent runs a policy in the environment for a ﬁxed horizon in an attempt to solve the assigned task and gains experience from its policy execution.
The agent’s performance is measured as the regret with respect to the optimal policy of the corresponding task.
We require that, for any task sequence, both the agent’s overall regret and number of planning calls to be sublinear in the number of episodes.
While lifelong RL is not new, the realistic need of simultaneously achieving 1) sublinear regret and 2) sublinear number of planning calls for 3) a potentially adversarial sequence of tasks and initial states makes the setup considered here particularly challenging.
To our knowledge, existing works only address a strict subset of these requirements; especially, the computation aspect is often ignored.
Most provable works in lifelong RL make the assumption that the tasks are ﬁnitely many (Ammar et al., 2015; Zhan et al., 2017; Brunskill & Li, 2015), or are i.i.d.
(Ammar et al., 2014; Brunskill &
Li, 2014; Abel et al., 2018a;b; Lecarpentier et al., 2021), while others considering similar setups to ours do not provide regret guarantees (Isele et al., 2016; Xie & Finn, 2021).
On the technical side, the closest lines of work are Modi & Tewari (2020); Abbasi-Yadkori & Neu (2014); Hallak et al. Modi et al.
; Kakade et al.
(2020) for contextual MDP and Wu et al. ; Abels et al.
for the dynamic setting of multi-objective RL, which study the sample complexity for arbitrary task sequences; however, they either assume the problem is tabular or require a model-based planning oracle with unknown complexity.
Importantly, none of the existing works properly addresses the need of sublinear planning calls, which creates a large gap between the abstract setup and practice need.
In this paper, we aim to establish a foundation for designing agents meeting these three practically important requirements, a problem which has been overlooked in the literature.
As the ﬁrst step, here we study lifelong RL with linear representation.
We suppose that the contextual MDP is linearly parameterized (Yang & Wang, 2019; Jin et al., 2020) and the agent needs to learn a multi-task policy based on this linear representation.
To make this possible, we introduce a new completenessstyle assumption on the representation which is sufﬁcient to ensure the optimal multi-task policy is realizable under the linear representation.
Under these assumptions, we propose the ﬁrst provably efﬁcient lifelong RL algorithm, U pper C onﬁdence B ound
L ifelong V alue D istillation (UCBlvd, pronounced as “UC Boulevard”), that possesses all three desired qualities.
Speciﬁcally, for K episodes of horizon H , we prove a regret bound ˜ O
( (cid:112) (
d 3 + d (cid:48) d ) H 4 K ) using ˜ O ( dH log( K )) planning calls, where d and d (cid:48) are the feature dimensions of the dynamics and rewards, respectively.
From a high-level viewpoint, UCBlvd uses a linear structure to identify what to transfer and operates by interleaving 1) independent planning for a set of representative tasks and 2) distilling the planned results into a multi-task value-based policy.
UCBlvd also constantly monitors whether the new experiences it gained are sufﬁciently signiﬁcant, based on a doubling schedule, to avoid unnecessary planning.
On the technical side, UCBlvd’s design is inspired by single-task LSVI-UCB (Jin et al., 2020), however, we introduce a novel distillation step based on QCQP, along with a new completeness assumption, to enable computation sharing across tasks; we also extend the low-switching cost technique (Abbasi-Yadkori et al., 2011; Gao et al., 2021; Wang et al., 2021) for single-task RL to the lifelong setup to achieve sublinear number of planning calls.
Notation.
Throughout the paper, we use lower-case letters for scalars, lower-case bold letters for vectors, and upper-case bold letters for matrices.
The Euclidean-norm of x is denoted by (cid:107) x (cid:107) 2 .
We denote the transpose of a vector x by x (cid:62) .
For any vectors x and y , we use (cid:104) x , y (cid:105) to denote their inner product.
We denote the Kronecker product by A ⊗ B .
Let A ∈ R d × d be a positive deﬁnite and ν ∈ R d .
The weighted 2-norm of ν with respect to A is deﬁned by (cid:107) ν (cid:107) A := √ ν (cid:62) A ν .
For a positive integer n , [ n ] denotes the { 1 , 2 , . . . , n } .
For a real number α , we denote { α } + = max { α, 0 } .
Finally, we use the notation ˜ O for big-O notation that ignores logarithmic factors.
P RELIMINARIES
We formulate lifelong RL as a regret minimization problem in contextual MDP (Abbasi-Yadkori &
Neu, 2014; Hallak et al., 2015) with adversarial context and initial state sequences.
We suppose that a context determines the task reward but does not affect the dynamics.
Such a context dependency is common for the lifelong learning scenario where an embodied agent consecutively solves multiple tasks.
Below we give the formal problem deﬁnition.
Finite-horizon contextual MDP.
We consider a ﬁnite-horizon contextual MDP denoted by M =
( S , A , W , H, P , r ) , where S is the state space,
A is the action space, W is the task context space, H is the horizon (length of each episode), P = { P h } Hh =1 are the transition probabilities, and r = { r h } Hh = are the reward functions.
We allow S and W to be continuous or inﬁnitely large, while we assume
A is ﬁnite such that max a ∈A can be performed easily.
For h ∈ [ H ] , r h ( s, a, w ) denotes the reward function whose range is assumed to be in [0 , 1] , and P h ( s (cid:48) | s, a ) denotes the probability of transitioning to state s (cid:48) upon playing action a at state s .
In short, a contextual MDP can be viewed as an MDP with state space S × W and action space A where the context part of the state remains constant in an episode.
2 To simplify the notation, for any function f , we write P h
[ f ]( s, a ) :
=
E s (cid:48) ∼ P h ( .
| s,a ) [ f ( s (cid:48) )] .
Policy and value functions.
In a ﬁnite-horizon contextual MDP, a policy π = { π h } Hh =1 is a sequence where π h : S×W
→ A determines the agent’s action at time-step h .
Given π , we deﬁne its state value function as V πh ( s, w ) :=
E [ (cid:80) Hh (cid:48)
=
h r h (cid:48) (cid:0) s h (cid:48) ,
π h (cid:48)
( s h (cid:48) , w ) , w ) | s h
= s (cid:3) and its action-value function as Q πh ( s, a, w ) :=
r h ( s, a, w ) + P h
[ V πh +1 ( ., w )]
( s, a ) , where Q πH +1 = 0 .
We denote the optimal policy as π ∗ h ( s, w ) :
= sup π V πh ( s, w ) , and let V ∗ h :=
V π ∗ h and Q ∗ h :=
Q π ∗ h denote the optimal value functions.
Lastly, we recall the Bellman equation of the optimal policy:
[Equation]
Interaction protocol of lifelong RL.
The agent interacts with a contextual MDP M in episodes.
For presentation simplicity, we assume that the reward functions r are known, while the transition probabilities P are unknown and must be learned online; we will discuss how reward learning can be naturally incorporated in Section 4.3.
At the beginning of episode k , the agent receives a task context w k ∈ W and is set to an initial state s k 1 , both of which can be adversarially chosen.
The agent can use past experiences to plan for the current task, if needed.
Then the agent executes its policy π k : at each time-step h ∈ [ H ] , it observes the state s kh , plays an action a kh = π kh ( s kh , w k ) , observes a reward
r kh :=
r h ( s kh , a kh , w k ) , and goes to the next state s kh +1 according to P h ( .
| s kh , a kh ) .
Let K be the total number of episodes.
The agent’s goal is to achieve sublinear regret, where the regret is deﬁned as
[Equation]
As the comparator policy above (namely π ∗ that deﬁnes V ∗ 1 ) also knows the task context, achieving sublinear regret implies that the agent would attain near task-speciﬁc optimal performance on average.
Linear Model Representation.
We focus on MDPs with linear transition kernels and reward functions (Jin et al., 2020; Yang & Wang, 2019) that are encapsulated in the following assumption.
Assumption 1 (Linear MDPs) .
M =
( S , A , H, P , r, W ) is a linear MDP with feature maps φ :
S ×
A → R d and ψ : S ×
A × W → R d (cid:48) .
That is, for any h ∈
[ H ] , there exist a vector η h and d measures µ h :=
[ µ h (1) , . . . , µ h ( d ) ] (cid:62) over S such that P h ( .
| s, a ) =
(cid:10) µ h ( . ) , φ ( s, a ) (cid:11) and r h ( s, a, w ) = (cid:10) η h , ψ ( s, a, w ) (cid:11) , for all ( s, a, w ) ∈ S
× A × W .
Without loss of generality, (cid:13)(cid:13) φ ( s, a ) (cid:13)(cid:13)
[Table]
In real-world problems, we can use the context to model the task speciﬁcation of a problem.
For example, if we want to design household robots to assist humans with a series of tasks like cooking, cleaning, washing dishes, lawn mowing, vacuuming, we can treat the the context as a natural language instruction that the human user would give to the robot, and we can view the representations ψ and φ as the embedding of a deep neural network model that has been pre-trained.
Example 1 (Weighted Rewards) .
An interesting and common special case is ψ ( s, a, w ) =
φ ( s, a ) ⊗ ρ ( w ) , for some mapping ρ : W → R m .
In this case, it holds that d (cid:48) =
md and r h ( s, a, w ) =
(cid:10) ρ ( w ) , r h ( s, a ) (cid:11) , where r h ( s, a ) =
A h φ ( s, a ) ∈
R m , for some A h ∈ R m × d , is the vector reward functions at time-step h .
We can view r h ( s, a, w ) as a weighted reward with weights ρ ( w ) that depend on task w .
This setting is closely related to Multi-Objective RL studied for tabular case in Wu et al.
(2021), which studies the case where ρ ( w ) =
w ∈ R m along with tabular S and A .
A WARM - UP ALGORITHM FOR LIFELONG RL
We ﬁrst present a warm-up algorithm based on linear representation, termed Lifelong Least-Squares
Value Iteration (Lifelong-LSVI), in Algorithm 1, which is a straightforward extension of the singletask LSVI-UCB algorithm proposed by Jin et al. (2020) to the lifelong learning setting.
The motivation of this warm-up algorithm is to give intuitions on how the problem structure in Assumption 1 can be used to achieve small regret and discuss the computational difﬁculty in lifelong learning.
We will show that Lifelong-LSVI has a sublinear regret bound, which matches the minimax optimal rate in the special case studied by Wu et al.
(2021) in terms of number of objectives, m (see Example 1).
However, we will also show that Lifelong-LSVI is not computationally efﬁcient, in the sense that the number of planning calls it requires grows linearly with the number of episodes, which would mean the overall computational complexity grows quadratically.
This high computation cost is because the agent never learns to internalize the task solving skills but requires going though all past experiences for planning every time a new task arrives.
Importantly, we will discuss why it cannot be made computationally efﬁcient in an easy manner without further assumptions on the representation.
This drawback motivates our new completeness assumption and our main algorithm, UCBlvd, which is provably efﬁcient in terms of both regret and number of planning calls, in Section 4.
We remark that Lifelong-LSVI is only a warm-up algorithm that guides the reader to understand the mechanisms used for addressing the problem, motivates the need for UCBlvd, and shows what regret bound is possible when computational complexity is not a concern (though being impractical).
3. A LGORITHMIC NOTATIONS
To begin, we introduce the template and the notations that will be used commonly in presenting the warm-up algorithm, Lifelong-LSVI, and later our main algorithm, UCBlvd.
For each algorithm, ﬁrst we will deﬁne an algorithm-speciﬁc action-value function Q kh : S ×
A × W → R , which determines the agent’s policy at time-step h in episode k ; then we present the full algorithm and its analysis using the quantities below, which are deﬁned with respect to each algorithm’s deﬁnition of Q kh .
Given { Q kh } h ∈ [ H ] , we deﬁne state value functions and their backups as
[Equation]
Thanks to the linear MDP structure in Assumption 1, it holds that
[Equation]
[Equation]
Let λ > 0 be a constant.
We deﬁne the λ -regularized least squares estimator of θ kh ( w ) as ˜ θ k h ( w ) :
{+= (cid:16) Λ kh (cid:17) − 1 k − 1 (cid:88) τ = φ τh V kh +
+}
( s τh +1 , w ) , where Λ kh : = λ I d + k − 1 (cid:88) τ = φ τh φ τh (cid:62) , (5) and ˜ θ k h ( w ) is the solution to min θ ∈ R d (cid:80)
k − 1 τ =1 ( (cid:104) θ , φ ( s τh , a τh ) (cid:105) − V kh +
( s τh +1 , w ))
+ λ (cid:107) θ (cid:107) 22 , φ τh :=
φ ( s τh , a τh ) , and I d ∈ R d
× d is the identity matrix.
3. D ETAILS OF L IFELONG -LSVI AND ITS THEORETICAL GUARANTEES
We deﬁne the upper conﬁdence bound (UCB) style action-value function of Lifelong-LSVI as follows:
[Equation]
1 Set: Q kH +1 ( ., ., . )
= 0 , ∀ k ∈ [ K ] 2 for episodes k
= 1 , . .
K do
Observe the initial state s k 1 and the task context w k .
for time-steps h = H, . . .
, 1 do
Compute ˜ θ k h ( w k ) as in (5) using Q kh +1 deﬁned in (6).
for time-steps h = 1 , . . .
, H do
Compute Q kh ( s kh , a, w k ) for all a ∈ A as in (6).
Play a kh = arg max a ∈A Q kh ( s kh , a, w k ) and observe s kh +1 and r kh .
where Q kH +1 = 0 and ˜ θ k h ( w ) and Λ kh are deﬁned in (5). Here, β is an exploration factor that will be appropriately chosen in Theorem 1. At episode k , given w k , Lifelong-LSVI ﬁrst performs planning backward in time based on past data to compute ˜ θ k h ( w k ) in (5) using Q kh +1 deﬁned in (6) (Lines 45).
Then, in execution, it uses ˜ θ k h ( w k ) to compute Q kh ( s kh , a, w k ) for the current state and all a ∈ A (Line 7) and executes the action with the highest value (Line 8).
We show that Lifelong-LSVI achieves sublinear regret for our lifelong RL setup.
The complete proof is reported in Appx.
A, which follows the ideas of LSVI-UCB (Jin et al., 2020).
Theorem 1.
Let T = KH .
Under Assumption 1, there exists an absolute constant c > 0 such that for any ﬁxed δ ∈ (0 , 0 . 5) , if we set λ = 1 and β = cH (cid:16) d + √ d (cid:48) (cid:17) (cid:112) log( dd (cid:48) T/δ ) in Algorithm 1, then with probability at least 1 − 2 δ , it holds that R K ≤ ˜ O (cid:16)(cid:112)
( d 3 + dd (cid:48) )
H 3 T (cid:17) .
Before introducing our main algorithm in Section 4, we make a few remarks on the regret and number of planning calls of Lifelong-LSVI.
First, Theorem 1 implies that for the special case studied by Wu et al.
(Example 1), the regret bound of Lifelong-LSVI becomes ˜ O
( √ md 3 H 3 T ) .
This rate is optimal in terms of its dependency on m , as shown in Wu et al.
Furthermore, this rate matches the LSVI-UCB’s regret dependencies on d and H for the single-task setting (Jin et al., 2020).
While Lifelong-LSVI has a decent regret guarantee, it requires computing ˜ θ
k h ( w k ) for all h ∈
[ H ] , whenever a distinct new task w k arrives.
Since the number of unique tasks may be as large as K , the total number of planning calls required in Lifelong-LSVI is K in the worst case.
Unfortunately, the number of planning calls of Lifelong-LSVI cannot be easily improved, because under Assumption 1 alone, the optimal Q-function Q ∗ h ( s, a, w ) of the CMDP can be nonlinear in the representation ψ .
As a result, for any algorithm that represents its policy linearly based on both ψ and φ , in general it is necessary to recompute the coefﬁcients for every new w to be optimal.
For
Lifelong-LSVI speciﬁcally, this nonlinear dependency shows up in ˜ θ
k h ( w ) of Q kh ( s, a, w ) in (6).
In the next section, we discuss how placing a completeness-style assumption, which ensures
Q ∗ h ( s, a, w ) can be linearly parameterized by ψ , would circumvent the issue of non-linear dependency of the action-value functions on w , and consequently would enable computation sharing to decrease the number of planning calls to O ( dH log( K )) .
UCB L IFELONG V ALUE D ISTILLATION (UCB LVD )
In this section, we present our main algorithm, UCB L ifelong V alue D istillation (UCBlvd), in
Algorithm 2.
Under new completeness-style assumption that we will introduce in Section 4.1, we show that UCBlvd shares the same regret bound as Lifelong-LSVI but signiﬁcantly reduces the number of planning calls to be logarithmic in K .
In contrast to Lifelong-LSVI which learns individual action-value function for each w k , UCBlvd learns a single action-value function for all w ∈ W based on ψ ( s, a, w ) to enable computation sharing across tasks, which is made possible by the extra completeness-style assumption.
In general, in order to directly extend Lifelong-LSVI to only use feature ψ ( s, a, w ) ∈ R d (cid:48) with d (cid:48) ≥ d , we need a context-dependent dynamics structure, which would eventually increase the regret.
UCBlvd maintains the same order of regret as Lifelong-LSVI by separating the planning into a novel two-step process: 1) independent planning with φ for a set of representative task contexts and 2) distilling the planned results into a multi-task value function parameterized by ψ .
In addition, UCBlvd runs a doubling schedule to decide whether replanning is necessary, which makes the total number of planning calls logarithmic in K .
4. E NABLING COMPUTATION SHARING
As lifelong RL with Assumption 1 alone would require replanning in every episode in general (see Section 3), here we introduce new structural assumptions on ψ to enable computation sharing across tasks.
First, we deﬁne the following class of functions
[Equation]
where S d ++ denotes the set of symmetric positive deﬁnite matrices.
We now state our main completeness-style assumption.
Assumption 2 (Completeness) .
For any f ∈ F and h ∈ [ H ] , there exists a vector ξ fh ∈ R d (cid:48) with (cid:13)(cid:13)(cid:13) ξ fh (cid:13)(cid:13)(cid:13) ≤ H √ d (cid:48) such that P h (cid:2) f ( ., w ) (cid:3) ( s, a ) = (cid:104)
[Equation]
[Equation]
( s, a,
[Equation]
) (cid:105) .
This assumption says that the backups of functions in F are captured by the feature ψ with bounded parameters.
The deﬁnition of F closely models the structure of action-value function used by
Lifelong-LSVI in (6), except (cid:104) ˜
θ k h ( w ) , φ ( s, a ) (cid:105) there is replaced by functions linear in ψ ( s, a, w ) .
We will see that the action-value function used by UCBlvd deﬁned in the next section is contained in
F .
In addition, by setting β = 0 in F and (1), we see Q ∗ h ( s, a, w ) is linearly realizable by ψ under
Assumption 2.
We note that a similar notion of this assumption is mentioned in previous work for single-task settings under the name of “optimistic closure” (Wang et al., 2020).
Inspired by Example 1, we now introduce the next assumption on the structure of ψ .
Assumption 3 (Mappings) .
We assume ψ ( s, a, w ) =
φ ( s, a ) ⊗ ρ ( w ) , for some mapping ρ : W →
R m , i.e., d (cid:48)
=
md .
We assume that there is a known set { w (1) , w (2) , . . .
, w ( n ) } of n ≤ m task contexts such that ρ ( w ) ∈
Span( { ρ ( w ( j ) ) } j ∈ [ n ] ) for all w ∈ W .
That is, for any w ∈ W , there exist coefﬁcients { c j ( w ) } j ∈ [ n ] such that ρ ( w )
= (cid:80) j ∈ [ n ] c j ( w ) ρ ( w ( j ) ) .
We assume (cid:80) j ∈ [ n ] (cid:12)(cid:12)
c j ( w ) (cid:12)(cid:12)
≤
L for all w ∈ W and some L < ∞ .
Note that, for ﬁnite-dimensional representations, such set { ρ ( w ( j ) ) } j ∈ [ n ] always exists.
We assume that this set { w (1) , w (2) , . . .
, w ( n ) } is known to the algorithm
4. D ETAILS OF UCB LVD
We deﬁne the UCB style action-value function of UCBlvd as follows:
[Equation]
The parameter ˆ ξ k h is computed by solving the convex quadratically constrained quadratic program (QCQP) in (8), which is deﬁned on a set of representative task contexts { w (1) , w (2) , . . .
, w ( n ) } in
Assumption 3 and state-action pairs D :=
(cid:8) ( s, a ) : φ ( s, a ) are d linearly independent vectors .
(cid:9) .
[Equation]
, ∀
[Equation]
∈ [ n ] and (cid:107) ξ (cid:107)
[Equation]
≤ H √ md, where ˜ θ k h ( w ) and Λ kh are deﬁned in (5).
In Appx.
B.3, we will show that the action-value function in (7) is an optimistic estimate of the optimal action-value function.
UCBlvd also uses the linear dependency of Q kh on ψ to reduce calls of the planning step in (8).
The agent triggers replanning only when it has gathered enough new information compared to the last update at episode ˜ k .
This is measured by tracking the variations in Gram matrices { Λ kh } h ∈
[ H ] (Line 4 for Algorithm 2).
Finally, when executing the policy at episode k , the agent chooses the action according to Q ˜ k h in Line 10.
1 Set: Q kH +1 ( ., ., . )
= 0 , ∀ k ∈ [ K ] , ˜ k
=2 for episodes k = 1 , . .
K do
Observe the initial state s k 1 and the task context w k .
if ∃ h ∈ [ H ] such that log det Λ kh − log det Λ ˜ kh > 1 then ˜ k
= k for time-steps h
= H, . .
. , 1 do
Compute ˆ ξ ˜ k h as in (8).
for time-steps h = 1 , . .
, H do
Compute Q ˜ kh
( s kh , a, w k ) for all a ∈ A as in (7).
Play a kh = arg max a ∈A Q ˜ kh
( s kh , a, w k ) and observe s kh +1 and r kh .
4. T HEORETICAL ANALYSIS OF UCB LVD
We present our main theoretical result which shows UCBlvd achieves sublinear regret in lifelong RL using sublinear number of planning calls, for any sequence of tasks.
The proof is given in Appx.
B.
Theorem 2.
Let T = KH .
Under Assumptions 1, 2, and 3, the number of planning calls in Algorithm 2 is at most dH log(1 + Kdλ ) , and there exists an absolute constant c > 0 such that for any ﬁxed δ ∈ (0 , 0 . 5) , if we set λ
= 1 and β = cH
( d + √ md ) (cid:112) log( mdT/δ ) in Algorithm 2, then with probability at least 1 − 2 δ , it holds that R K ≤ ˜ O (cid:16) L (cid:112) ( d 3 + md 2 ) H 3 T (cid:17) .
Theorem 2 shows that UCBlvd has the same regret bound as Lifelong-LSVI in Theorem 1, but reduces the number of planning calls from K to dH log(1 + K/dλ ) .
As we discussed before, this is made possible by the unique QCQP-based distillation step of UCBlvd in (8).
If we were to simply perform least-squares regression to ﬁt (cid:104) ψ ( s, a, w ) , ˆ ξ k h (cid:105) to {(cid:104) φ ( s, a ) , ˜ θ k h ( w ( j ) ) } j ∈ [ n ] for distillation, we cannot guarantee the required optimism, because (cid:104) φ ( s, a ) , ˜ θ k h ( w ) (cid:105) computed based on ﬁnite samples can be an irregular function that cannot be modelled by ψ ( s, a, w ) .
Remark 1.
If the rewards are unknown, we can adopt a slightly different completeness assumption with an extra bonus in terms of ψ , and then combine tools from linear bandits (Abbasi-Yadkori et al., 2011) and our proof of Theorem 2.
Because reward learning affects the radius of the conﬁdence intervals for θ kh
( w ) , the number of planning calls and regret would increase by factors of O ( m ) and
O ( √ m ) 3 , respectively, compared to those in Theorem 2.
See Appx.
C for details.
Remark 2.
It is possible to eliminate the assumption that ψ ( s, a, w ) =
φ ( s, a ) ⊗ ρ ( w ) .
In this case, our analysis would instead require a set { w (1) , w (2) , . .
, w ( n ) } of n tasks such that ψ ( s, a, w ) ∈
Span( { ψ ( s, a, w ( j ) ) } j ∈ [ n ] ) for all ( s, a, w ) ∈ S ×
A × W .
In Appx.
D, we provide details of this relaxation, and show that this version still enjoys the same planning calls and regret as in Theorem 2.
Remark 3.
We can eliminate Assumptions 1 and 3 and instead design a computation-sharing version of Lifelong-LSVI under a sightly different completeness assumption with a class F , whose exploration bonus is β (cid:13)(cid:13) ψ ( s, a, w ) (cid:13)(cid:13)
˜ Λ − 1 .
This assumption naturally includes settings with linear MDP in which dynamics also change with task context, i.e., for all h ∈
[ H ] , it holds that P h ( .
| s, a, w ) =
(cid:104)
µ h ( . ) , ψ ( s, a, w ) (cid:105) for d (cid:48) unknown measures [ µ (1) h , . . . , µ ( d (cid:48) )
h ] (cid:62) .
Under this assumption, a slightly modiﬁed version of Lifelong-LSVI would use Q kh ( s, a, w )
=
{ r h ( s, a, w ) + (cid:104) ˜
ν
kh , ψ ( s, a, w ) (cid:105) + β (cid:13)(cid:13) ψ ( s, a, w ) (cid:13)(cid:13) ( ˜ Λ k h ) − 1 } + , where ˜ ν
kh =
( ˜ Λ k h ) − 1 (cid:80) k − 1 τ =1 ψ τh .
min { max a ∈A Q kh +
( s τh +1 , a, w τ ) , H } , ˜ Λ k h
= λ I d (cid:48)
+ (cid:80) k − 1 τ =1 ψ τh ψ τh (cid:62) , ψ τh = ψ ( s τh , a τh , w τ ) , and β =
˜ O
( d (cid:48) ) .
However, in Appx.
E, we show how these new algorithm and assumption result in ˜ O ( mdH ) number of planning calls and a regret scaling with ˜ O ( √ m 3 d 3 ) for settings with ψ ( s, a, w ) =
φ ( s, a ) ⊗ ρ ( w ) .
These are worse than the number of planning calls and regret in Theorem 2 of UCBlvd by a factor of O ( m ) .
Remark 4.
A natural follow-up relaxation of Assumption 2 is when the equality holds up to an error of ζ .
In Appx.
F, we show that this relaxation results in a regret ˜ O (cid:16) √ mdTζ
+ (cid:112) λ ( d 3 + md 2 ) H 3 T (cid:17) and the same number of planning calls as that in Theorem 2.
When ζ is sufﬁciently small, i.e., ζ = O ( (cid:112) d 2 H 3 /mT ) , UCBlvd will still enjoy a regret of the same order as that in Theorem 2.
4. P ROOF SKETCH OF T HEOREM
Because the proof of planning calls’ upper bound follows standard arguments in low switching cost analysis of Abbasi-Yadkori et al.
(2011), in this section, we focus on the proof sketch for the regret bound.
We start by introducing the high probability event E 1 , which is the foundation of our analysis:
[Equation]
The following lemma highlights the importance of the carefully designed planning step in (8), which ensures good estimators for ξ V ∗ h +1 h without the need of bonus term (cid:13)(cid:13) ψ
( s, a, w ) (cid:13)(cid:13) (cid:16) ˜ Λ kh (cid:17) − 1 .
This step saves a factor O ( m ) in planning calls and regret.
Lemma 1.
Let (cid:102) W = { w τ : τ ∈
[ K ] } ∪ { w ( j ) : j ∈ [ n ] } .
Under the setting of Theorem 2 and conditioned on events {E 1 ( w ) } w ∈ (cid:102) W deﬁned in (9) , for all ( s, a, w, h, k ) ∈ S × A × (cid:102) W ×
[ H ] ×
[ K ] , it holds that (cid:12)(cid:12)(cid:12)(cid:12) (cid:104)
ˆ ξ k h , ψ ( s, a, w ) (cid:105) − P h
[ V kh +
( ., w )]
( s, a ) (cid:12)(cid:12)(cid:12)(cid:12) ≤ 2 Lβ
(cid:13)(cid:13)
φ ( s, a ) (cid:13)(cid:13) ( Λ kh ) − 1 .
As the ﬁnal step in the regret analysis, we use Lemma 1 to prove the optimistic nature of UCBlvd, i.e.,
Q kh ( s, a, w k ) ≥ Q ∗ h ( s, a, w k ) for all ( s, a, h, k ) ∈ S
×
A ×
[ H ] ×
[ K ] .
Then following the standard analysis of single-task LSVI-UCB we derive the regret bound in Theorem 2.
4. E XPERIMENTS
We implemented our main algorithm UCBlvd on synthetic environments and compared its performance with the warm-up algorithm Lifelong-LSVI, which is viewed as an idealized baseline ignoring the computational complexity.
In all the experiments, the same setting, task sequences and feature mappings were used for both UCBlvd and Lifelong-LSVI.
Figure 1a depicts per-episode rewards for the main setup considered throughout the paper, and Figure 1b shows those for the setup in Remark 2.
The plots verify that Lifelong-LSVI and UCBlvd statistically perform almost the same while UCBlvd uses much smaller numbers of planning calls ( 1000 vs ∼ 20 ).
We remark that Lifelong-LSVI has an overall computation complexity of O ( K 2 ) , which makes it not practical for the lifelong learning setting, as its planning complexity increases linearly with the number of samples.
The details on the parameters of simulations are deferred to Appx.
H.
[Figure]
R ELATED WORK
We consider the regret minimization setup of lifelong RL under the contextual MDP framework, where the agent receives tasks speciﬁed by contexts in sequence and needs to achieve a sublinear regret for any task sequence.
Below, we contrast our work with related work in the literature.
Lifelong RL.
Generally lifelong RL studies how to learn to solve a streaming sequence of tasks using rewards.
While it was originally motivated by the need of endless learning of robots (Thrun &
Mitchell, 1995), historically many works on lifelong RL (Ammar et al., 2014; Brunskill & Li, 2014;
Abel et al., 2018a;b; Lecarpentier et al., 2021) assume that the tasks are i.i.d.
(similar to multi-task
RL; see below).
There are works for adversarial sequences, but most of them assume ﬁnite number of tasks (Brunskill & Li, 2015; Ammar et al., 2015; Zhan et al., 2017) or are purely empirical (Xie &
Finn, 2021).
The work by Isele et al.
(2016) uses contexts to enable zero-shot learning like here, but it (as well as most works above) does not provide formal regret guarantees.
4 Brunskill & Li (2015) and Xie & Finn (2021) assume the task identity is latent, which requires additional exploration; in this sense, their problem is harder than the setup here where the task context is revealed.
Extending the setup here to consider latent context is an important future direction.
Contextual MDP and multi-objective RL.
Our setup is closely related to the exploration problem studied in the contextual MDP literature, though contextual MDP is originally not motivated from the lifelong learning perspective.
A similar mathematical problem appears in the dynamic setup of multi-objective RL (Wu et al., 2021; Abels et al., 2019), which can be viewed as a special case of contextual MDP where the context linearly determines the reward function but not the dynamics.
Most contextual MDP works allow adversarial contexts and initial states, but a majority of them focuses on the tabular setup (Abbasi-Yadkori & Neu, 2014; Hallak et al., 2015; Modi et al., 2018;
Modi & Tewari, 2020; Levy & Mansour, 2022; Wu et al., 2021), whereas our setup allows continuous states.
Kakade et al.
(2020) and Du et al.
(2019) allow continuous state and action spaces, but the former assumes a planning oracle with unclear computational complexity and the latter focuses on only LQG problems.
While generally contextual MDP allows both the reward and the dynamics to vary with contexts, we focus on the effects of context-independent dynamics similar to Kakade et al.
; Wu et al.
In particular, the recent work of Wu et al.
(2021) is the closest to ours, but they study the sample complexity in the tabular setup with linearly parameterized rewards.
In view of
Example 1, their proposed algorithm has a regret bound ˜ O
( (cid:112) min { m, |S|} H |S||A| K ) .
However, they need linear number of planning calls.
On the contrary, our algorithm, UCBlvd, allows continuous states, nonlinear context dependency, and has both sublinear regret and number of planning calls.
Multi-task RL.
Another closely related line of work is multi-task RL.
Compared to our setting, multi-task RL assumes that there are beforehand known ﬁnite tasks and/or they are i.i.d.
samples from a ﬁxed distribution.
For example, in Yang et al.
; Hessel et al.
(2019); Brunskill & Li (2013);
Fifty et al.
; Zhang & Wang (2021); Sodhani et al.
(2021), tasks are assumed to be chosen from a known ﬁnite set, and in Yang et al.
; Wilson et al.
; Brunskill & Li (2013); Sun et al.
(2021), tasks are sampled from a ﬁxed distribution.
By contrast, our setting provides guarantees on regret and number of planning calls for adversarial task sequences.
D ISCUSSION
In this paper, we frame lifelong RL as contextual MDPs and identify a new completeness-style assumption to enable provably efﬁcient lifelong RL with linear representation.
We propose UCBlvd, an algorithm that simultaneously satisﬁes the practical need of achieving 1) sublinear regret and 2) sublinear number of planning calls for 3) any sequence of tasks and initial states.
Speciﬁcally, for K task episodes of horizon H , we prove that UCBlvd has a regret bound ˜ O
( (cid:112) (
d 3 + d (cid:48) d ) H 4 K ) based on ˜ O
( dH log( K )) number of planning calls, where d and d (cid:48) are the feature dimensions of the dynamics and rewards, respectively.
We believe that our results would inspire new research directions in the literature of CMDP and multi-objective RL, as existing work to our knowledge does not cover the computation-sharing aspect of lifelong RL.
That said, our work’s limitations motivate further investigations in the following directions: 1) extension to more general class of MDPs, potentially using general function approximation/representation tools, 2) establishing an information-theoretic lower bound on the number of planning calls/computation complexity. A CKNOWLEDGEMENT This work is partially supported by DARPA grant HR00112190130 and NSF grant 2221871. Sanae
Amani is partially supported by Amazon science hub fellowship.
A P ROOFS OF SECTION
To prove Theorem 1, we will use the high probability event E 2 deﬁned in Lemma 3 to prove the UCB nature of Lifelong-LSVI in Lemma 4, which is the key to controlling the regret.
We ﬁrst state the following lemma that will be used in the proof of Lemma 3.
Lemma 2.
Under the setting of Theorem 1, let c
β
be the constant in the deﬁnition of β .
Then, for a ﬁxed w , there is an absolute constant c 0 independent of c β , such that for all ( h, k ) ∈
[ H ] ×
[ K ] , with probability at least 1 − δ it holds that
[Equation]
where c 0 and c β are two independent absolute constants.
Proof.
We note that (cid:107) η h (cid:107) 2 ≤ √ d (cid:48) (Assumption 1), (cid:13)(cid:13) θ kh
( w ) (cid:13)(cid:13) 2 ≤ H √ d
(Lemma 18), and (cid:13) (cid:13)(cid:13)(cid:13)(cid:16) Λ kh (cid:17) − 1 (cid:13) (cid:13)(cid:13)(cid:13) ≤ 1 λ .
Thus, Lemmas 19 and 21 together imply that for all ( h, k ) ∈
[ H ] ×
[ K ] , with probability at least 1 − δ it holds that
[Equation]
λ
[Equation]
If we let (cid:15) =
dHk and β =
c β ( d + √ d (cid:48) )
H (cid:112) log( dT/δ )
[Equation]
then, there exists an absolute constant C >
that is independent of c β such that
[Equation]
Lemma 3.
Let the setting of Theorem 1 holds.
The event
[Equation]
holds with probability at least 1 − δ for a ﬁxed w .
Proof.
[Equation]
Thus, in order to upper bound (cid:13)(cid:13)(cid:13)(cid:13) θ kh [Equation] w ) − ˜ θ
k h ( w ) (cid:13)(cid:13)(cid:13)(cid:13) Λ kh , we bound (cid:107) q 1 (cid:107) Λ kh and (cid:107) q 2 (cid:107) Λ kh separately.
From Lemma 18, we have
[Equation]
Thanks to Lemma 2, for all ( w, h, k ) , with probability at least 1 − δ , it holds that
[Equation]
[Equation]
where c 0 and c β are two independent absolute constants.
Combining (11) and (12), for all ( w, h, k ) , with probability at least 1 − δ , it holds that
[Equation]
[Table]
for some absolute constant c > 0 .
Lemma 4.
Let (cid:102) W = { w 1 , w 2 , . . .
, w K } .
Under the setting of Theorem 1 and conditioned on events {E 2 ( w ) } w ∈ (cid:102) W deﬁned in (10) , and with Q kh computed as in (6) , it holds that Q kh ( s, a, w ) ≥
Q ∗ h ( s, a, w ) for all ( s, a, w, h, k ) ∈ S
×
A × (cid:102) W ×
[ H ] ×
[ K ] .
Proof.
We ﬁrst note that conditioned on events {E 2 ( w ) } w ∈ (cid:102) W , for all ( s, a, w, h, k ) ∈ S ×
A × (cid:102) W ×
[ H ] ×
[ K ] , it holds that
[Equation]
≤
[Equation]
(Lemma 3) for any policy π .
Now, we prove the lemma by induction.
The statement holds for H because Q kH +1 ( ., ., . )
=
Q ∗ H +1 ( ., ., . )
= 0 and thus conditioned on events {E 2 ( w ) } w ∈ (cid:102) W , deﬁned in (10), for all ( s, a, w, k ) ∈
S ×
A × (cid:102) W ×
[ K ] , we have
[Equation]
[Table]
Therefore, conditioned on events {E 2 ( w ) } w ∈ (cid:102) W , for all ( s, a, w, k ) ∈ S ×
A × (cid:102) W ×
[ K ] , we have
[Equation] [Equation]
Now, suppose the statement holds at time-step h
+
and consider time-step h .
Conditioned on events {E 2 ( w ) } w ∈ (cid:102) W , for all ( s, a, w, h, k ) ∈ S
×
A × (cid:102) W ×
[ H ] ×
[ K ] , we have
[Equation]
(Induction assumption)
Therefore, conditioned on events {E 2 ( w ) } w ∈ (cid:102) W , for all ( s, a, w, h, k ) ∈ S ×
A × (cid:102) W ×
[ H ] ×
[ K ] , we have
[Equation]
s, a, w ) + (cid:68)
[Equation]
[Equation]
This completes the proof.
A. P ROOF OF T HEOREM
Let δ kh
=
V kh ( s kh , w k )
− V π k h ( s kh , w k ) and ξ kh +
=
E (cid:104)
δ kh +1 | s kh , a kh (cid:105) −
δ kh +1 .
Conditioned on events {E 2 ( w ) } w ∈ (cid:102) W , for all ( s, a, w, h, k ) ∈ S
×
A × (cid:102) W ×
[ H ] ×
[ K ] , we have
[Equation]
Note that δ kh ≤ Q kh ( s kh , a kh , w k ) − Q π k h
( s kh , a kh , w k ) .
Thus, combining (13), Lemma 3, and a union bound over (cid:102) W , we conclude that for all ( h, k ) ∈
[ H ] ×
[ K ] , with probability at least 1 − δ , it holds that
[Equation]
Now, we complete the regret analysis
[Equation]
[Equation]
The third inequality is true because of the following: we observe that { ξ kh } is a martingale difference sequence satisfying | ξ kh |≤ 2 H .
Thus, thanks to Azuma-Hoeffding inequality, we have
[Equation] [Equation]
In order to bound (cid:80) Kk =1 (cid:80)
[Equation]
=1 (cid:13)(cid:13)(cid:13) φ kh (cid:13)(cid:13)(cid:13)
[Equation]
Λ kh ) − 1 , note that for any h ∈ [ H ] , we have
K (cid:88)
[Equation]
(Cauchy-Schwartz inequality)
[Equation]
In inequality (15), we used the standard argument in regret analysis of linear bandits (Abbasi-Yadkori et al., 2011, Lemma 11) as follows:
[Equation]
where V
[Equation]
In inequality (16), we used Assumption 1 and the fact that det( A ) =
(cid:81) di
=
λ i ( A ) ≤ (trace( A ) /d )
d .
B P ROOFS OF SECTION
We start by introducing the high probability event E 1 , which is the foundation of our analysis in the following lemma.
Lemma 5.
Follow the setting of Theorem 2.
The event
[Equation]
holds with probability at least 1 − δ for a ﬁxed w .
Proof of Lemma 5 is given in Appx.
B.1.
B. P ROOF OF L EMMA
First, we state the following lemma that will be used in the proof of Lemma 5.
Lemma 6.
Under the setting of Lemma 5, let c β be a constant in the deﬁnition of β .
Then, for a ﬁxed w , there is an absolute constant c 0 independent of c β , such that for all ( h, k ) ∈
[ H ] ×
[ K ] , with probability at least 1 − δ it holds that
[Equation]
where c 0 and c β are two independent absolute constants.
Proof.
We note that (cid:13)(cid:13)(cid:13)(cid:13)
η h + ˆ ξ k h (cid:13)(cid:13)(cid:13)(cid:13) ≤
(1 + H ) √ md and (cid:13)(cid:13)(cid:13)(cid:13)(cid:16) Λ kh (cid:17)
[Equation]
1 (cid:13)(cid:13)(cid:13)(cid:13) ≤ 1 λ .
Thus, Lemmas 19 andtogether imply that for all ( h, k ) ∈
[ H ] ×
[ K ] , with probability at least 1 − δ it holds that
[Equation]
If we let (cid:15) =
dHk and β =
c β ( d + √ md ) H (cid:112) log( dT/δ ) , then, there exists an absolute constant
C > 0 that is independent of c β such that
[Equation]
Now
[Equation]
we begin the formal proof of Lemma 5:
[Equation]
) (cid:17) 
 (cid:124) (cid:123)(cid:122) (cid:125)
q.
Thus, in order to upper bound (cid:13)(cid:13)(cid:13)(cid:13) θ kh ( w ) − ˜ θ
k h ( w ) (cid:13)(cid:13)(cid:13)(cid:13) Λ kh , we bound (cid:107) q 1 (cid:107) Λ kh and (cid:107) q 2 (cid:107) Λ kh separately.
From Lemma 18, we have
[Equation]
Thanks to Lemma 6, for all ( w, h, k ) , with probability at least 1 − δ , it holds that
[Equation] [Equation]
where c 0 and c β are two independent absolute constants.
Combining (19) and (20), for all ( h, k ) ∈
[ H ] ×
[ K ] , with probability at least 1 − δ , it holds that
[Equation]
for some absolute constant c > 0 .
B. P ROOF OF L EMMA
Thanks to Assumption 2 and conditioned on events {E 1 ( w ) } w ∈ (cid:102) W , one set of solution for (8) is (cid:26) θ
[Equation]
w ( j )
[Equation]
j ∈ [ n ] and ξ V kh +1 h with corresponding zero optimal objective value.
Therefore, it holds that
[Equation]
Let (cid:16) s ( i ) , a ( i ) (cid:17) be the i -th element of D and { c (cid:48)
i ( s, a ) } i ∈
[ d ] be the coefﬁcients such that
[Equation]
For any triple ( s, a, j ) ∈ S × A × [ n ] , we have
[Equation]
[Equation]
For any ( s, a, w ) ∈ S × A × W , it holds that
[Equation]
Finally, conditioned on events {E 1 ( w ) } w
[Table]
(cid:102) W , for all ( s, a, w, h, k ) ∈ S
×
A × (cid:102) W ×
[ H ] ×
[ K ] , it
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
∈ [ n ] c j ( w ) (cid:28) ˜ θ k h (cid:16) w ( j ) (cid:17) − θ kh (cid:16) w ( j ) (cid:17) , φ ( s, a ) (cid:29)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (Eqn.
≤ 2 Lβ (cid:13)(cid:13) φ ( s, a ) (cid:13)(cid:13) ( Λ kh ) − 1 .
(Lemma 5)
B. P ROOF OF OPTIMISTIC NATURE OF UCB LVD
Lemma 7.
Let (cid:102) W = { w τ : τ ∈
[ K ] } ∪ { w ( j ) : j ∈ [ n ] } .
Under the setting of Theorem 2 and conditioned on events {E 1 ( w ) } w ∈ (cid:102) W deﬁned in (9) , and with Q kh computed as in (7) , it holds that
Q kh ( s, a, w ) ≥ Q ∗ h ( s, a, w ) for all ( s, a, w, h, k ) ∈ S ×
A × (cid:102) W ×
[ H ] ×
[ K ] .
Proof.
We ﬁrst note that conditioned on events {E 1 ( w ) } w ∈ (cid:102) W , for all ( s, a, w, h, k ) ∈ S ×
A × (cid:102) W ×
[ H ] ×
[ K ] , it holds that
[Equation]
=
[Equation]
(Lemma 1) for any policy π .
Now, we prove the lemma by induction.
The statement holds for H because Q kH +1 ( ., ., . )
=
Q ∗ H +1 ( ., ., . )
=
and thus conditioned events {E 1 ( w ) } w ∈ (cid:102) W , deﬁned in (9), for all ( s, a, w, k ) ∈
S ×
A × (cid:102) W ×
[ K ] , we have
[Equation]
Therefore, conditioned on events {E 1 ( w ) } w ∈
[Equation]
W , for all ( s, a, w, k ) ∈ S ×
A × (cid:102) W ×
[ K ] , we have
[Equation]
where the ﬁrst equality follows from the fact that Q ∗ H ( s, a, w ) ≥ 0 .
Now, suppose the statement holds at time-step h
+
and consider time-step h .
Conditioned on events {E 1 ( w ) } w ∈ (cid:102) W , for all ( s, a, w, h, k ) ∈ S
×
A × (cid:102) W ×
[ H ] ×
[ K ] , we have
[Equation]
(Induction assumption)
Therefore, conditioned on events {E 1 ( w ) } w ∈ (cid:102) W , for all ( s, a, w, h, k ) ∈ S ×
A × (cid:102) W ×
[ H ] ×
[ K ] , we have
[Equation]
where the ﬁrst equality follows from the fact that Q ∗ h ( s, a, w ) ≥ 0 .
This completes the proof.
B. P ROOF OF T HEOREM
First, we bound the number of times Algorithm 2 updates ˆ ξ k h , i.e., number of planning calls.
Let P be the total number of updates and k p be the episode at which, the agent did replanning for the p -th time.
Note that det Λ 1 h
= λ d and det Λ Kh ≤ trace( Λ Kh /d )
d ≤ (cid:16) λ
+ Kd (cid:17) d , and consequently:
[Equation]
, and therefore
H (cid:89) h
[Equation]
det
[Equation]
Since 1 ≤ det Λ kph det Λ kp − 1 h for all p ∈
[ P ] , we can deduce from (24) that ∃ h
[Equation]
[ H ] such that e < det Λ k h det Λ ˜ kh happens for at most dH log (cid:16) 1 + Kdλ (cid:17) number of episodes k ∈
[ K ] .
This concludes that the number of planing calls in UCBlvd is dH log (cid:16) 1 + K dλ (cid:17) .
Now, we prove the regret bound.
Let δ kh
=
V ˜ kh
( s kh , w k ) − V π k h ( s kh , w k ) and ξ kh +1 =
E (cid:104)
δ kh +1 | s kh , a kh (cid:105) −
δ kh +1 .
Conditioned on events {E 1 ( w ) } w ∈ (cid:102) W , for all ( s, a, w, h, k ) ∈ S
×
A × (cid:102) W ×
[ H ] ×
[ K ] , we have Q
[Equation]
Note that δ kh ≤ Q ˜ kh
( s kh , a kh , w k ) − Q π k h ( s kh , a kh , w k ) .
Thus, combining (25), Lemma 5, and a union bound over (cid:102) W , we conclude that for all ( h, k ) ∈
[ H ] ×
[ K ] , with probability at least 1 − δ , it holds that gives
[Equation]
Note that for any positive semi-deﬁnite matrices A , B , and C such that A = B + C , we have:
[Equation]
and for any x (cid:54) =
((Abbasi-Yadkori et al., 2011, Lemm
[Equation]
[Equation]
[Equation]
Now, we complete the regret analysis following similar steps as those of Theorem 1’s proof:
[Equation]
Lemma 7)
[Equation]
B. D ISCUSSION ON THE TIME COMPLEXITY OF UCB LVD AND L IFELONG -LSVI
In what follows, we clarify on how the time complexity of UCBlvd compares to that of Lifelong
LSVI.
When we compute (cid:16) Λ kh (cid:17) − 1 by the Sherman-Morrison formula, the computational complexity of Lifelong-LSVI is dominated by Line 5 in computing max a ∈A Q k h
+
( s τ h
+1 , a ) for all τ ∈ [ k ] .
This takes O ( d 2 |A| K ) per step, which gives a total runtime
O ( d 2 |A| HK 2 ) .
In UCBlvd, every planning call takes ˜ O
( md 2 |A| K + m 3 d 3 ) , where the second term is the time-complexity of thE convex QCQP with m + 1 constraints and 2 md variables.
This gives a total runtime of ˜ O
( H 2 ( md 3 |A| K + m 3 d 4 )) .
Therefore, UCBlvd enjoys a smaller time complexity by a factor of K compared to that of LifelongLSVI, which is a signiﬁcant reduction in practical scenarios where K >> d (cid:48)
=
md .
1 Set: Q kH +1 ( ., ., . )
= 0 , ∀ k ∈ [ K ] , ˜ k
=2 for episodes k = 1 , . .
K do
Observe the initial state s k 1 and the task context w k .
if ∃ h ∈ [ H ] such that det Λ kh det Λ ˜ kh > e or det ˜ Λ k h det ˜ Λ ˜
k h > e then ˜ k
= k for time-steps h
= H, . .
. , 1 do
Compute ˆ ξ k h as in (30).
for time-steps h = 1 , . . .
, H do
Compute Q ˜ kh
( s kh , a, w k ) for all a ∈ A as in (28).
Play a kh = arg max a ∈A Q ˜ kh
( s kh , a, w k ) and observe s kh +1 and r kh .
C D ETAILS OF R EMARK 1: UCB LVD WITH UNKNOWN REWARDS
In order for our analysis to go through, we need a slightly different completeness assumption as below:
Assumption 4.
Given feature maps φ : S ×A → R d and ψ : S ×A×W → R d (cid:48) , consider function class
[Equation]
Then for any f ∈ F , and h ∈ [ H ] , there exists a vector ξ fh ∈ R d (cid:48) with (cid:13)(cid:13)(cid:13) ξ fh (cid:13)(cid:13)(cid:13) ≤ H √ d
[Equation]
such that
[Equation]
C. O VERVIEW
Let ψ τh = ψ
( s τh , a τh , w τ ) .
UCBlvd with unknown rewards works with the following action-value functions:
[Equation]
where
[Equation]
and ˜
[Equation]
and
[Equation]
[Table]
[Table]
We note that compared to (7), action-value function deﬁned in (28) involves an extra term (cid:68) ˜
η kh , ψ ( s, a, w ) (cid:69) + ˜ β
(cid:13)(cid:13) ψ ( s, a, w ) (cid:13)(cid:13) (˜ Λ kh ) − 1 .
This term is in fact an upper bound on r h ( s, a, w ) .
Specifically, from Theorem 2 in Abbasi-Yadkori et al.
, we know that for ˜ β = √ λmd , it holds that
[Equation]
Theorem 3.
Let T = KH .
Under Assumptions 1, 3, and 4, the number of planning calls in Algorithm 3 is at most dH log (cid:16) 1 + Kdλ (cid:17)
+ mdH log (cid:16) 1 + Kmdλ (cid:17) , and there exists an absolute constant c >such that for any ﬁxed δ ∈ (0 , 0 . 5) , if we set λ = 1 , β
= cH ( md ) (cid:112) log( mdT/δ ) and ˜
β = √ md in
Algorithm 3, then with probability at least 1 − 2 δ , it holds that
[Equation]
C.
N ECESSARY ANALYSIS FOR THE PROOF OF T HEOREM
Lemma 8.
Let c β be a constant in the deﬁnition of β .
Then, under Assumptions 1, 3, and 4, for a ﬁxed w , there is an absolute constant c 0 independent of c β , such that for all ( h, k ) ∈
[ H ] ×
[ K ] , with probability at least 1 − δ it holds that
[Equation]
where c 0 and c β are two independent absolute constants.
Proof.
We note that (cid:13)(cid:13)(cid:13)(cid:13) ˜
η kh + ˆ ξ k h (cid:13)(cid:13)(cid:13)(cid:13) ≤ H √ md + K/λ and (cid:13)(cid:13)(cid:13)(cid:13)(cid:16) Λ kh (cid:17) − 1 (cid:13)(cid:13)(cid:13)(cid:13) ≤ 1 λ and (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:18) ˜
Λ k h (cid:19) − 1 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ 1 λ .
Thus, Lemmas 19 and 23 together imply that for all ( h, k ) ∈
[ H ] ×
[ K ] , with probability at least 1 − δ it holds that (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) k − 1 (cid:88) τ
= φ τh (cid:16) V kh
[Equation]
If we let (cid:15) =
dHk and β =
c β ( md ) H (cid:112) log( mdT/δ ) , then, there exists an absolute constant C >
that is independent of c β such that
[Equation]
Lemma 9.
Under Assumptions 1,
[Equation]
and 4, if we let β
= cmdH (cid:112) λ log( mdT/δ ) with an absolute constant c > 0 , then the event
[Equation]
holds with probability at least 1 − δ for a ﬁxed w .
Proof.
The proof follows the same steps as those of Lemma 5, except that it uses Lemma 8 instead of Lemma 6 due to different structure of action-value functions Q kh in this section.
Lemma 10.
Let (cid:102) W = { w τ : τ ∈
[ K ] } ∪ { w ( j ) : j ∈ [ n ] } .
Under the setting of Theorem 3 and conditioned on events {E 3 ( w ) } w ∈ (cid:102) W deﬁned in (32) , for all ( s, a, w, h, k ) ∈ S ×A× (cid:102) W ×
[ H ] ×
[ K ] , it holds that
[Equation]
Proof.
The proof follows the exact same steps as those of Lemma 1’s proof.
Lemma 11.
Let (cid:102) W = { w τ : τ ∈
[ K ] } ∪ { w ( j ) : j ∈ [ n ] } .
Under the setting of Theorem 3 and conditioned on events {E 3 ( w ) } w ∈ (cid:102) W deﬁned in (32) , and with Q kh computed as in (28) , it holds that
Q k h ( s, a, w ) ≥ Q ∗ h ( s, a, w ) for all ( s, a, w, h, k ) ∈ S ×
A × (cid:102) W ×
[ H ] ×
[ K ] .
Proof.
We ﬁrst note that conditioned on events {E 3 ( w ) } w ∈ (cid:102) W , for all ( s, a, w, h, k ) ∈ S ×
A × (cid:102)
[Table]
[ H ] × [ K ] , it holds that
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
, (Lemma 10) for any policy π .
Now, we prove the lemma by induction.
The statement holds for H because Q kH +1 ( ., ., . )
=
Q ∗ H +1 ( ., ., . )
=
and thus conditioned events {E 3 ( w ) } w ∈ (cid:102) W , deﬁned in (32), for all ( s, a, w, k ) ∈
S ×
A × (cid:102) W ×
[ K ] , we have
[Equation]
Therefore, conditioned on events {E 3 ( w ) } w ∈ (cid:102) W , for all ( s, a, w, k ) ∈ S ×
A × (cid:102) W ×
[ K ] , we have
Q ∗ H ( s, a, w ) ≤
[Equation]
= Q kH ( s, a, w ) , where the ﬁrst equality follows from the fact that Q ∗ H ( s, a, w ) ≥ 0 .
Now, suppose the statement holds at time-step h
+
and consider time-step h .
Conditioned on events {E 3 ( w ) } w ∈ (cid:102) W , for all
[Equation]
s, a, w, h, k ) ∈ S ×
A × (cid:102) W ×
[ H ] ×
[ K ] , we have
[Equation]
(Induction assumption)
Therefore
[Equation]
conditioned on events {E
[Equation]
, for all [Equation] [Equation] × [ H ] × [ K ] , we have
[Equation]
where the ﬁrst equality follows from the fact that Q ∗ h ( s, a, w ) ≥ 0 .
This completes the proof.
C. P ROOF OF T HEOREM
First, we bound the number of times Algorithm 3 updates ˆ ξ k h , i
[Equation]
e., number of planning calls.
Let P be the total number of policy updates and k p be the episode at, the agent did replanning for the p -th time.
Note that det Λ 1 h
= λ d and det Λ Kh ≤ trace( Λ Kh /d )
d ≤ (cid:16) λ
+ Kd (cid:17) d , and consequently:
[Equation]
and therefore
H
[Equation]
We similarly have
[Equation]
[Equation]
Since 1 ≤ det Λ kph det Λ kp
[Equation] for all p ∈ [ P ] , we can deduce from (34) and (35) that ∃
[Equation]
] such that e < det Λ kh
[Equation]
[Equation]
happens for at most dH log (cid:16) 1 + Kdλ (cid:17) + mdH log (cid:16) 1 + Kmdλ (cid:17) number of episodes k ∈
[ K ] .
This concludes that number of planning calls in Algorithm 3 is at most dH log (cid:16) 1 + Kdλ (cid:17) + mdH log (cid:16) 1 + Kmdλ (cid:17) .
Now, we prove the regret bound.
Let δ kh
=
V ˜ kh
( s kh , w k ) − V π k h ( s kh , w k ) and ξ kh +1 =
E (cid:104)
δ k h
+1 | s k h , a k h (cid:105) − δ k h
+1 .
Conditioned on events {E 3 ( w ) } w ∈ (cid:102) W , for all ( s, a, w, h, k ) ∈ S ×
A ×
[Equation]
W × [ H ] × [ K ] , we have
[Equation]
[Equation]
Note that δ kh ≤ Q ˜ kh
( s kh , a kh , w k ) − Q π k h ( s kh , a kh , w k ) .
Thus, combining (37), Lemma 9, and a union bound over (cid:102) W , we conclude that for all ( h, k ) ∈
[ H ] ×
[ K ] , with probability at least 1 − δ , it holds that gives
[Equation]
Now, we complete the regret analysis following similar steps as those of Theorem 1’s proof:
[Equation]
Lemma 11)
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13) (cid:16)
[Equation]
(cid:17)
[Equation]
(cid:88)
[Equation]
(cid:88)
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
[Equation]
kh
[Equation]
) (cid:13)(cid:13)(cid:13) (cid:18)
[Equation]
(cid:19) [Equation] (cid:118)(cid:117)(cid:117)(cid:116) det [Equation]
det
[Equation]
(cid:88)
[Equation]
=
[Equation]
(cid:88)
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
( s
[Equation]
a kh
[Equation]
w k ) [Table] ˜ Λ kh [Table]
−
[Table]
det ˜ Λ k h det ˜ Λ
˜ k h
(Eqn. (27))
≤ 2 H (cid:112) T log( dT/δ )
+ 4 H √ K (cid:16)
Lβ (cid:112)
d log(1 + K/λ ) + ˜ β (cid:112) 2 md log(
+ K/λ ) (cid:17)
≤ ˜ O (cid:16) L √ λm 2 d 3 H 3 T (cid:17) .
D D ETAILS OF R EMARK 2: R ELAXATION OF A SSUMPTION
In this section, we replace Assumption 3 with the following assumption:
Assumption 5.
There is a known set { w (1) , w (2) , . . .
, w ( n ) } of n ≤ d (cid:48) tasks such that ψ ( s, a, w ) ∈
Span (cid:18)(cid:110) ψ ( s, a, w ( j ) ) (cid:111)
j ∈ [ n ] (cid:19) for all ( s, a, w ) ∈ S
× A × W .
This implies that for any ( s, a, w ) ∈
S ×
A × W , there exist coefﬁcients { c j ( s, a, w ) } j ∈ [ n ] such that
[Equation]
Moreover, (cid:80) j ∈ [ n ] (cid:12)(cid:12) c
j ( s, a, w ) (cid:12)(cid:12) ≤
L for all ( s, a, w ) ∈ S
× A × W .
Deﬁne the concatenated mapping
[Equation]
ψ :
[Equation]
→
[Equation]
d
[Equation] d [Equation] such that ˜ ψ ( s, a, w ) = (cid:2) φ ( s, a ) (cid:62) , ψ ( s, a, w ) (cid:62) (cid:3) (cid:62) .
For any w ∈
W , deﬁne D ( w ) =
(cid:110) ( s, a ) :
˜ ψ ( s, a, w ) are d + d (cid:48) linearly independent vectors .
(cid:111) .
Given Assumption 5, we mod-
1 Set: Q kH +1 ( ., ., . )
= 0 , ∀ k ∈ [ K ] , ˜ k
=2 for episodes k = 1 , . .
K do
Observe the initial state s k 1 and the task context w k .
if ∃ h ∈ [ H ] such that det Λ kh det Λ ˜ kh > e then ˜ k
= k for time-steps h
= H, . .
. , 1 do
Compute ˆ ξ k h as in (39).
for time-steps h = 1 , . . .
, H do
Compute Q ˜ kh
( s kh , a, w k ) for all a ∈ A as in (7).
Play a kh = arg max a ∈A Q ˜ kh
( s kh , a, w k ) and observe s kh +1 and r kh .
ify the planning step of UCBlvd to the following:
[Equation]
The only change we make in Algorithm 2 is in Line 9, in which ˆ ξ k h is now computed as deﬁned in (39).
We present this modiﬁcation in Algorithm 4 for completeness.
Theorem 4.
Let T = KH .
Under Assumptions 1, 2, and 5, the number or planning calls in
Algorithm 4 is at most dH log (cid:16) 1 + Kdλ (cid:17) and there exists an absolute constant c > 0 such that for any ﬁxed δ ∈ (0 , 0 . 5) , if we set λ = 1 and β = cH (cid:16) d + √ d (cid:48) (cid:17) (cid:112) λ log( dd (cid:48)
T/δ ) in Algorithm 4, then with probability at least 1 − 2 δ , it holds that
[Equation]
Proof of Theorem 4 follows exactly the same steps as those of Theorem 2.
The only difference is the proof of Lemma 1, which we clarify in the proof of following lemma.
Lemma 12.
Let (cid:102) W = { w τ : τ ∈
[ K ] } ∪ { w ( j ) : j ∈ [ n ] } .
Under Assumptions 1, 2, and 5, if we let β = cH (cid:16) d + √ d (cid:48) (cid:17) (cid:112) λ log( dd (cid:48)
T/δ ) with an absolute constant c > 0 , then for all ( s, a, w, h, k ) ∈ S
×
A × W ×
[ H ] ×
[ K ] with probability at least 1 − δ , it holds that
[Equation]
≤ 2 Lβ (cid:13)(cid:13) φ ( s, a ) (cid:13)(cid:13) ( Λ kh ) − 1 .
Proof.
We let
[Equation]
be the i -th element of ˜ D ( w ) =
(cid:110)
˜
ψ ( s, a, w ) : ( s, a ) ∈ D ( w ) (cid:111) and for any triple ( s, a, w )
∈
S ×A×W , we let { c (cid:48)
i ( s, a, w ) } i ∈
[ d + d (cid:48) ] be the coefﬁcients such that
[Equation]
which implies that
[Equation]
Thanks to Assumption 2 and conditioned on events {E 1 ( w ) } w ∈ (cid:102) W , one set of solution for (39) is
[Equation]
and ξ V
[Equation]
with corresponding zero optimal objective value.
Therefore, it holds that
[Equation]
Moreover, for any triple ( s, a, j ) ∈ S ×
A × [ n ] , we have
[Equation]
For any ( s, a, w ) ∈ S × A × W , it holds that
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
holds that [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Eqn
[Equation]
[Equation]
Lβ (cid:13)(cid:13) φ (
[Table]
[Equation]
a ) (cid:13)(cid:13) (
[Table]
k
[Table]
) −
[Equation]
Lemma 5) 1 Set: Q kH +1 ( ., ., . ) = 0 , ∀ k ∈ [ K ] , ˜ k
=2 for episodes k = 1 , . . K do
Observe the initial state s k 1 and the task context w k . if ∃ h ∈ [ H ] such that det ˜ Λ k h det ˜ Λ ˜ k h > e then ˜ k = k for time-steps h
= H, . .
. , 1 do Compute Compute ˜ ν ˜ kh as in (49). for time-steps h = 1 , . . . , H do
Compute Q ˜ kh ( s kh , a, w k ) for all a ∈ A as in (48). Play a kh = arg max a ∈A Q ˜ kh ( s kh , a, w k ) and observe s kh +1 and r kh .
E D ETAILS OF R EMARK
In this section, we only rely on the following two assumptions:
Assumption 6.
Given a feature map ψ : S ×
A × W → R d (cid:48) , consider function class F
[Equation]
[Equation] Then for any f ∈ F and h ∈ [ H ] , there exists a vector ν fh ∈ R d (cid:48) with (cid:13)(cid:13)(cid:13)
ν fh (cid:13)(cid:13)(cid:13) 2 ≤ H √ d (cid:48) such that
[Equation]
Moreover, for every h ∈ [ H ] , there exists a vector η h such that r h
( s, a, w ) =
(cid:10) η h , ψ ( s, a, w ) (cid:11) .
Assumption 7.
Without loss of generality, (cid:13)(cid:13) ψ ( s, a, w ) (cid:13)(cid:13) 2 ≤ 1 for all ( s, a, w ) ∈ S ×
A × W , and (cid:107) η h (cid:107) 2 ≤ √ d (cid:48) for all h ∈
[ H ] .
E. O VERVIEW
Let ψ τh = ψ
( s τh , a τh , w τ ) .
Standard Lifelong-LSVI with computation sharing works with the following action-value functions:
[Equation]
48) where
[Equation]
and
[Equation]
Theorem 5.
Let T = KH .
Under Assumptions 6 and 7, the number of planning calls in 5 is at most d (cid:48)
[Equation] λ (cid:17) and there exists an absolute constant c > 0 such that for any ﬁxed δ ∈ (0 , 0 . 5) , if we set λ = 1 and β = cd (cid:48) H (cid:112) log( d (cid:48) T/δ ) in Algorithm 5, then with probability at least 1 − 2 δ , it holds that
[Equation]
E. N ECESSARY ANALYSIS FOR THE PROOF OF T HEOREM
Thanks to Assumption 6, we have
[Equation]
where ν
kh =
ν V kh +1 h .
Lemma 13.
Let c β be a constant in the deﬁnition of β .
Then, under Assumption 7, there is an absolute constant c 0 independent of c β , such that for all ( h, k ) ∈
[ H ] ×
[ K ] , with probability at least 1 − δ it holds that
[Equation]
where c 0 and c β are two independent absolute constants.
Proof.
We note that (cid:13)(cid:13)(cid:13)
[Equation]
2 ≤
[Equation]
1 + H
[Equation]
√ d (cid:48) and (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:18) ˜ Λ k h (cid:19) − 1 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ 1 λ .
Thus, Lemmas 19 andtogether imply that for all ( h, k ) ∈
[ H ] ×
[ K ] , with probability at least 1 − δ it holds that
[Equation]
λ
[Equation]
If we let (cid:15) =
dHk and β =
c β ( d (cid:48)
+ √ d (cid:48) )
H (cid:112) log( dT/δ )
[Equation]
then, there exists an absolute constant C >
that is independent of c β such that
[Equation]
Lemma 14.
Under Assumptions 6 and 7, if we let β = cd (cid:48) H (cid:112)
λ log( d (cid:48) T/δ ) with an absolute constant c > 0 , then the event
[Equation]
holds with probability at least 1 − δ .
Proof.
[Equation]
[Equation]
Eqn. Thus, in order to upper bound (cid:13)(cid:13)(cid:13) ν kh − ˜ ν kh
[Equation]
) (cid:13)(cid:13)(cid:13) ˜
Λ kh , we bound (cid:107) q 1 (cid:107) ˜
Λ kh and (cid:107) q 2 (cid:107)
˜ Λ kh separately.
From Assumption 7, we have
[Equation]
Thanks to Lemma 13, for all ( h, k ) ∈
[ H ] ×
[ K ] , with probability at least 1 − δ , it holds that
[Equation]
where c 0 and c β are two independent absolute constants.
Combining (52) and (53), for all ( h, k ) ∈
[ H ] ×
[ K ] , with probability at least 1 − δ , it holds that (cid:13)(cid:13)(cid:13) [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
for some absolute constant c > 0 .
Lemma 15.
Let the setting of Lemma 14 holds.
Conditioned on events E 4 deﬁned in (51) , and with Q kh computed as in (48) , it holds that Q kh ( s, a, w ) ≥ Q ∗ h ( s, a, w ) for all ( s, a, w, h, k ) ∈
S ×
A × W ×
[ H ] ×
[ K ] .
Proof.
We ﬁrst note that conditioned on the event E 4 , for all ( s, a
, w, h, k ) ∈ S ×A×W × [ H ] × [ K ] , it holds that
[Equation]
[Equation]
for any policy π .
Now
[Equation]
we prove the lemma by induction.
The statement holds for H because Q kH +1 ( ., ., . )
=
Q ∗ H +1 ( ., ., . )
= 0 and thus conditioned on the event E 4 , deﬁned in (51), for all ( s, a, w, k ) ∈
S ×
A × W ×
[ K ] , we have
[Equation]
Therefore, conditioned on the event E 4 , for all ( s, a, w, k ) ∈ S ×
A × W ×
[ K ] , we have
[Equation]
where the ﬁrst equality follows from the fact that Q ∗ H ( s, a, w ) ≥ 0 .
Now, suppose the statement holds at time-step h
+
and consider time-step h .
Conditioned on events E 4 , for all ( s, a, w, h, k ) ∈
S ×
A × W ×
[ H ] ×
[ K ] , we have
[Equation]
(Induction assumption)
Therefore, conditioned on events E 4 , for all ( s, a, w, h, k ) ∈ S
×
A × W ×
[ H ] ×
[ K ] , we have
[Equation]
where the ﬁrst equality follows from the fact that Q ∗ H ( s, a, w ) ≥ 0 .
This completes the proof.
E. P ROOF OF T HEOREM
First, we bound the number of times Algorithm 5 updates ˜ ν
kh .
Let P be the total number of updates and k p be the episode at which, the agent did replanning for the p -th time.
Note that det ˜ Λ 1 h
= λ d (cid:48) and det ˜ Λ K h ≤ trace( ˜ Λ K h /d (cid:48) )
d (cid:48) ≤ (cid:16) λ
[Equation]
(cid:48) (cid:17) d (cid:48) , and consequently:
[Equation]
and therefore
H (cid:89)
[Equation]
Since 1 ≤ det ˜ Λ kp h det [Equation] Λ [Equation]
[Equation] for all p ∈ [ P ] , we can deduce from (54) that
∃ h ∈
[ H ] such that e < det ˜ Λ
[Equation]
h happens for at most d (cid:48)
H log (cid:16) 1 + Kd (cid:48) λ (cid:17) number of episodes k ∈
[ K ] .
This concludes that number of planning calls in Algorithm 5 is at most d (cid:48)
H log (cid:16) 1 + Kd (cid:48) λ (cid:17) .
Now, we prove the regret bound.
Let δ kh
=
V ˜ kh
( s kh , w k ) − V π k h ( s kh , w k ) and ξ kh +1 =
E (cid:104)
δ kh +1 | s kh , a kh (cid:105) −
δ kh +1 .
Conditioned on E 4 , for all ( s, a, w, h, k ) ∈ S
×
A × W ×
[ H ] ×
[ K ] , we have
[Equation]
Note that δ ˜ kh ≤ Q kh ( s kh , a kh , w k ) − Q π k h
( s kh , a kh , w k ) .
Thus, (55) and Lemma 14 imply that for all ( h, k ) ∈
[ H ] ×
[ K ] , it holds that
[Equation]
Now, we complete the regret analysis following similar steps as those of Theorem 1’s proof:
[Equation]
Lemma 15)
[Equation]
F D ETAILS OF R EMARK 4: A MISSPECIFIED SETTING
We ﬁrst present a deﬁnition for an approximate completeness model.
Assumption 8 ( ζ -Approximate Completeness) .
Given feature maps φ : S ×
A → R d and ψ :
S ×
A × W → R d (cid:48) in Assumption 1, consider the function class
F =
[Equation]
For any f ∈ F and h ∈ [ H ] [Equation] there exists a vector ξ fh ∈ R d (cid:48) with (cid:13)(cid:13)(cid:13) ξ fh (cid:13)(cid:13)(cid:13) ≤ H √ d (cid:48) such that for all ( s, a, w ) ∈ S ×
A × W (cid:12)
[Equation]
Theorem 6.
Let T = KH .
Under Assumptions 1, 8, and 3, the number of planning calls in Algorithm 2 is at most dH log(1 + Kdλ ) , and there exists an absolute constant c > 0 such that for any ﬁxed δ ∈ (0 , 0 . 5) , if we set λ
= 1 and β = cH
( d + √ md ) (cid:112) log( mdT/δ ) in Algorithm 2, then with probability at least 1 − 2 δ , it holds that
[Equation]
F. N ECESSARY ANALYSIS FOR THE PROOF OF T HEOREM
Let (cid:16) s ( i ) , a ( i ) (cid:17) be the i -th element of D and { c (cid:48)
i ( s, a ) } i ∈
[ d ] be the coefﬁcients such that
[Equation]
Then, L φ is a positive constant such that (cid:80) i ∈
[ d ] (cid:12)(cid:12) c (cid:48)
i ( s, a ) (cid:12)(cid:12) ≤
L φ for all ( s, a ) ∈ S × A .
Lemma 16.
Let (cid:102) W = { w τ : τ ∈
[ K ] } ∪ { w ( j ) : j ∈ [ n ] } .
Under the setting of Theorem 6 and conditioned on events {E 1 ( w ) } w ∈ (cid:102) W deﬁned in (9) , for all ( s, a, w, h, k ) ∈ S × A × (cid:102) W ×
[ H ] ×
[ K ] , it holds that
[Equation]
Proof.
Thanks to Assumption 8 and conditioned on events {E 1 ( w ) } w ∈ (cid:102) W , one set of feasible parameters for (8) is
[Equation]
and [Equation] [Equation]
such that
[Equation]
For any triple ( s, a, j ) ∈ S × A × [ n ] , we have
[Equation]
Similarly, it holds that
[Equation]
(cid:17)(cid:29) ≥
[Equation]
√ mdζ
[Equation]
, φ ( s, a ) (cid:29) .
Therefore, for any ( s, a, j ) ∈ S ×
A × [ n ] , it holds that (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:28) ˆ ξ k h , ψ (cid:16) s
[Equation]
a
[Equation]
w ( j ) (cid:17)(cid:29)
[Equation]
For any ( s, a, w ) ∈ S × A × W , it holds that
[Equation]
Assumption
[Equation]
Similarly, it holds that P h (cid:104)
V kh
+
( ., w ) [Equation] , a [Equation]
Lζ [Equation] (cid:80) j ∈ [ n ] c j ( w )
[Equation]
φ
[Equation]
s, a )
[Equation]
Therefore, for any ( s, a, w ) ∈ S × A × W , it holds that [Equation]
[Table]
holds that [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Equation] [Table]
(Lemma 5)
As the ﬁnal step in the regret analysis, we state the following lemma which uses Lemma 16 to prove the optimistic nature of UCBlvd.
Then following the standard analysis of single-task LSVI-UCB we derive the regret bound for misspeciﬁed settings.
Lemma 17.
Let (cid:102) W = { w τ : τ ∈
[ K ] } ∪ { w ( j ) : j ∈ [ n ] } .
Under the setting of Theorem 6 and conditioned on events {E 1 ( w ) } w ∈ (cid:102) W deﬁned in (9) , and with Q kh computed as in (7) , it holds that (2 L + L φ √ md )( H − h + 1) ζ + Q kh
( s, a, w ) ≥ Q ∗
h ( s, a, w ) for all ( s, a, w, h, k ) ∈
S ×
A × (cid:102) W ×
[ H ] ×
[ K ] .
Proof.
We ﬁrst note that conditioned on events {E 1 ( w ) } w ∈ (cid:102) W , for all ( s, a, w, h, k ) ∈ S ×
A × (cid:102) W ×
[ H ] ×
[ K ] , it holds that
[Equation]
Now, we prove the lemma by induction.
The statement holds for H because Q kH +1 ( ., ., . )
=
Q ∗ H +1 ( ., ., . )
=
and thus conditioned events {E 1 ( w ) } w ∈ (cid:102) W , deﬁned in (9), for all ( s, a, w, k ) ∈
S ×
A × (cid:102) W ×
[ K ] , we have
[Equation]
Therefore, conditioned on events {E 1 ( w ) } w ∈ (cid:102) W , for all ( s, a, w, k ) ∈ S ×
A × (cid:102) W ×
[ K ] , we have
[Equation]
where the ﬁrst equality follows from the fact that Q ∗ H ( s, a, w ) ≥ 0 .
Now, suppose the statement holds at time-step h
+
and consider time-step h .
Conditioned on events {E 1 ( w ) } w ∈ (cid:102) W , for all ( s, a, w, h, k ) ∈ S
×
A × (cid:102) W ×
[ H ] ×
[ K ] , we have
[Equation]
(Induction assumption)
Therefore, conditioned on events {E 1 ( w ) } w ∈ (cid:102) W , for all ( s, a, w, h, k ) ∈ S ×
A × (cid:102) W ×
[ H ] ×
[ K ] , we have Q
[Equation]
where the ﬁrst equality follows from the fact that Q ∗ h ( s, a, w ) ≥ 0 .
This completes the proof.
F. P ROOF OF T HEOREM
The proof for establishing the upper bound on the number of planning calls for misspeciﬁed settings follows exactly the steps as those in the proof of Theorem 2.
Now, we prove the regret bound.
Let δ kh
= V ˜ kh
( s kh , w k ) − V π k h ( s kh , w k ) and ξ kh +1 =
E (cid:104)
δ kh +1 | s kh , a kh (cid:105) −
δ kh +1 .
Conditioned on events {E 1 ( w ) } w ∈ (cid:102) W , for all ( s, a, w, h, k ) ∈ S
×
A × (cid:102) W ×
[ H ] ×
[ K ] , we have
[Equation]
Note that δ kh ≤ Q ˜ kh
( s kh , a kh , w k ) − Q π k h ( s kh , a kh , w k ) .
Thus, combining (59), Lemma 5, and a union bound over (cid:102) W , we conclude that for all ( h, k ) ∈
[ H ] ×
[ K ] , with probability at least 1 − δ , it holds that gives
[Equation]
Now, we complete the regret analysis following similar steps as those of Theorem 1’s proof:
[Equation]
[Table]
[Equation]
where the last two inequalities follow from the similar steps in the proof of Theorem 1.
G A UXILIARY LEMMAS
Notations.
N (cid:15) ( V ) denotes the (cid:15)
-covering
number of the class V of functions mapping S to R with respect to the distance dist( V, V (cid:48) )
=
sup s (cid:12)(cid:12) V ( s ) − V (cid:48) ( s ) (cid:12)(cid:12) .
Lemma 18 (Bound on Weights θ kh ( w ) ) .
Under Assumption 1, for any set of action-value functions { Q kh } h ∈ [ H ] , and ( w, h, k ) ∈ W ×
[ H ] ×
[ K ] , it holds that
[Equation]
Proof.
Recall that
[Equation]
and
[Equation]
[-S V kh +
-]
Thus, we have
[Equation]
Lemma 19 (Lemma D.4 in Jin et al.
Let { s τ } ∞ τ =1 be a stochastic process on state space
S with corresponding ﬁltration {F τ } ∞ τ =0 .
Let { φ τ } ∞ τ =0 be an R d -valued stochastic process where φ τ ∈ F τ − 1 , and (cid:107) φ τ (cid:107) ≤ 1 .
Let Λ k
= λ I d + (cid:80) k −
1 τ =1 φ τ φ (cid:62) τ .
Then with probability at least 1 − δ , for all k ≥ 0 and V ∈ V such that sup s ∈S (cid:12)(cid:12) V ( s ) (cid:12)(cid:12) ≤ H , we have (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) k (cid:88) τ = φ τ . (cid:16) V ( s τ ) − E (cid:2) V ( s τ ) |F
τ − 1 (cid:3)(cid:17)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) Λ − 1 k ≤ 4 H(cid:32) d 2 log (cid:18) k + λ λ (cid:19) + log (cid:18) N (cid:15) ( V ) δ (cid:19)(cid:33) + 8 k 2 (cid:15)λ .
Lemma 20.
For any (cid:15) > 0 , the (cid:15) -covering number of the Euclidean ball in R d with radius R > 0 is upper bounded by (1 + 2 R/(cid:15) )
d .
Lemma 21.
For a ﬁxed w , let V denote a class of functions mapping from S to R with following parametric form
[Equation]
where the parameters β ∈ R , z ∈ R d (cid:48) , y ∈ R d , and Y ∈ R
d × d satisfy 0 ≤ β ≤ B , (cid:107) z (cid:107) ≤
z , (cid:107) y (cid:107) ≤ y , and (cid:107) Y (cid:107) ≤ λ − 1 .
Assume (cid:13)(cid:13) φ ( s, a ) (cid:13)(cid:13) ≤ 1 and (cid:13)(cid:13) ψ
( s, a, w ) (cid:13)(cid:13) ≤ 1 for all ( s, a, w ) ∈ S
× A × W .
Then log [Equation]
Proof.
First, we reparametrize V by letting ˜ Y =
β 2 Y .
We have
[Equation]
+ (cid:113) φ ( ., a ) (cid:62) ˜ Y φ ( ., a ) , H (cid:27) , for (cid:107) z (cid:107) ≤ z , (cid:107) y (cid:107) ≤ y
[Table]
and (cid:13)(cid:13)(cid:13) Y (cid:13)(cid:13)(cid:13) B 2 λ . any two functions V 1 , V 2 ∈ V with parameters (cid:16) z 1 , y 1 ,
Y 1 (cid:17) and (cid:16) z
(cid:17) , respectively, we have [Table] − z 2 (cid:13)(cid:13)(cid:13) + (cid:13)(cid:13)(cid:13) y 1 − y 2 (cid:13)(cid:13)(cid:13) + (cid:114)(cid:13)(cid:13)(cid:13) ˜ Y 1 − ˜ Y 2 (cid:13)(cid:13)(cid:13)
[Equation]
Let C z and C y be (cid:15)/ 2 -covers of { z ∈ R d (cid:48) :
(cid:107) z (cid:107) ≤ z } and { y ∈ R d : (cid:107) y (cid:107) ≤ y } , respectively, with respect to the 2 -norm, and C Y be an (cid:15) 2 / 4 -cover of { Y ∈ R d × d : (cid:107) Y (cid:107) F ≤ B 2 √ d λ } , with respect to the Frobenius norm.
By Lemma 20, we know
[Equation]
According to (60), it holds that N (cid:15) ( V ) ≤|C z | (cid:12)(cid:12) C y (cid:12)(cid:12)
|C Y | , and therefore log (cid:0) N (cid:15) ( V ) (cid:1) ≤ d (cid:48)
log(1 + 4 z/(cid:15) ) + d log(1 + 4 y/(cid:15) ) + d 2 log (cid:32) 1 + 8 B 2 √ d λ(cid:15)(cid:33) .
Lemma 22.
For a ﬁxed w , let V denote a class of functions mapping from S to R with following parametric form
[Equation]
where the parameters β ∈ R , z ∈ R d (cid:48) and Y ∈ R
d × d satisfy 0 ≤ β ≤ B , (cid:107) z (cid:107) ≤ z , and (cid:107) Y (cid:107) ≤ λ − 1 .
Assume (cid:13)(cid:13) φ ( s, a ) (cid:13)(cid:13) ≤ 1 and (cid:13)(cid:13) ψ ( s, a, w ) (cid:13)(cid:13) ≤ 1 for all ( s, a, w ) ∈ S ×
A × W .
Then log (cid:0) N (cid:15) ( V ) (cid:1) ≤ d [Equation] log [Equation]
1 +
[Equation] z [Equation] + d 2 log
[Equation]
λ(cid:15)(cid:33) .
Proof.
First, we reparametrize V by letting ˜ Y =
β 2 Y .
We have
V ( . ) =
min
[Equation] [Equation]
+
[Equation] ˜ [Equation]
[Equation]
, H
[Equation]
, for (cid:107) z (cid:107) ≤ z , and (cid:13)(cid:13)(cid:13) ˜ Y (cid:13)(cid:13)(cid:13) ≤ B 2 λ .
For any two functions V 1 , V 2 ∈ V with parameters (cid:16) z 1 , ˜ Y
[Table]
(cid:16) z 2 , ˜ Y 2 (cid:17) , respectively, we
[Table]
[Equation]
[Equation]
Let C z be an (cid:15)/ 2 -cover of { z ∈ R d (cid:48) :
(cid:107) z (cid:107) ≤ z } with respect to the 2 -norm, and C Y be an (cid:15) 2 / 4 -cover of { Y ∈ R d
×
d : (cid:107) Y (cid:107) F ≤ B 2 √ d λ } , with respect to the Frobenius norm.
By Lemma 20, we know
[Equation]
According to (61), it holds that N (cid:15) ( V ) ≤|C z ||C Y | , and therefore
[Equation]
Lemma 23. For a ﬁxed w
[Equation]
let V denote a class of functions mapping from S to R with following parametric form
[Equation]
where the parameters β, ˜ β ∈ R , z ∈ R d (cid:48) , Y ∈ R d × d and ˜ Y ∈ R d (cid:48)
× d (cid:48) satisfy 0 ≤ β ≤ B , 0 ≤ ˜ β ≤ ˜ B (cid:107) z (cid:107) ≤
z , (cid:107) Y (cid:107) ≤ λ − 1 and (cid:13)(cid:13)(cid:13) ˜
Y (cid:13)(cid:13)(cid:13) ≤ λ − 1 .
Assume (cid:13)(cid:13) φ ( s, a ) (cid:13)(cid:13) ≤ 1 and (cid:13)(cid:13) ψ
( s, a, w ) (cid:13)(cid:13) ≤for all ( s,
a, w ) ∈ S ×
A × W .
Then [Equation]
Proof.
First, we reparametrize V by letting Z = β 2 Y and ˜ Z = ˜ β 2 ˜ Y .
We have
V ( . )
=
min (cid:26) max [Equation] [Equation] , w )
[Equation]
[Equation]
, for (cid:107) z (cid:107) ≤ z , (cid:107) Z (cid:107) ≤ B 2 λ , and (cid:13)(cid:13)(cid:13)
˜ Z (cid:13)(cid:13)(cid:13) ≤ ˜ B 2 λ .
For any two functions V 1 , V 2 ∈ V with parameters (cid:16) z 1 , Z 1 , ˜ Z 1 (cid:17) and (cid:16) z 2 , Z 2 , ˜
Z 2 (cid:17) , respectively, we have dist
[Table]
V 1 , V 2 ) ≤
[Table]
[Table]
[Table]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] Let C z be an (cid:15)/ 2 -cover of { z ∈ R d (cid:48) :
(cid:107) z (cid:107) ≤ z } with respect to the 2 -norm, C Z be an (cid:15) 2 / 4 -cover of { Z ∈ R d × d : (cid:107) Z (cid:107) F ≤ B 2 √ d λ } , and C ˜
Z be an (cid:15) 2 / 4 -cover of { ˜ Z ∈ R d (cid:48) × d (cid:48) : (cid:13)(cid:13)(cid:13)
˜ Z (cid:13)(cid:13)(cid:13) F ≤ ˜ B 2 √ d λ } with respect to the Frobenius norm.
By Lemma 20, we know
[Equation]
According to (62), it holds that N (cid:15) ( V ) ≤|C z ||C Y | , and therefore
[Equation]
Lemma 24.
Let V denote a class of functions mapping from S to R with following parametric form
[Equation]
z , ψ ( ., a, . )
(cid:11) + 2 Lβ (cid:113) ψ ( ., a, . ) (cid:62) Y ψ ( ., a, . ) (cid:27) + , H (cid:41) , where the parameters β ∈ R , z ∈ R d (cid:48) and Y ∈ R d (cid:48) × d (cid:48) satisfy 0 ≤ β ≤ B , (cid:107) z (cid:107) ≤ z , and (cid:107) Y (cid:107) ≤ λ − 1 .
Assume (cid:13)(cid:13) ψ
( s, a, w ) (cid:13)(cid:13) ≤ 1 for all ( s, a, w ) ∈ S
× A × W .
Then log (cid:0) N (cid:15) ( V ) (cid:1) ≤ d (cid:48) log(1 + 4 z/(cid:15) ) + d (cid:48) 2 log (cid:32) 1 + 8 B 2 √ d (cid:48) λ(cid:15)(cid:33) .
Proof.
First, we reparametrize V by letting ˜ Y =
β 2 Y .
We have
[Equation]
for (cid:107) z (cid:107) ≤ z , and (cid:13)(cid:13)(cid:13) ˜ Y (cid:13)(cid:13)(cid:13) ≤ B 2 λ . For any two functions V 1 , V 2 ∈ V with parameters (cid:16) z 1 , ˜ Y 1 (cid:17) and (cid:16) z 2 , ˜ Y 2 (cid:17) , respectively, we have
[Equation]
Let C z be an (cid:15)/ 2 -cover of { z ∈ R d (cid:48) :
(cid:107) z (cid:107) ≤ z } with respect to the 2 -norm, and C Y be an (cid:15) 2 / 4 -cover of { Y ∈ R d [Equation] × [Equation]
Y
[Equation] B [Equation] d
[Equation]
λ } , with respect to the Frobenius norm.
By Lemma 20, we know
[Equation]
According to (63), it holds that N (cid:15) ( V ) ≤|C z ||C Y | , and therefore
[Equation]
H D ETAILS OF THE EXPERIMENTS
In all the experiments, we have chosen δ = 0 .
01 , λ = 1 , d = 5 , and H = 5 .
The parameters { η h } h ∈ [ H ] are drawn from N (0 , I d (cid:48) ) .
In order to tune parameters { µ h ( . ) } h ∈ [ H ] and the feature mappings φ such that they are compatible with Assumption 1, we consider that the feature space { φ ( s, a ) : ( s, a ) ∈ S × A} is a subset of the d -dimensional simplex, { φ ∈ R d : (cid:80) di
=1 φ i = 1 , φ i ≥ 0 , φ i ≤ 1 , ∀ i ∈ [ d ] } , and e (cid:62) i µ h ( . ) is an arbitrary probability measure over S for all i ∈
[ d ] .
The results shown in Figure 2a depict averages over 50 realizations for the main setup considered throughout the paper with m = 5 and the results shown in Figure 2b depict averages overrealizations, for the more general setup of Remark 2 with d (cid:48)
= 10 .
For the results shown in Figure 2a, the mappings ρ ( w ) are drawn from N (0 , I m ) except for the n = m representative tasks { w ( j ) } j ∈ [ m ] introduced in Assumption 3, for which we set ρ ( w ( j ) )
=
e j for j ∈ [ m ] .
For the results shown in
Figure 2b, the mappings ψ ( s, a, w ) are drawn from N (0 , I d (cid:48) ) and we set ψ ( s, a, w ( j ) )
=
e j for j ∈ [ d (cid:48) ] , where { w ( j ) } j ∈ [ d (cid:48) ] are n = d (cid:48) representative tasks introduced in Assumption 5 in Appx.
D.
The parameters { η h } h ∈ [ H ] are drawn from N (0 , I d (cid:48) ) , where d (cid:48)
= m × d = 25 in Figure 2a.
In our experiments, the exact same settings are used for both UCBlvd and Lifelong-LSVI in both Figures 2a and 2b.
We chose fairly large d , m , and d (cid:48) and by checking online, we noticed that the optimal value of QCQP in (8) happens always to be zero.
All these together suggest that the assumptions made in the paper approximately hold.
Figures 2a and 2b depict the average per-episode reward of UCBlvd and state the average number of planning calls and compare them to those of baseline algorithm
Lifelong-LSVI, a direct extension of LSVI-UCB in Jin et al.
The results emphasize the value of UCBlvd in terms of requiring much smaller numbers of planning calls.
The plots verify that the performances of Lifelong-LSVI and UCBlvd are almost the same statistically, while UCBlvd uses much smaller numbers of planning calls ( 1000 vs ∼ 20 ).
In Figure 3, we plot UCBlvd’s number of planning calls for different number of task episodes, K , while the setting is same as that in 2a.
In this ﬁgure, we empirically verify the logarithmic dependence of number of planning calls on K as suggested by Theorem 2.
[Figure]
[Figure]