N EAR -O PTIMAL R EWARD -F REE E XPLORATION FOR L INEAR M IXTURE MDP S WITH P LUG - IN S OLVER
A BSTRACT
Although model-based reinforcement learning (RL) approaches are considered more sample efﬁcient, existing algorithms are usually relying on sophisticated planning algorithm to couple tightly with the model-learning procedure.
Hence the learned models may lack the ability of being re-used with more specialized planners.
In this paper we address this issue and provide approaches to learn an
RL model efﬁciently without the guidance of a reward signal.
In particular, we take a plug-in solver approach, where we focus on learning a model in the exploration phase and demand that any planning algorithm on the learned model can give a near-optimal policy.
speciﬁcally, we focus on the linear mixture MDP setting, where the probability transition matrix is a (unknown) convex combination of a set of existing models.
We show that, by establishing a novel exploration algorithm, the plug-in approach learns a model by taking ˜ O
( d 2 H 3 /(cid:15) 2 ) episodes with the environment and any (cid:15) -optimal planner on the model gives an O ( (cid:15) ) -optimal policy on the original model.
This sample complexity matches our lower bound for non-plug-in approaches and is statistically optimal .
We achieve this result by leveraging a careful maximum total-variance bound using Bernstein inequality and properties speciﬁed to linear mixture MDPs.
I NTRODUCTION
In reinforcement learning, an agent repeatedly interacts with the unknown environment in order to maximize the cumulative reward.
To achieve this goal, an RL algorithm must be equipped with effective exploration mechanisms to learn the unknown environment and ﬁnd a near-optimal policy.
Efﬁcient exploration is critical to the success of reinforcement learning algorithms, which has been widely investigated from both the empirical and the theoretical perspectives (e.g. Stadie et al.
Pathak et al.
; Azar et al.
; Jin et al.
Model-based RL is one of the important approaches to solve for the RL environment.
In model-based RL, the agent learns the model of the environment and then performs planning in the estimated model.
It has been widely applied in many RL scenarios, including both online setting (Kaiser et al., 2019; Luo et al., 2019; Azar et al., 2017) and ofﬂine setting (Yu et al., 2020; Kidambi et al., 2020).
It is also believed that modelbased RL is signiﬁcantly more sample-efﬁcient than model-free RL, which has been justiﬁed by many recent empirical results (e.g. Kaiser et al.
; Wang et al.
Though the theoretical model-based learning in small scale problems has been studied extensively (Azar et al., 2017; Zhou et al., 2020a; Jin et al., 2020a), it is still far from complete, especially with the presence of a function approximator.
As an important implication of model-based approaches, the power of plug-in approach have been studied in several works (Cui & Yang, 2020; Agarwal et al., 2020).
The idea of plug-in approach is rather simple: We construct an empirical Markov Decision Process (MDP) using maximum likelihood estimate, then return the (approximate) optimal policy with efﬁcient planning algorithm in this empirical model.
The signiﬁcance of plug-in approaches is two-folded.
For one thing, it preserves an empirical model that keeps the value of the policies, which is of independent interests.
For another, the empirical model can be used for any down-stream tasks, which makes the application much more ﬂexible.
It is shown that the plug-in approach achieves the minimax sample complexity to compute the (cid:15) -optimal policies with a generative model in the tabular (Agarwal et al., 2020) and linear settings (Cui & Yang, 2020).
In this paper, we aim to understand the power of plug-in approach in the reward-free exploration with linear function approximation.
We study the linear mixture MDPs, where the transition probability kernel is a linear mixture of a number of basis kernels (Ayoub et al., 2020; Zhou et al., 2020a;b).
We ﬁrst build an empirical model with an estimation of the transition dynamics in the exploration phase, and then ﬁnd a near-optimal policy by planning with the empirical model via arbitrary plug-in solver in the planning phase.
Our setting is different from the reward-free exploration with linear function approximation without plug-in model (Wang et al., 2020; Zanette et al., 2020b), in which the agent can directly observe all history samples and design specialized model-free algorithm in the planning phase.
Our results show that the plug-in approach can achieve near-optimal sample complexity in the reward-free setting.
In particular, we proposed a statistically efﬁcient algorithm for reward-free exploration.
Our algorithm samples ˜
O
( d 2 H 4 /(cid:15) 2 ) trajectories during the exploration phase, which sufﬁces to obtain O ( (cid:15) )
-optimal policies for an arbitrary reward function with an (cid:15) -optimal pluging solver in the planning phase.
Here d is the feature dimension, and H is the planning horizon.
Furthermore, with a more reﬁned trajectory-wise uncertainty estimation, we further improve the sample complexity bound to ˜ O (cid:0) d 2 H 3 /(cid:15) 2 (cid:1) in the regime where d > H and (cid:15) ≤ H/ √ d .
This matches our lower bound Ω( d 2 H 3 /(cid:15) 2 ) for reward-free exploration in linear mixture MDPs, which indicates that our upper bound is near-optimal except for logarithmic factors.
To the best of our knowledge, this is the ﬁrst work that obtains minimax sample complexity bounds for the plug-in approach in reward-free exploration with linear function approximation.
R ELATED W ORK
RL with Linear Function Approximation Reinforcement learning with linear function approximation has been widely studied in the recent few years (e.g. Jiang et al.
; Yang & Wang (2019; 2020); Jin et al.
(2020b);
Modi et al.
Du et al.
; Zanette et al.
(2020a)
; Cai et al.
Ayoub et al.
; Weisz et al.
; Zhou et al.
(2020a;b)).
The linear mixture MDPs model studied in our work assumes the transition probability function is parameterized as a linear function of a given feature mapping over state-action-next-state triple (Ayoub et al., 2020;
Zhou et al., 2020b;a).
Based on the Bernstein inequality for vector-valued martingales, Zhou et al.
(2020a) proposed an efﬁcient algorithm that obtains minimax regret in the regime where d > H .
Besides linear mixture MDPs, linear MDPs is another category of RL with linear function approximation, which assumes both the transition probability function and reward function are parameterized as a linear function of a given feature mapping over state-action pairs.
The algorithms with best regret bounds were proposed by Jin et al.
(2020b)
and Yang & Wang (2020), which studied model-free algorithm and model-based algorithm respectively.
The minimax regret bound for linear MDPs is still unclear.
Reward-Free Reinforcement Learning
In contrast to the standard RL setting, reward-free reinforcement learning separates the exploration problem and the planning problem, which allows one to handle them in a theoretically principled way.
For tabular setting, reward-free reinforcement learning has been well-exploited in many previous results (Jin et al., 2020a; Kaufmann et al., 2021;
M´enard et al., 2020; Zhang et al., 2020; 2021b;
Wu et al., 2021a; Bai & Jin, 2020;
Liu et al., 2021), where the minimax rate is obtained by M´enard et al.
For reward-free exploration with linear function approximation, Wang et al.
(2020) proposed the ﬁrst efﬁcient algorithm that obtains
O ( d 3 H 6 /(cid:15) 2 ) sample complexity for linear MDPs.
However, their algorithm is model-free in nature and cannot guarantee good performance with any plug-in solver.
Further, Qiu et al.
(2021) proposed the ﬁrst provably efﬁcient reward-free algorithm with kernel and neural function approximation.
We also noticed that there is a concurrent work which also studied reward-free exploration for linear mixture MDPs (Zhang et al., 2021a).
Compared with their results, we focus on the setting of the reward-free exploration with plug-in solver, which covers the standard reward-free setting studied in the previous results.
Furthermore, our sample complexity bounds are tighter than theirs by a factor of H 2 1 .
The above two differences introduce new challenges in both the algorithmic design and the complexity analysis in this work, which makes our algorithms much more complicated than theirs.
Besides, our lower bound is tighter than theirs in the dependence on d .
Plug-in Approach
The plug-in approach has been studied in tabular/linear case in restrictive settings.
E.g., Agarwal et al.
and Cui & Yang (2020) studied the standard plug-in approach with a generative model, where the algorithm is allowed to query the outcome of any state action pair from an oracle.
They showed the plug-in approach also achieved the minimax optimal sample complexity to ﬁnd an (cid:15) -optimal policy in both tabular MDPs and linear MDPs.
The reward-free algorithms proposed by Jin et al.
(2020a) are model-based in nature, thus can be regarded as a solution in the plug-in solver setting.
However, their algorithms are restricted to the tabular case and cannot be applied to the setting with linear function approximation.
P RELIMINARIES 3. E PISODIC MDP S
We consider the setting of episodic Markov decision processes (MDPs), which can be denoted by a six-tuple ( S , A , P, R, H, ν ) , where S is the set of states, A is the action set, P is the transition probability matrix so that P h ( ·| s, a ) gives the distribution over states if action a is taken on state s at step h ,
R h ( s, a ) is the deterministic reward function of taking action a on state s with support [0 , 1] in step h , H is the number of steps in each episode, and ν is the distribution of the initial state.
In episode k , the agent starts from an initial state s k, 1 sampled from the distribution ν .
At each step h ∈
[ H ] , the agent observes the current state s k,h ∈ S , takes action a k,h ∈ A , receives reward
R h ( s k,h , a k,h ) , and transits to state s k,h +1 with probability P h
( s k,h +1 | s k,h , a k,h ) .
The episode ends when s
H +1 is reached.
A deterministic policy π is a collection of H policy functions { π h : S → A} h ∈
[ H ] .
We use Π to denote the set of all deterministic policies.
For a speciﬁc reward function R , we use V πh : S × R
→ R to denote the value function at step h under policy π w.r.t.
reward R , which gives the expected sum of the remaining rewards received under policy π starting from s h
= s , i.e.
V πh ( s, R ) = E (cid:104)(cid:80) Hh (cid:48)
=
h R ( s h (cid:48) , π h (cid:48)
( s h (cid:48) )) | s h
= s, P (cid:105) .
Accordingly, we deﬁne Q πh ( s, a, R ) as the expected Q-value function at step h : Q πh ( s, a, R ) =
E (cid:104) R ( s h , a h ) + (cid:80) Hh (cid:48)
=
h +1 R ( s h (cid:48) , π h (cid:48)
( s h (cid:48) )) | s h
= s, a h
= a, P (cid:105) .
We use π ∗ R to denote the optimal policy w.r.t.
reward R, and we use V ∗ h ( · , R ) and Q ∗ h ( · , · , R ) to denote the optimal value and Q-function under optimal policy π ∗ R at step h .
We say a policy π is (cid:15) -optimal w.r.t.
reward R if E (cid:104)(cid:80) [Equation]
(cid:105) [Equation] (cid:104)(cid:80) [Equation]
[Equation]
(cid:105) − (cid:15).
For the convenience of explanation, we assume the agent always starts from the same state s 1 in each episode.
It is straightforward to extend to the case with stochastic initialization, by adding a initial state s 0 with no rewards and only one action a 0 , and the transition probability of ( s 0 , a 0 ) is the initial distribution µ .
We use P h V ( s, a, R ) as a shorthand of (cid:80) s (cid:48)
P h ( s (cid:48) | s, a ) V ( s (cid:48) , R ) .
3. L INEAR M IXTURE MDP S
We study a special class of MDPs called linear mixture MDPs, where the transition probability kernel is a linear mixture of a number of basis kernels (Ayoub et al., 2020; Zhou et al., 2020b;a).
This model is deﬁned as follows in the previous literature.
Deﬁnition 1.
(Ayoub et al., 2020) Let φ ( s, a, s (cid:48) ) :
S ×A×S → R d be a feature mapping satisfying that for any bounded function V : S → [0 , 1] and any tuple ( s, a ) ∈ S × A , we have (cid:107) φ V ( s, a ) (cid:107) 2 ≤ 1 , where φ V ( s, a ) = (cid:80) s (cid:48) ∈S φ ( s, a, s (cid:48) )
V ( s (cid:48) ) .
An MDP is called a linear mixture MDP if there exists a parameter vector θ h ∈ R d with (cid:107) θ h (cid:107) 2 ≤ B for a constant B and a feature vector φ ( · , · , · ) , such that P h ( s (cid:48) | s, a )
= θ (cid:62) h φ ( s, a, s (cid:48) ) for any state-action-next-state triplet ( s, a, s (cid:48) )
∈ S ×
A × S and step
h ∈
[ H ] .
3. R EWARD -F REE R EINFORCEMENT L EARNING
We study the problem of reward-free exploration with plug-in solver.
Our setting is different from the reward-free exploration setting studied in the previous literature.
Formally, there are two phases in this setting: exploration phase and planning phase.
During the exploration phase, the agent interacts with the environment for K episodes.
In episode k , the agent chooses a policy π k which induces a trajectory.
The agent observes the states and actions s k, 1 , a k, 1 , · · · , s k,H , a k,H as usual, but does not observe any rewards.
After K episodes, the agent calculates the estimated model { ˜ P h
=
˜
θ (cid:62) h φ } h ∈
[ H ] , which will be used in the planning phase to calculate the optimal policy.
During the planning phase, the agent is no longer allowed to interact with the MDP.
Also, it cannot directly observe the history samples obtained in the exploration phase.
Instead, the agent is given a set of reward function { R h } h ∈
[ H ] , where R h : S ×
A → [0 , 1] is the deterministic reward in step h .
For notation convenience, we occasionally use R as a shorthand of { R h } h ∈ [ H ] during the analysis.
We deﬁne ˆ V π, ˜
P h ( s, R ) as the value function of transition ˜ P and reward R , i.e. ˆ V π, ˜ P h ( s, R [Equation] =
E (cid:104)(cid:80)
[Equation]
In the planning phase, the agent calculates the optimal policy ˆ π R with respect to the reward function
R in the estimated model ˜ P using any (cid:15) opt -optimal model-based solver.
That is, the returned policy ˆ π R satisﬁes: ˆ
[Equation]
[Equation]
The agent’s goal is to output an accurate model estimation ˜
P after the exploration phase, so that the policy ˆ π R calculated in planning phase can be (cid:15) + (cid:15) opt -optimal w.r.t.
any reward function R .
Compared with the reward-free setting studied in the previous literature (Jin et al., 2020a; Wang et al., 2020; M´enard et al., 2020; Kaufmann et al., 2021), the main difference is that we require that the algorithm maintains a model estimation instead of all the history samples after the exploration phase, and can use any model-based solver to calculate the near-optimal policy in the planning phase.
R EWARD -F REE RL WITH P LUG - IN S OLVER 4. A LGORITHM
The exploration phase of the algorithm is presented in Algorithm 1.
Recall that for a given value function ˜
V k,h
+1 , we have P h
˜
V
k,h +1 ( s k,h , a k,h ) =
θ (cid:62) h
φ ˜ V k,h +1 ( s k,h , a k,h ) for any k, h .
Therefore, ˜ V k,h +1 ( s k,h +1 ) and φ ˜ V k,h +1 ( s k,h , a k,h ) can be regarded as the stochastic reward and the linear feature of a linear bandits problem with linear parameter θ h .
We employ the standard least-square regression to learn the underlying parameter θ h .
In each episode, we ﬁrst update the estimation ˆ θ k based on history samples till episode k − 1 .
We deﬁne the auxiliary rewards R k,h to guide exploration.
We calculate the optimistic Q-function using the parameter estimation ˆ θ k , and then execute the greedy policy with respect to the updated Q-function to collect new samples.
The main problem is how to deﬁne the exploration-driven reward R k,h
( s, a ) , which measures the uncertainty for state-action pair ( s, a ) at the current step.
In the setting of linear mixture MDPs, the linear feature φ V ( s, a ) is a function of both state-action pair ( s, a ) and the next-step value function V .
This is not a big deal in the standard exploration setting (Ayoub et al., 2020; Zhou et al., 2020a).
However, in the reward-free setting, since the reward function is not given beforehand, we need to upper bound the estimation error of value functions for any possible rewards.
To tackle
[Table]
,h ( (cid:80) s (cid:48)
φ ( s, a, s (cid:48) )
V k,h +1 ( s (cid:48) ))
+ R k,h ( s, a ) + u k,h
( s, a ) , H (cid:111)
V k,h ( s ) ← max a ∈A Q k,h ( s, a ) , π k,h ( s ) =
arg max a ∈A Q k,h ( s, a ) end for 15: for step h
= 1 , 2 , · · · , H do
Take action a k,h = π k,h ( s k,h ) and observe s k,
h +1 ∼ P h
( s k,h , a k,h ) end for end for
Find ˜ P
h such that the transition ˜
P h ( ·|· , · ) =
˜ θ (cid:62) h φ ( · , · , · ) is well-deﬁned and (cid:13)(cid:13)(cid:13)
˜ θ h − ˆ θ K,h (cid:13)(cid:13)(cid:13) Λ K,h ≤ β for h ∈ [ H ] 20: Output: { ˜ P h } Hh = this problem, we use max V ∈V β (cid:107) φ V ( s, a ) (cid:107) Λ − 1 k,h = max V ∈V β (cid:113) φ V ( s, a ) (cid:62) (Λ k,h ) − 1 φ V ( s, a ) as a measure of the maximum uncertainty for the state-action pair ( s, a ) , where V =
{ V : S → [0 , H ] } is the set of all possible value functions, and Λ k,h is the summation of all the history samples { s t,h , a t,h , s t,h +1 } k − 1 t =1 with feature φ t,h ( s t,h ,
a t,h ) =
arg max φ V β (cid:107) φ V ( s, a ) (cid:107) Λ − 1 t,h .
In each episode, we deﬁne R k,h ( s, a ) = u k,h
[Equation]
s, a ) = max V ∈V β (cid:107) φ V [Equation] s, a ) (cid:107) Λ − 1 k,h , where R k,h ( s, a ) is the exploration-driven reward used to guide exploration, and u k,h ( s, a ) is the additional bonus term which helps to guarantee that Q k,h ( s, a ) is an optimistic estimation.
Finally, the algorithm returns the model estimation { ˜ P h } h that is well deﬁned (i.e. (cid:80) s (cid:48)
˜
P h ( s (cid:48) | s, a ) = 1 , ˜
P h ( s (cid:48) | s, a ) ≥ 0 , ∀ s, a, s (cid:48) ), and satisfy the constraints (cid:13)(cid:13)(cid:13)
˜ θ h − ˆ θ K,h (cid:13)(cid:13)(cid:13) Λ K,h ≤ β .
4. I MPLEMENTATION D ETAILS
Algorithm 1 involves two optimization problems in line 8 and line 19.
These problems can be formulated as the standard convex optimization problem with a slight modiﬁcation.
Speciﬁcally, the optimization problem in line 8 of Algorithm 1 can be formulated in the following way: [Equation] In general, solving this optimization problem is hard. For the case of ﬁnite state space ( S ≤ ∞ ), a recent work of Zhang et al.
(2021a) relaxed the problem to the following linear programming problem:
[Equation]
where Φ ( s, a ) =
(cid:0) φ ( s, a, S 1 ) , · · · , φ (cid:0) s, a, S |S| (cid:1)(cid:1) and f = (cid:0) f ( S 1 ) , · · · , f (cid:0) S |S| (cid:1)(cid:1) (cid:62) .
As discussed in Zhang et al.
(2021a), the sample complexity will be worse by a factor of d if we solve the linear programming problem as an approximation.
For the case where the state apace is inﬁnite, we can use state aggregation methods (Ren & Krogh, 2002; Singh et al., 1995) to reduce the inﬁnite state space to ﬁnite state space and apply the approximation approaches to solve it.
The optimization problem in line 19 of Algorithm 1 is to ﬁnd parameter ˜ θ h satisfying several constraints.
For the case where the state space is ﬁnite, we can solve this problem in the following way:
[Equation] [Equation] The above problem can be regarded as a quadratic programming problem and can be solved efﬁciently by the standard optimization methods. By Lemma 4, we know that the true parameter θ h satisﬁes (cid:107) ˆ θ K,h − θ h (cid:107) Λ K,h ≤ β with high probability.
Therefore, the solution ˜ θ h satisﬁes the constraint (cid:13)(cid:13)(cid:13) ˜ θ h − ˆ θ K,h (cid:13)(cid:13)(cid:13) Λ K,h ≤ β with high probability.
For the case where the state space is inﬁnite, we can also solve the above problem using state aggregation methods.
In particular, if the linear mixture MDP model can be regarded as a linear combination of several base MDP models (i.e. φ ( s, a, s (cid:48) )
= [Equation] | s, a )) (cid:62) where P i ( s (cid:48) | s, a ) is the transition probability of certain MDP model), then we can formulate the optimization problem in the following way: [Equation]
which can also be solved efﬁciently in the case of inﬁnite state space.
4. R EGRET
Theorem 1.
With probability at least 1 − δ , after collecting K = ˜ O (cid:16) d 2 H 4 (cid:15) 2 (cid:17) trajectories , Algorithm 1 returns a transition model ˜ P , then for any given reward in the planning phase, a policy returned by any (cid:15) opt -optimal plug-in solver on ( S, A, ˜ P, R, H, ν ) is O ( (cid:15) + (cid:15) opt ) -optimal for the true
MDP, ( S, A, P, R, H, ν ) .
We also propose a lower bound for reward-free exploration in Appendix C. Our lower bound indicates that Ω( d 2 H 3 /(cid:15) 2 ) episodes are necessary to ﬁnd an (cid:15)
-optimal policy with constant probability.
This lower bound is achieved by connecting the sample complexity lower bound with the regret lower bound of certain constructed learning algorithms in the standard online exploration setting.
Compared with this bound, our result matches the lower bound w.r.t.
the dimension d and the precision (cid:15) except for logarithmic factors.
There is also a recent paper of Wang et al.
(2020) studying reward-free exploration in the setting of linear MDPs.
Their sample complexity is ˜ O
( d 3 H 6 /(cid:15) 2 ) , thus our bound is better than theirs by a factor of H 2 .
Though the setting is different, we ﬁnd that our parameter choice and the more reﬁned analysis is applicable to their setting, which can help to further improve their bound by a factor of H 2 .
Please see Appendix D for the detailed discussion.
[-=
-]
[-=
-]
I MPROVING THE DEPENDENCE ON H
In this section, we close the gap on H with a maximum total-variance bound using Bernstein inequality.
In the previous results studying regert minimization in online RL setting (Azar et al., 2017; Zhou et al., 2020a; Wu et al., 2021b), one commonly-used approach to obtain the minimax rate is to upper bound the regret using the total variance of the value function by the Bernstein’s concentration inequalities, and ﬁnally bound the summation of the one-step transition variance by the law of total variance (Lattimore & Hutter, 2012; Azar et al., 2013; 2017).
However, the situation becomes much more complicated in the reward-free setting with linear function approximation.
Recall that ˜ V k,h +1 ,s,a ( s (cid:48) ) deﬁned in Line 9 of Algorithm 1 is the next-step value function that maximizes the uncertainty (cid:107) φ V ( s, a ) (cid:107) Λ − 1 k,h for state-action pair ( s, a ) .
One naive approach is to still use max V (cid:107) φ V ( s, a ) (cid:107) Λ − 1 k,h as the uncertainty measure for ( s, a ) and upper bound the error rate by the summation of the one-step transition variance of ˜ V k,h +1 ,s k,h ,a k,h ( s (cid:48) ) .
However, we can not upper bound the variance summation of ˜ V k,h +1 ,s k,h ,a k,h ( s (cid:48) ) in H steps by O ( H 2 ) similarly by the law of total variance, since { ˜ V k,h,s k,h ,a k,h } h ∈ [ H ] is not the value functions induced from the same policy and transition dynamics.
To tackle the above problem, we need to deﬁne the exploration-driven reward and the conﬁdence bonus for each state-action pair in a more reﬁned way.
Our basic idea is to carefully measure the expected total uncertainty along the whole trajectories w.r.t.
each MDPs in the conﬁdence set.
We will explain the detail in the following subsections.
5. A LGORITHM
Input: Failure probability δ > 0 and target accuracy (cid:15) >λ ← B −
, ˆ β ← 16 (cid:112) d log(1 + KH 2 / ( dλ ))
log(32 K 2 H/δ ) + √ λB ˇ β ← 16 d (cid:112) log(1 + KH 2 / ( dλ ))
log(32 K 2 H/δ ) + √ λB ˜ β ← 16 H 2 (cid:112)
d log(1 + KH 4 / ( dλ ))
log(32 K 2 H/δ ) + √ λB 5: Set Λ i,k,h ← λI , ˆ θ
i,k,h ← 0 for k = 1 , h ∈
[ H ] , i = 1 , 2 , 3 , 4 ,
Set U 1 ,h to be the set containing all the ˜ θ h that makes ˜ P
h well-deﬁned, h ∈
[ H ] .
for episode k = 1 , 2 , · · · , K do
Calculate π k , ˜ θ
k , R k = arg max π, ˜ θ h ∈U k,h ,R V π, ˜ P k, 1 ( s 1 , R ) , where V is deﬁned in Eqn 7.
for step h = 1 , 2 , · · · , H do 10:
Take action according to the policy π k,h and observe s k,h +1 ∼ P h ( ·| s k,h , a k,h ) end for for step h = 1 , 2 , · · · , H do
Update { Λ i,k +1 ,h } 5 i =1 using Eqn 67, 68, 71, 72 and
Update the model estimation { ˆ θ i,k +1 ,h } 5 i =1 using Eqn 69, 70, 73, 74 and15:
Add the constraints (Eqn 3) to the conﬁdence set U k,h , and obtain U k
+1 ,h end for end for
Output:
[Equation]
Our algorithm is described in Algorithm 2.
At a high level, Algorithm 2 maintains a high-conﬁdence set U k,h for the real parameter θ h in each episode k , and calculates an optimistic value function
V π, ˜ P k, 1 ( s 1 , R ) for any reward function R and transition ˜ P h
=
˜
θ (cid:62) h φ with ˜ θ h ∈
U k,h .
Roughly speaking, the value function V π, ˜ P k, 1 ( s 1 , R ) measures the expected uncertainty along the whole trajectory induced by policy π in the MDP with transition ˜ P and reward R .
To collect more “informative” samples and minimize the worst-case uncertainty over all possible transition dynamics and reward function, we calculate π k = arg max π max ˜ θ h ∈U k,h ,R V π, ˜ P k, 1 ( s 1 , R ) , and execute the policy π k to collect more data in episode k .
To ensure that the model estimation ˜ θ k +1 ,h is close to the true model θ h w.r.t.
features φ V ( s, a ) of different value functions V , we use the samples collected so far to calculate ﬁve model estimation { ˆ θ i,k,h } 5 i =1 and the corresponding constraints at the end of the episode k .
Each constraint is an ellipsoid in the parameter space centered at the parameter estimation ˆ θ i,k,h with covariance matrix Λ i,k,h and radius β i , i.e.
[Equation]
i
[Equation]
We update U k
+1 ,h by adding these constraints to the the conﬁdence set U k,h .
The remaining problems are how to deﬁne the value function V π, ˜ P k, 1 ( s 1 , R ) that represents the expected uncertainty for policy π , transition ˜ P and reward R , and how to update the model estimation θ i,k,h and the conﬁdence set U k,h .
Uncertainty Measure Instead of using max V (cid:107) φ V ( s, a ) (cid:107) Λ − 1 k,h to measure the maximum uncertainty for the state-action pair ( s, a ) in Algorithm 1, we separately deﬁne the uncertainty along the trajectories induced by different policy π , reward function R and transition dynamics ˜
P h =
˜ θ (cid:62) h φ .
Speciﬁcally, Recall that ˆ V π, ˜
P h ( s, R ) is the value function of policy π in the MDP model with transition ˜ P and reward R .
We deﬁne the following exploration-driven reward for transition ˜ P , reward R and policy π :
[Equation]
[Equation]
Suppose ˜ V π, ˜ P k,H +1 ( s, R )
= 0 , ∀ s ∈ S , we deﬁne the corresponding value function recursively from step H + 1 to step 1 .
That is,
[Equation]
˜ V π, ˜ P k,h ( s, R ) can be regarded as the expected uncertainty along the trajectories induced by policy π for the value function ˆ V π, ˜
P h ( s, R ) .
In each episode k , we wish to collect samples by executing the policy π that maximizes the uncertainty measure max ˜ θ ∈U k,h ,R ˜ V π, ˜ P k, 1 ( s, R ) .
However, the deﬁnition of ˜ V π, ˜
P k, 1 ( s, R ) explicitly depends on the real transition dynamics P , which is unknown to the agent.
To solve this problem, we construct an optimistic estimation of ˜ V π, ˜ P k,h ( s, R ) .
We deﬁne V π, ˜ P k,h ( s, R ) from step H + 1 to step 1 recursively.
Suppose V π, ˜
P k,H +1 ( s, R ) = 0 .
We calculate
[Equation]
where u π, ˜ P 2 ,k,h is the conﬁdence bonus which ensures that the optimism V π, ˜ P k,h ( s, R ) ≥ ˜ V π, ˜ P k,h ( s, R ) holds with high probability.
After calculating V π, ˜ P k,h ( s, R ) , we take maximization over all ˜ θ h ∈
U k,h and reward function R , and calculate π k = arg max π
[Equation]
We execute π k to collect more samples in episode k .
Weighted Ridge Regression and Conﬁdence Set
In Algorithm 2, we maintain ﬁve model estimation { ˆ θ
i,k,h } 5 i =1 and add ﬁve constraints to the conﬁdence set U k,h in each episode.
These constraints can be roughly classiﬁed into three categories.
The ﬁrst and the third constraints are applied to ensure
that ˜ P k,h ˆ V π, ˜ P
h +1 ( s, a, R ) is an accurate estimation of P k,h ˆ V π, ˜
P h +
( s, a, R ) for any reward function
R and ˜ P k,h = ˜
θ (cid:62) k,h φ satisfying ˜ θ k,
h ∈
U k,h , while the second and the forth constraints are used to guarantee that ˜ P k,h V π, ˜
P h +
( s, a, R ) is an accurate estimation of P h V π,
˜ P
h +1 ( s, a, R ) for any reward function R and ˜ P k,h =
˜
θ (cid:62) k,h φ satisfying ˜ θ k,
h ∈
U k,h .
The last constraint is applied due to technical issue and will be explained in Appendix B.2.
In this subsection, we introduce the basic idea behind the construction of the ﬁrst and the third constraints.
The second and the forth constraints follow the same idea but consider the different value function V π, ˜
P h +
( s, a, R ) .
The formal deﬁnition of the parameters { ˆ θ i,k,h } 5 i =1 and the constraints are deferred to Appendix B.2.
For notation convenience, we use V k,h ( s ) as a shorthand of ˆ V π k , ˜ P k k,h ( s, R k ) in this part.
The construction of our conﬁdence sets is inspired by a recent algorithm called UCRL-VTR + proposed by Zhou et al.
(2020a).
Recall that we use ( s k,h , a k,h ) to denote the state-action pair that the agent encounters at step h in episode k .
We use the following ridge regression estimator to calculate the corresponding ˆ θ 1 ,k,h in episode k :
[Equation]
where ¯ σ 21 ,k,h = max (cid:8) H 2 /d, [¯ V k,h V k,h +1 ]( s k,h , a k,h ) + E 1 ,k,h (cid:9) is an optimistic estimation of the one-step transition variance:
[Equation]
In the deﬁnition of ¯ σ 21 ,k,h , ¯ V k,h V k,h +1 ( s k,h , a k,h ) is an empirical estimation for the variance V
h V k,h +1 ( s k,h , a k,h ) , and E 1 ,k,h is a bonus term deﬁned in Eqn.
61 which ensures that ¯ σ k,h is an optimistic estimation of V h V k,h +1 ( s k,h , a k,h ) .
One technical issue here is how to estimate the variance V
h V k,h +1 ( s k,h , a k,h ) .
Recall that by deﬁnition,
[Equation]
We use
[Equation]
as our variance estimator ¯ V k,h V k,h +1 ( s k,h , a k,h ) , where ˜ θ k,h ∈
U k,h is the parameter which maximizes the value
V π, ˜ P k, 1 ( s 1 , R ) in episode k .
To ensure that ¯ V k,h V k,h +1 ( s k,h , a k,h ) is an accurate estimator, we maintain another parameter estimation ˆ θ 3 ,k,h using history samples w.r.t.
the feature φ V 2 k,h
+
( s k,h , a k,h ) .
[Equation]
After calculating ˆ θ 1 ,k,h and ˆ θ 3 ,k,h , we add the ﬁrst and the third constraints (Eqn. 3) to the conﬁdence set U k,h , where Λ 1 ,k,h and Λ 3 ,k,h is the corresponding covariance matrix of all history samples, i.e. Λ 1 ,k,h
= (cid:80) k − 1 t =1 ¯ σ − 2 1 ,t,h φ V t,h +1 ( s t,h , a t,h ) φ (cid:62) V t,h +1 ( s t,h , a t,h ) and Λ 3 ,k,h = (cid:80) k − 1 t =1 φ V 2 t,h +1 ( s t,h , a t,h ) φ (cid:62) V 2 t,h +1 ( s t,h , a t,h ) .
5. R EGRET
We present the regret upper bound of Algorithm 2 in Theorem 2.
In the regime where d ≥ H and (cid:15) ≤ H/ √ d , we can obtain ˜ O
( d 2 H 3 /(cid:15) 2 ) sample complexity upper bound, which matches the sample complexity lower bound except logarithmic factors.
Theorem 2.
With probability at least 1 − δ , after collecting K = ˜
O
[Equation]
trajectories, Algorithm 2 returns a transition model ˜ P K , then for any given reward in the planning phase, a policy returned by any (cid:15) opt -optimal plug-in solver on ( S, A, ˜ P K , R, H, ν ) is O ( (cid:15) + (cid:15) opt ) - optimal for the true MDP, ( S, A, P, R, H, ν ) .
C ONCLUSION
This paper studies the sample complexity of plug-in solver approach for reward-free reinforcement learning.
We propose a statistically efﬁcient algorithm with sample complexity ˜ O (cid:0) d 2 H 4 /(cid:15) 2 (cid:1) .
We further reﬁne the complexity by providing an another algorithm with sample complexity ˜
O (cid:0) d 2 H 3 /(cid:15) 2 (cid:1) in certain parameter regimes.
To the best of our knowledge, this is the ﬁrst minimax sample complexity bound for reward-free exploration with linear function approximation.
As a side note, our approaches provide an efﬁcient learning method for the RL model representation, which preserves values and policies for any other down-stream tasks (speciﬁed by different rewards).
Our sample complexity bound matches the lower bound only when d ≥ H and (cid:15) ≤ H/ √ d .
It is unclear whether minimax rate could be obtained in a more broader parameter regimes.
We plan to address this issue in the future work.
A CKNOWLEDGMENTS
Liwei Wang was supported by National Key R&D Program of China (2018YFB1402600), Exploratory Research Project of Zhejiang Lab (No. 2022RC0AN02), BJNSF (L172037), Project 2020BD006 supported by PKUBaidu Fund.
A O MITTED D ETAILS IN S ECTIONA. N OTATIONS
In this subsection, we summarize the notations used in Section A.
Symbol
Explanation
[Table]
The high-probability event for Theorem
[Table]
The state and action that the agent encounters in episode k and step h
[Table]
The value function with maximum uncertainty: arg max
[Table]
The estimation of
[Table]
in episode
[Table]
The covariance matrix in
[Table]
The uncertainty measure:
[Table]
R
[Table]
The exploration-driven reward which equals u k,h ( s, a )
Q k,h ( s, a )
The Q function deﬁned in line 12 of Algorithm
V k, [Table] ( s )
The value function deﬁned in line 13 of Algorithm 1 ˜
V [Table] s, R [Table]
The value function deﬁned in Eqn 12 ˜
Q [Table] s, R )
The Q value similarly deﬁned as ˜ V ∗ h ( s, R )
˜
V πh ( s, R )
The value function of policy π similarly deﬁned as ˜ V ∗ h ( s, R )
˜
θ h
The parameter estimation returned at the end of the exploration phase ˆ
[Equation]
Now we brieﬂy explain the main idea in the proof. Firstly, we introduce the value function ˜ V ∗ h ( s, R ) , which is recursively deﬁned from step H + 1 to step 1 :
[Equation]
[Equation]
Compared with the deﬁnition of V ∗ h ( s, R ) , the main difference is that we take minimization over the value and H at each step. We state the following lemma, which gives an upper bound on the sub-optimality gap of ˆ π in the planning phase.
Lemma 1.
With probability at least 1 − δ , the sub-optimality gap of the policy ˆ π R for any reward function R in the planning phase satisﬁes V ∗ 1 ( s 1 , R ) − V ˆ π R 1 ( s 1 , R ) ≤ 4 ˜ V ∗ 1 ( s 1 , R K ) + (cid:15) opt , where
R K is the exploration-driven reward used for episode K in the exploration phase. This lemma connects the sub-optimality gap with the value function of the auxiliary reward in episode K .
So the remaining problem is how to upper bound ˜ V ∗ 1 ( s 1 , R K ) . Since the explorationdriven reward R k is non-increasing w.r.t. k , it is not hard to prove that K ˜ V ∗ 1 ( s 1 , R K ) ≤
[Equation]
Kk = [Equation]
V ∗ 1 ( s 1 , R k ) . We use the {+following two lemmas to upper bound (cid:80) Kk =
+}
˜ V ∗ 1 ( s 1 , R k ) .
Lemma 2.
With probability at least 1 − δ , ˜ V ∗ h ( s, R k ) ≤ V k,h ( s ) holds for any ( s, a ) ∈ S × A , h ∈ [ H ] and k ∈
[ K ] .
Lemma 3. With probability at least 1 − δ ,
[Equation]
With the help of the optimistic bonus term u k,h , we can prove that the estimation value V k,h is always optimistic w.r.t ˜ V ∗ h , which is illustrated in Lemma 2. Therefore, we have (cid:80) Kk
{+=
 ˜+}
V ∗ 1 ( s 1 , R k ) ≤ (cid:80) Kk =1 V k, 1 ( s 1 ) . By adapting the regret analysis in the standard RL setting to the reward-free setting, we can upper bound the summation of V k, 1 ( s 1 ) in Lemma 3. Combining the above two lemmas, we derive the upper bound of ˜ V ∗ 1 ( s 1 , R K ) , then we bound the sub-optimality gap of ˆ π by Lemma 1.
A. H IGH - PROBABILITY E VENTS
We ﬁrstly state the following high-probability events.
Lemma 4.
With probability at least 1 − δ/ 2 , the following inequality holds for any k ∈ [ K ] , h ∈
[ H ] :
[Equation]
Proof.
For some step h ∈
[ H ] , the agent selects an action with feature x k,h = φ k,h ( s k,h , a k,h ) .
The noise η k,h = ˜
V k,h,s k,h ,a k,h ( s (cid:48) ) − θ (cid:62) h φ k,h ( s k,h , a k,h ) satisﬁes H -sub-Gaussian.
By Lemma 20, the following inequality holds with probability 1 − δ 2 H :
[Equation]
By taking union bound over all h ∈
[ H ] , we can prove the lemma.
Lemma 5.
With probability at least 1 − δ/ 2 , we have
[Equation]
Proof.
This lemma follows directly by Azuma’s inequality.
During the following analysis, we denote the high-probability events deﬁned in Lemma 4 and
Lemma 5 as E 1 .
A. P ROOF OF L EMMA
Lemma 6.
(Restatement of Lemma 1)
Under event E 1 , the sub-optimality gap of the policy ˆ π R for any reward function R in the planning phase can be bounded by
[Equation]
Proof.
[Equation]
The inequality is because that the policy ˆ π R is the (cid:15) opt -optimal policy in the estimated MDP ˆ M , i.e.
ˆ V π
[Equation]
R ,
[Equation]
For the notation convenience, for a certain sequence { R h } Hh =1 , we deﬁne the function W h ( { R h } ) recursively from step H +1 to step 1 .
Firstly, we deﬁne W H +
( { R h } )
= 0 .
W h ( { R h } ) is calculated recursively from W h +1 ( { R h } ) :
[Equation]
[Equation]
Similarly with the deﬁnition of ˜ V ∗ h ( s, R ) , we introduce the value function ˜ V πh ( s, R ) , which is recursively deﬁned from step H + 1 to step 1 :
[Equation]
[Equation]
We use traj ∼ ( π, P ) to indicate that the trajectory { s h , a h } Hh =1 is sampled from transition P with policy π .
For any policy π , we have
[Equation]
The second inequality is due to lemma 4 and the deﬁnition of ˜ θ .
Plugging this inequality back to
Inq.
18, we can prove the lemma.
A. P ROOF OF L EMMA
Lemma 7.
(Restatement of Lemma 2)
Under event E 1 , ˜ V ∗ h ( s, R k ) ≤ V k,h ( s ) holds for any ( s, a ) ∈
S × A , h ∈
[ H ] and k ∈
[ K ] . .
Proof.
We prove the lemma by induction.
Suppose ˜ V ∗ h
+1 ( s, R k ) ≤ V k,h +1 ( s ) ,
[Equation]
∗
[Equation]
( s
[Equation]
a
[Equation]
R k
[Equation]
− Q k,h ( s
[Equation]
a
[Equation] [Equation]
a
[Equation] [Equation]
[Equation]
[Equation]
[Equation] (37) =0 .
The ﬁrst inequality is due to induction condition ˜
V ∗ h
+1 ( s, R k ) ≤ V k,h +1 ( s ) .
The last inequality is due to Lemma 4.
Since ˜ Q ∗ h ( s, a ) ≤ Q k,h ( s, a, R k ) for any a ∈ A , we have ˜
V ∗ h ( s, R k ) ≤
V k,h ( s ) .
A. P ROOF OF L EMMA
Lemma 8.
(Restatement of
Lemma 3)
Under event
[Equation]
Proof.
[Equation]
The last inequality is from
[Equation]
From Inq 39, we have
[Equation]
For the ﬁrst term, by Lemma 21, we have for any h ∈
[ H ] ,
K (cid:88)
[Equation]
Since λ = B − 2 , we have
[Equation]
For the second term, by Lemma 5,
[Equation]
[Equation]
Therefore, we have
[Equation] [Equation]
Lemma 9.
Under event E 1 , ˜ V
[Equation]
[Equation]
s 1 , R
[Equation]
Proof.
Since ˜ V ∗ 1 ( s k, 1 , R k ) ≤ V k, 1 ( s k, 1 ) by lemma 7, we have
[Equation]
[Equation]
By the deﬁnition of R k , we know that R k ( s, a ) is non-increasing w.r.t k .
Therefore,
K
[Equation]
The lemma is proved by dividing both sides by K .
A. P ROOF OF T HEOREM
Combining the results in Lemma 9 and Lemma 6, we know that for any reward function R ,
[Equation]
Choosing K =
C 1 H 4 d 2 log(4 H 3 KB 2 /
[Equation]
)log
[Equation]
1+ KH 2 B 2 /d )
(cid:15)for some constant C 1 sufﬁces to guarantee that V ∗ 1 ( s 1 , R ) − V
ˆ π R 1 ( s 1 , R ) ≤ (cid:15) + (cid:15) opt .
B O MITTED D ETAILS IN S ECTIONB. N OTATIONS
In this subsection, we summarize the notations used in Section B.
E The high-probability event for Theorems k,h , a k,h The state and action that the agent encounters in episode k and step h φ V ( s, a, R ) (cid:80) s (cid:48)
φ ( s, a, s (cid:48) ) V π,P ( s (cid:48) , R ) for certain value function V ˆ V π, ˜
P h ( s, R )
The value function of policy π in the MDP model with transition ˜ P and reward R ˜ V π, ˜ P k,h ( s, R )
The expected uncertainty along the trajectories induced by policy π for ˆ V π, ˜ P h ( s, R ) (Deﬁned in Eqn 5)
V π, ˜ P k,h ( s, R )
The optimistic estimation of ˜ V π, ˜ P k,h ( s, R ) (Deﬁned in Eqn 7) u π, ˜ P 1 ,k,h ( s, a, R )
The exploration-driven reward for transition ˜ P , reward R and policy π (Deﬁned in Eqn 4)
u ˜ P,π 2 ,k,h ( s, a, R )
The conﬁdence bonus ensuring the optimism V π, ˜ P k,h ( s, R ) ≥ ˜ V π, ˜ P k,h ( s, R ) (Deﬁned in Eqn 6)
V h V ( s, a )
The one-step transition variance w.r.t.
certain value function V ¯ V 1 ,k,h ( s, a )
The empirical variance estimation w.r.t.
the value ˆ V π k , ˜ P k h ( s, R k ) (Deﬁned in Eqn 59) ¯ V 2 ,k,h ( s, a )
The empirical variance estimation w.r.t.
the value V π k , ˜ P k h ( s, R k ) (Deﬁned in Eqn 60)
E 1 ,k,h , E 2 ,k,h
The conﬁdence bonus for the variance estimation ¯ V 1 ,k,h ( s, a ) and ¯ V 2 ,k,h ( s, a ) , respectively ¯ σ 21 ,k,h , ¯ σ 22 ,k,h
The optimistic variance estimation for ¯ V 1 ,k,h ( s, a ) and ¯ V 2 ,k,h ( s, a ) , respectively ˆ
[Equation]
h
The parameter estimation w.r.t.
certain value function (Deﬁned in Section B.2) Λ [Equation]
The empirical covariance matrix w.r.t.
certain value function (Deﬁned in Section B.2)
U [Equation]
The conﬁdence set containing θ h with high probability
[Equation]
˜
[Equation]
The “value function” for the MDP with transition ˜ P k and reward ¯ V 1 ,k,h τ
[Equation]
B. O MITTED D ETAILS OF A LGORITHM
In algorithm 2, we maintain ﬁve different parameter estimation ˆ θ 1 ,k,h , ˆ θ 2 ,k,h , ˆ θ 3 ,k,h , ˆ θ 4 ,k,h and ˆ θ 5 ,k,h , and the corresponding covariance matrix Λ 1 ,k,h , Λ 2 ,k,h , Λ 3 ,k,h , Λ 4 ,k,h and Λ 5 ,k,h .
ˆ θ 1 ,k,h , ˆ θ 2 ,k,h are the parameter estimation using history samples w.r.t the features of variancenormalized value function
ˆ V π k , ˜
P
k k,h +1 / ¯ σ 1 ,k,h and V π k , ˜
P k k,h +1 / ¯ σ 2 ,k,h , respectively.
ˆ θ 3 ,k,h , ˆ θ 4 ,k,h are the parameter estimation using samples w.r.t the features of value function (cid:16)
ˆ V π k , ˜ P k k,h +1 (cid:17) 2 and (cid:16) V π k , ˜ P k k,h +1 (cid:17) 2 .
ˆ θ 5 ,k,h is somewhat technical and will be explained later.
for any i ∈ [5] and h ∈
[ H ] , ˆ θ
i, 1 ,h are initialized as 0 , and Λ i, 1 ,h are initialized as λI in Algorithm 2.
U 1 ,h is the set containing all the ˜ θ h that makes ˜ P
h well-deﬁned, i.e. (cid:80) s (cid:48)
˜ θ (cid:62) h φ ( s (cid:48) | s, a ) = 1 and ˜ θ (cid:62) h φ ( s (cid:48) | s, a ) ≥ 0 , ∀ s, a .
After observing { s k,h , a k,h } Hh =1 in episode k , we calculate ¯ V 1 ,k,h
( s, a ) and ¯ V 2 ,k,h ( s, a ) as the corresponding variance in the empirical MDP with transition dynamics ˜
P k .
[Equation]
To guarantee the variance estimation is optimistic, we calculate the conﬁdence bonus E 1 ,k,h and
E 2 ,k,h for variance estimation ¯ V 1 ,k,h
( s k,h , a k,h ) and ¯ V 2 ,k,h ( s k,h , a k,h ) :
E 1 ,k,h = min
[Equation]
[Equation]
[Equation]
We add the bonuses to ¯ V 1 ,k,h ( s k,h , a k,h ) and ¯ V 2 ,k,h ( s k,h , a k,h ) and maintain the optimistic variance estimation ¯ σ 21 ,k,h and ¯ σ 22 ,k,h :
[Equation]
[Equation]
We use ¯ σ i,k,h to normalize the value obtained in each step.
By setting λ = B − 2 and solving the ridge regression deﬁned in Eqn. 8, we know that ˆ θ
i,k +1 ,h and Λ i,k +1 ,h are updated in the following
[Table]
[Equation]
Similarly, we update the estimation w.r.t (cid:16) ˆ V π k , ˜ P k k,h +1 ( s (cid:48) , R k ) (cid:17) 2 and (cid:16) V π k , ˜ P k k,h +1 ( s (cid:48) , R k ) (cid:17) 2 : Λ 3 ,k +1 ,h = Λ [-
-]3 ,k,h + (cid:32)(cid:88)
[Equation]
(cid:33)
[Equation]
71) Λ
[Equation]
k
[Equation]
+
[Equation]
ˆ θ 3 ,k +1 ,h
= (Λ 3 ,k +1 ,h ) −
[Equation]
(73) ˆ θ 4 ,k +1 ,h
= (Λ 4 ,k +1 ,h ) −k (cid:88)
[Equation]
We deﬁne ˜
Y k,h to be the “value function” with transition ˜ P k and reward ¯ V 1 ,k,h .
˜
[Equation]
k
[Equation]
[Equation]
(76) ˆ θ 5 ,k,h is the parameter estimation using samples w.r.t.
˜
Y k,h ( s ) :
[Equation]
We update the high-conﬁdence set U k,h by adding following ﬁve constraints:
[Equation]
Note that we add new constraints to this conﬁdence set U k,h in each episode, instead of updating U k,h as the intersection of the new constraints.
This operation is designed to ensure that the cardinality of the conﬁdence set U k,h is non-increasing w.r.t.
the episode k , so that V π, ˜ P k, 1 ( s 1 , R ) is always non-increasing w.r.t.
k .
During the proof, we use φ V π,P ( s, a, R ) as a shorthand of (cid:80) s (cid:48) φ ( s, a, s (cid:48) )
V π,P ( s (cid:48) , R ) .
B. H IGH - PROBABILITY EVENTS
For notation convenience, we use V 1 ,k,h ( s, a ) and V 2 ,k,h
( s, a ) to denote the one-step transition variance with regard to ˆ V π k ,P k k,
h +
and V π k ,P k k,h +1 , i.e. V 1 ,k,h ( s, a [Equation] =
[Equation]
s
[Equation]
,a )
[Equation]
Lemma 10.
With probability at least 1 − δ [Equation] the following event holds for any h ∈
[ H ] [Equation] k ∈ [ K ] :
[Equation]
Proof.
We ﬁrstly prove the ﬁrst and the third inequality, the second and the fourth inequality can be proved following the same idea.
By the deﬁnition of ¯ V 1 ,k,h ( s k,h , a k,h ) and V 1 ,k,h ( s k,h , a k,h ) :
[Equation]
Let x k =
(¯ σ 1 ,k,h ) − 1 φ ˆ V πk,Pkk,h +1 ( s k,h , a k,h ) , and the noise η k = (¯ σ 1 ,k,h ) − 1 ˆ V π k ,P k k,h +1 ( s k,h
+1 , R k ) − (¯ σ 1 ,k,h ) −
[Equation]
Since ¯ σ 1 ,k,h ≥ (cid:112) H 2 /d , we have (cid:107) x k (cid:107) 2 ≤ √ d , E (cid:2)
η 2 k | G k (cid:3) ≤ d .
By Lemma 22, we have with prob at least 1 − δ/ (8 H ) , for all k ≤ K ,
[Equation]
Similarly, we can prove that
[Equation]
Since (cid:13)(cid:13)(cid:13) ˜ θ k,h − ˆ θ 1 ,k,h (cid:13)(cid:13)(cid:13) Λ 1 ,k,h ≤ ˆ β ≤ ˇ β by Inq 79, we have (cid:13)(cid:13)(cid:13) θ h − ˜ θ k,h (cid:13)(cid:13)(cid:13) Λ 1 ,k,h ≤ 2ˇ β .
Similarly, since (cid:13)(cid:13)(cid:13) ˜ θ k,h − ˆ θ 3 ,k,h (cid:13)(cid:13)(cid:13) Λ 3 ,k,h ≤ ˜ β by Inq 81, we have (cid:13)(cid:13)(cid:13) θ
[Equation]
θ
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
3 ,k,h ≤ 2˜ β . Plugging the above inequalities back to Inq.
92, we have
[Equation]
The above inequality indicates that ¯ σ 1 ,k,h ≥ V 1 ,k,h , which is an optimistic estimation.
As a result, the variance of the single-step noise η k = (¯ σ 1 ,k,h ) − 1 ˆ V π k ,P k k,
h +1 ( s k,h
+1 , R k ) − (¯ σ 1 ,k,h ) −
[Equation]
satisﬁes E (cid:2)
η 2 k | G k (cid:3) ≤ 1 .
By Lemma 22, we can prove a tighter conﬁdence guarantee for ˆ θ 1 ,k,h :
[Equation]
[Equation]
Combining with (cid:13)(cid:13)(cid:13) ˜ θ 1 ,k,h − ˆ θ 1 ,k,h (cid:13)(cid:13)(cid:13) Λ 1 ,k,h ≤ ˆ β by Inq 79, we have (cid:13)(cid:13)(cid:13) θ h − ˜ θ 1 ,k,h (cid:13)(cid:13)(cid:13) Λ 1 ,k,h ≤ 2ˆ β .
Now we prove the last inequality in this lemma.
Recall that we deﬁne
[Equation]
Note that ˜ Y k,h ( s ) ≤ H 2 by the law of total variance (Lattimore & Hutter, 2012; Azar et al., 2013).
Therefore, the variance on the single-step noise in at most H 2 .
By Lemma 22, we have
[Equation]
Λ
[Equation]
[Equation]
[Equation]
Combining with (cid:13)(cid:13)(cid:13)
˜ θ h − ˆ θ 5 ,k,h (cid:13)(cid:13)(cid:13) Λ 5 ,k,h ≤ ˜ β by Inq 83, we have (cid:13)(cid:13)(cid:13) θ h − ˜ θ 5 ,k,h (cid:13)(cid:13)(cid:13) Λ 5 ,k,h ≤ 2˜ β.
Lemma 11.
With probability at least 1 − δ/ 2 , we have
[Equation]
Proof.
This lemma follows directly by Azuma’s inequality for martingale difference sequence and union bound.
During the following analysis, we denote the high-probability events deﬁned in Lemma 10 and
Lemma 11 as E 2 .
B. P ROOF OF T HEOREM
Lemma 12.
Under event E 2 , we have K
[Equation]
2 ( KH 4 B 2 ) (cid:19) .
This lemma can be proved with the technique for regret analysis.
We will explain it in Appendix B.5.
Lemma 13.
(Optimism)
Under event E 2 , for any h ∈ [ H ] , k ∈
[ K ] , s ∈ S , π ∈ Π , R and ˜ P satisfying ˜ P h
=
˜
θ (cid:62) h φ , we have
[Equation]
Proof.
This lemma is proved by induction.
Suppose V π, ˜ P k,h +1 ( s, R ) ≥ ˜ V π, ˜ P k,h +1 ( s, R ) , we have [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation] [Table] [Equation] ≥ 0 .
[Equation]
This indicates that V π, ˜ P k,h ( s, R ) ≥ ˜ V π, ˜ P k,h ( s, R ) holds for step h .
Lemma 14.
under event E 2 , we have V π K , ˜ P K K,
[Equation]
Proof.
Firstly, we prove that V π, ˜ P k, 1 ( s, R ) is non-increasing w.r.t k for any ﬁxed π, ˜ P, s and R .
This can be proved by induction.
Suppose for any k 1 ≤ k 2 , V π, ˜
P
k 1 ,h +1 ( s, R ) ≥ V π, ˜ P
k 2 ,h +1 ( s, R ) for any s .
Recall that
[Equation]
Since Λ 1 ,k 1 ,h (cid:52) Λ 1 ,k 2 ,h , Λ 2 ,k 1 ,h (cid:52) Λ 2 ,k 2 ,h and V π,
˜ P k 1 ,h +1 ( s, R ) ≥ V π, ˜
P
k 2 ,h +1 ( s, R ) for any s , we can prove that
[Equation]
[Equation]
Therefore, V π, ˜ P k 1 ,h ( s, R ) ≥ V π, ˜
P k 2 ,h ( s, R ) holds for step h and any s .
Since the cardinality of the transition set U k,h is non-increasing (We add more constraints in each episode), we know that ˜ θ k 2 ,h ∈ U
k 1 ,h .
By the optimality of π k 1 , ˜ θ k 1 and R k 1 in episode k 1 , we have V
[Equation]
Combining the above two inequalities, we have
[Equation]
This indicates that the value V π k , ˜ P k k,( s 1 , R k ) is non-increasing w.r.t.
k .
Therefore,
[Equation]
Lemma 15.
Under event E 2 , the sub-optimality gap of the policy ˆ π R returned in the planning phase for any reward function R can be bounded by
[Equation]
Proof.
Recall that we use ˆ V πh to denote the value function of policy π on the estimated model ˜ θ K .
[Equation]
The inequality is because that the policy ˆ π R is the (cid:15) opt -optimal policy in the estimated MDP ˆ M .
For notation convenience, we use traj ∼ ( π, P ) to denote that the trajectory ( { s h , a h } Hh =1 ) is sampled with transition P and policy π .
For a certain sequence { R h } Hh =1 , we deﬁne the function
W h ( { R h } ) recursively from step H + 1 to step 1 .
Firstly, we deﬁne W H +
( { R h } )
= 0 .
W h
( R ) is calculated from W h
+
( R ) :
[Equation] = min { H, R h + W h
[-+
-][Equation]
( { R h }
[Equation]
[Equation]
With the above notation
[Equation]
we can prove that for any policy π ∈ Π ,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Equation]
The second inequality is due to lemma 10 .
The third inequality is due to Lemma 13.
The last inequality is due to the optimality of π K and R K .
Plugging this inequality back to Inq.
119, we can prove the lemma
[Equation]
Proof.
(Proof of Theorem 2)
Combining Lemma 12, Lemma 14 and Lemma 15, we know that
[Equation]
for some constant C 2 sufﬁces to guarantee that V ∗ 1 ( s 1 , R
[Equation]
(cid:15) + (cid:15) opt .
B. P ROOF OF L EMMA
Proof.
From the standard value decomposition technique (Azar et al., 2017; Jin et al., 2018)
[Equation]
we can decompose V π k , ˜ P k k,( s 1 , R k ) into following terms:
[Equation]
where the second inequality is derived from
[Equation]
Eqn 138 is a martingale difference sequence.
By Lemma 11, the summation over all k ∈
[ K ] is at most (cid:112) 2 H 3 K log(4 /δ ) .
We mainly focus on Eqn 136 and Eqn 137.
Upper bound of Eqn 136 Firstly, we bound the summation of min (cid:110) H, u π k , ˜ P k 1 ,k,h
( s k,h , a k,h , R k ) (cid:111) .
[Equation]

[Equation]
[Equation]
The ﬁrst inequality is due to Cauchy-Schwarz inequality.
The second inequality is due to Lemma 21.
By the deﬁnition of ¯ σ 21 ,k,h , we have
[Equation]
Now we bound (cid:80) k,h E 1 ,k,h and (cid:80) k,h ¯ V 1 ,k,h
( s k,h , a k,h ) respectively.
For (cid:80) k,h E 1 ,k,h , we have (cid:88) k,h
E 1 ,k,h ≤ H (cid:88) h = K (cid:88)
[Equation]
=
[Equation]
min
[Equation]

[Equation]
+
[Equation]
h =
[Equation]
For the part,
[Equation]
min
[Equation]
  
[Equation]
where the last inequality is due to Lemma 21.
Similarly, for the second part, by Lemma 21, we have
H
[Equation]
Note that ¯ V 1 ,k,h
( s k,h , a k,h ) is the empirical variance of ˆ V π k ,
˜ P k k,h +1 ( s (cid:48) , R k ) with transition ˜
P k ( s (cid:48) | s k,h , a k,h ) .
We bound the summation of ¯ V 1 ,k,h ( s k,h , a k,h ) by the law of total variance (Lattimore & Hutter, 2012; Azar et al., 2013).
Recall that we deﬁne
[Equation]
[Equation]
By the law of total variance, we have ˜ Y k, 1 ( s ) ≤ H 2 holds for any s, a and k .
We now bound the difference between (cid:80) Kk =
˜ Y k, 1 ( s k, 1 ) and (cid:80)
Kk =1 (cid:80)
Hh =1 ¯ V 1 ,k,h
( s k,h , a k,h ) .
[Equation] [Equation]
≤ min (cid:110) H 2 , (cid:16) ˜ P k
[Equation]
1 − P
[Equation]
Y
[Equation] [Equation]
a
[Equation]
[Equation]
[Equation]
Therefore, we have
[Equation]
For Eqn
[Equation]
this term can be regarded as a martingale difference sequence, thus can be bounded by
H 2 √ KH by Lemma 11.
For Eqn 163, we can bound this term in the following way:
[Equation]
The second inequality is due to Lemma 10.
The third inequality is due to Cauchy-Schwarz inequality.
The last inequality is due to Lemma 21.
From the above analysis [Equation] we have
[Equation]
Therefore, the summation of ¯ σ 21 ,k,h can be bounded as (cid:88)
[Equation]
We deﬁne τ = log(32 K 2 H/δ ) log 2 (1 + KH 4 B 2 ) .
Solving for (cid:80) k,h ¯ σ 21 ,k,h , we have
[Equation]
where c 1 denote a certain constant.
The last inequality is due to 2 √ ab ≤ a + b for a, b ≥ 0 .
Plugging the above inequality back to Inq 142, we have for a constant c 2 ,
[Equation]
Upper bound of Eqn 137 Now we focus on the summation of u π k , ˜ P k 2 ,k,h
( s k,h , a k,h , R k ) .
The proof follows almost the same arguments as the summation of u π k , ˜ P k 1 ,k,h ( s k,h , a k,h , R k ) , though the upper bound of (cid:80)
Kk =1 (cid:80)
Hh =1 ¯ V 2 ,k,h
( s k,h , a k,h ) is derived in a different way.
Following the same proof idea, we can show that
[Equation]
Combining Inq 178 and Inq 179 and solving for (cid:80) k,h ¯ σ 22 ,k,h , we have (cid:88) k,
[Equation]
[Equation] [Equation]
≤
[Equation]
Plugging this inequality back to Inq 177, we have
[Equation]
We now bound the last term in the above equation.
Deﬁne τ = log(32 K 2 H/δ ) log
[Equation]
1 + KH 4 B [Equation] ) , we have:
[Equation]
where ˜ c 3 ≥ 1 is a constant to be deﬁned later.
The last inequality is due to 2 √ ab ≤ a + b for any a, b ≥ 0 .
Note that by Lemma 10, Eqn 192 is upper bounded by
[Equation]
( s k,h , a k,h , R k ) .
Eqn 191 is a martingale difference sequence, and can be bounded by (cid:112) HK log(1 /δ ) with high probability.
Plugging the above inequality back to Inq 183 and solving for (cid:80) Kk = [Equation] =
[Equation] u [Equation] k
[Equation]
˜ P k 2 ,k,h ( s k,h , a k,h , R k ) , we have
[Equation]
where c 3 ≥ 1 is a constant here.
We set ˜ c
3 = c 3 , then we have
[Equation]
Bounding the summation of V π k , ˜ P k k,( s 1 , R k )
In the above analysis, we derive the upper bound of Eqn 136 and Eqn 137 (Inq 176 and Inq 195).
Now we can ﬁnally bound (cid:80)
Kk =1 V π k , ˜ P
k k,( s k, 1 , R k ) .
For a constant c 4 ,
K (cid:88)
[Equation]
Following the same analysis, the above inequality actually also holds for any step
h ∈
[ H ] :
[Equation]
and a
[Equation]
=
[Equation]
The above inequality can be simpliﬁed into the following form:
[Equation]
and we have a H
+1 = 0 .
From the elementary calculation, we can prove that a 1 ≤ (1+ 1 H ) H G ≤ eG .
Therefore [Equation] we have
[Equation]
for a constant c 5 = ec 4 .
C T HE L OWER B OUND FOR R EWARD - FREE E XPLORATION
In this section, we prove that even in the setting of non-plug-in reward-free exploration, the sample complexity in the exploration phase to obtain an (cid:15) -optimal policy is at least Ω( d 2 H 3 (cid:15) 2 ) .
We say an algorithm can ( (cid:15), δ ) -learns the linear mixture MDP M if this algorithm returns an (cid:15) -optimal policy for M with probability at least 1 − δ in the planning phase after receiving samples for K episodes in the exploration phase.
Our theorem is stated as follows.
Theorem 3.
Suppose (cid:15) ≤ min (cid:16) C 1 √ H, C 2 √ dH 4 B (cid:17) , B > 1 , H > 4 , d > 3 for certain positive constants C 1 and C 2 .
Then for any algorithm ALG there exists an linear mixture MDP instance
M = ( S , A , P, R, H, ν ) such that if ALG ( (cid:15), δ ) -learns the problem M , ALG needs to collect at least K =
Cd 2 H 3 /(cid:15) 2 episodes during the exploration phase, where C is an absolute constant, and
0 < δ < 1 is a positive constant that has no dependence on (cid:15), H, d, K .
Compared with the lower bound proposed by Zhang et al.
(2021a), we further improve their result by a factor of d .
This lower bound also indicates that our sample complexity bound in Theorem 2 is statistically optimal.
We prove Theorem 3 in Appendix C.1.
C. P ROOF OF T HEOREM
Our basic idea to prove Theorem 3 is to connect the sample complexity lower bound with the regret lower bound of another constructed learning algorithm in the standard online exploration setting.
To start with, we notice that the reward-free exploration is strictly harder than the standard nonreward-free online exploration setting (e.g. Jin et al.
; Zhou et al.
(2021; 2020a)), where the reward function is deterministic and known during the exploration.
Therefore, if we can prove the sample complexity lower bound in the standard online exploration setting, this bound can also be applied to the reward-free setting.
For readers who are not familiar with the formulation of online exploration setting studied in this section, we ﬁrstly introduce the preliminaries.
Compared with the reward-free exploration, the only difference is that the reward R ( s, a ) is ﬁxed and known to the agent.
In each episode, the agent starts from an initial state s k, 1 sampled from the distribution ν .
At each step h ∈
[ H ] , the agent observes the current state s k,h ∈ S , takes action a k,h ∈ A , receives the deterministic reward R h ( s k,h , a k,h ) , and transits to state s k,
h +1 with probability P h
( s k,h +1 | s k,h , a k,h ) .
The episode ends when s H +1 is reached.
The agent’s goal is to ﬁnd a (cid:15) -optimal policy π after K episodes.
We say a policy π is (cid:15) -optimal if
[Equation]
where π ∗ is the optimal policy for the MDP ( S , A , P, R, H, ν ) .
Theorem 4.
Suppose (cid:15) ≤ min (cid:16) C 1 √ H, C 2 √ dH 4 B (cid:17) , B > 1 , H > 4 , d > 3 .
Then for any algorithm ALG 1 solving the non-reward-free online exploration problem, there exists an linear mixture MDP M =
( S , A , P, R, H, ν ) such that ALG 1 needs to collect at least K
=
Cd 2 H 3 /(cid:15)episodes to output an (cid:15) -optimal policy for the linear mixture MDP M with probability at least 1 − δ , where C is an absolute constant, and 0 < δ < 1 is a positive constant that has no dependence on (cid:15), H, d, K .
From the above discussion, Theorem 3 can be directly proved by reduction if Theorem 4 is true.
Proof.
(Proof of Theorem 3)
The theorem can be proved by contradiction.
Suppose there is an algorithm ALG which can ( (cid:15), δ ) -learns any linear mixture MDP instance M with only K (cid:48)
≤ Cd 2 H 3 /(cid:15)episodes.
Then we can use this algorithm to solve the online exploration problem by simply ignoring the information about the reward function and directly calling the exploration algorithm of ALG during the exploration phase.
Then in the planning phase, we use the planning algorithm of ALG to output a policy based on the reward function as well as samples collected in the exploration phase.
Therefore, this indicates that ALG can output an (cid:15) -optimal policy for the non-reward-free online exploration problem with probability at least 1 − δ after only K (cid:48)
≤ Cd 2 H 3 /(cid:15) 2 episodes.
This contradicts the sample complexity lower bound in Theorem 4.
Now we discuss on how to prove Theorem 4.
Proof.
(Proof of Theorem 4) Set K 1 = cK for a positive constant c ≥ 2 .
We construct another algorithm ALG 2 for any possible ALG 1 in the following way: In ALG 2 , the agent ﬁrstly runs
ALG 1 for K 1 /c
=
K episodes.
After K episodes, suppose ALG 1 outputs a policy ˆ π according to certain policy distribution ν π .
ALG 2 executes the policy ˆ π in the following c − 1 c K 1 episodes.
The interaction ends after K 1 episodes.
ALG 2 can be regarded as an algorithm which ﬁrstly runs the online exploration algorithm ALG 1 for K episodes, and then evaluates the performance of the policy ˆ π in the following episodes.
We study the total regret of ALG 2 from episode K +1 to episode
K 1 .
Recently, Zhou et al.
(2020a) proposed the following regret lower bound for linear mixture MDPs.
In order to avoid confusion, we use the notation K 2 instead of K to denote the number of total episodes of the regret minimization problem.
Lemma 16.
(Theorem 5.6 in Zhou et al.
(2020a))
Let B > 1 and suppose K≥ max (cid:8) ( d − 1) 2 H/ 2 , ( d − 1) /
(32 H ( B − 1)) (cid:9) , d ≥ 4 , H ≥ 3 .
Then for any algorithm there exists an episodic, B -bounded linear mixture MDP parameterized by Θ
=
( θ 1 , · · · , θ H ) such that the expected regret is at least Ω( dH √ HK 2 ) .
To prove the above regret lower bound, Zhou et al.
(2020a) construct a class of hard instances which is an extension of the hard instance class for linear bandits (cf.
Theorem 24.1 in Lattimore & Szepesv´ari (2020)).
They show that for any algorithm, there exists a hard instance in the instance class such that the regret is at least Ω( dH √ HK 2 ) .
By Theorem 16 and setting
K
= K 1 , we know that for any possible algorithm ALG 2 , there exists a hard instance M
such that (cid:80) K 1 k =1 E
( V ∗ ( s 1 ) − V π k
( s 1 ))
≥ c 2 dH √ HK 1 for a positive constant c 2 , where π k denotes the policy used in episode k for algorithm ALG 2 .
The expectation is over all randomness of the algorithm and the environment.
Note that in the hard instance constructed in Zhou et al.
[Equation]
2020a)
, the per-step regret is at most
E
[Equation]
V
[Equation]
s
[Equation]
s
[Equation]
[Equation]
HK 1 for any episode k ≤ K 1 .
By choosing c = max (cid:110)
[Equation]
we know that for the instance M ,
[Equation] [Equation]
By the deﬁnition of ALG 2 , we have π k = ˆ π if k > K 1 /c .
Therefore, we have
[Equation] [Equation]
Dividing both sides by ( c − 1) K , we have
[Equation]
The above inequality indicates that, for any algorithm ALG 1 , there exists an instance M such that the expected sub-optimality gap of the policy returned by ALG 1 is at least
[Equation]
[Equation] [Equation]
dH (cid:112) H/K after collecting samples for K episodes.
Suppose ALG 1 returns an (cid:15) -optimal
policy
ˆ π with probability at least 1 − δ .
Recall that the per-step sub-optimality gap is at most E ( V ∗ ( s 1 ) − V π k ( s 1 )) ≤ dH 4 √ 2 (cid:113) HcK in the constructed hard instance.
We have
[Equation]
We set δ to be a constant satisfying 0 < δ < min (cid:110) 1 ,
[Equation]
(cid:111) .
Solving the above inequality, we have K ≥ Cd 2 H 3 (cid:15)for a positive constant C .
D I MPROVED B OUND FOR R EWARD - FREE E XPLORATION IN L INEAR MDP S
In this section, we explain how our choice of the exploration-driven reward can be used to improve the sample complexity bound for reward-free exploration in linear MDPs (Wang et al., 2020).
We study the same reward-free setting as that in Wang et al.
(2020), which is brieﬂy explained in Appendix D.1.
We describe our algorithm and bound in Appendix D.2, and prove our theorem in
Appendix D.3.
D. P RELIMINARIES
For the completeness of explanation, we brieﬂy restate the reward-free setting studied in Wang et al.
Compared with the setting in this work, the main differences are twofold: Firstly, they study the linear MDPs setting instead of linear mixture MDPs in this work.
Secondly, they study the standard reward-free exploration setting without the constraints of the plug-in solver.
The linear MDP assumption, which was ﬁrst introduced in Yang & Wang (2019); Jin et al.
(2020b), states that the model of the MDP can be represented by linear functions of given features.
Assumption 1.
(Linear MDP) an MDP M =
( S , A , P, R, H, ν ) is said to be a linear MDP if the following hold:
• There are d unknown signed measures µ h =
( µ (1) h , µ (2) h , · · · , µ ( d ) h ) such that for any ( s, a, s (cid:48) )
∈ S ×
A × S , P h ( s (cid:48) | s, a ) =
(cid:104)
µ h ( s (cid:48) ) , φ ( s, a ) (cid:105) .
• There exists H unknown vectors η 1 , η 2 , · · · , η
H ∈ R
d such that for any ( s, a ) ∈ S × A , R h
( s, a ) =
(cid:104) φ ( s, a ) , η h (cid:105) .
We assume for all ( s, a ) ∈ S
× A and h ∈
[ H ] , (cid:107) φ ( s, a ) (cid:107) ≤ 1 , (cid:107) µ h ( s ) (cid:107) 2 ≤ √ d and (cid:107) η (cid:107) 2 ≤ √ d .
For the reward-free exploration studied in Wang et al.
(2020), the agent can collects a dataset of visited state-action pairs D =
(cid:8)(cid:0)
s kh , a kh (cid:1)(cid:9) ( k,h ) ∈
[ K ] × [ H ] which will be used in the planning phase.
Then during the planning phase, the agent can follow a certain designed learning algorithm to calculate an (cid:15) -optimal policy w.r.t.
the reward function R using the dataset D .
D. A LGORITHM AND T HEOREM
Our algorithm can be divided into two parts.
The exploration phase of the algorithm is presented in
Algorithm 3, and the planning phase is presented in Algorithm 4. Algorithm
Input: Failure probability δ > 0 and target accuracy (cid:15) >β ← c β · dH (cid:112) log ( dHδ − 1 ε − 1 ) for some c β >
K ← c K · d 3 H 4 log (cid:0) dHδ − 1 ε − 1 (cid:1) /ε
2 for some c K >for episode k = 1 , 2 , · · · , K do 5:
Q k,H +1 ( · , · ) ← 0 , V k,H +1 ( · ) ←for step h
= H, H − 1 , · · · , 1 do Λ k,h ← (cid:80) k − 1 t =1 φ ( s t,h , a t,h ) φ ( s t,h , a t,h ) (cid:62)
+ I u k,h ( · , · ) ← β (cid:113) φ ( · , · ) (cid:62) (Λ k,h ) − 1 φ ( · , · )
Deﬁne the exploration-driven reward function R k,h ( · , · ) =
u k,h ( · , · ) 10: ˆ [Equation]
[Equation]
←
[Equation]
[Equation] [Equation] +
[Equation]
V
[Equation]
[Equation] ( s ) ← max a ∈A Q [Equation]
[Equation] [Equation] , a ) , π
[Equation]
[Equation] ( s ) =
arg max a ∈A Q k,h ( s, a ) end for for step h
= 1 , 2 , · · · , H do 15:
Take action a k,h = π k,h ( s k,h ) and observe s k,
h +1 ∼ P h
( s k,h , a k,h ) end for end for
Output: D ← { ( s k,h , a k,h ) } ( k,h ) ∈ [ K ] ×
[ H ] .
Input: Dataset D ← { ( s k,h , a k,h ) }
( k,h ) ∈
[ K ] × [ H ] , reward functions R =
{ R h } h ∈ [ H ]
Q k,H +1 ( · , · ) = 0 , V k,H +1 ( · ) =for step h
= H, H − 1 , · · · , 1 do Λ h ← (cid:80) Kt =1 φ ( s t,h , a t,h ) φ ( s t,h , a t,h ) (cid:62) + I 5: u h ( · , · ) ← min (cid:26) β (cid:113) φ ( · , · ) (cid:62) (Λ k,h ) − 1 φ ( · , · ) , H (cid:27) ˆ w h ← (Λ h ) − 1 (cid:80) Kt =1 φ ( s t,h , a t,h ) V t,h +
( s t,h +1 )
Q h ( · , · ) ← min (cid:8) ˆ w (cid:62) h φ ( · , · ) + R h ( · , · ) + u h ( · , · ) , H (cid:9)
V h ( s ) ← max a ∈A Q h ( s, a ) , π h ( s ) =
arg max a ∈A Q h ( s, a ) end for 10: Output: π = { π h } h ∈
[ H ]
Compared with the algorithm in Wang et al.
(2020), the main difference is that we set R k,h = u k,h in the exploration phase, instead of R k,h = u k,h /H .
The following theorem states the complexity bound of our algorithms.
Theorem 5.
With probability at least 1 − δ , after collecting O ( d 3 H 4 log( dHδ − 1 (cid:15) − 1 ) /(cid:15) 2 ) trajectories during the exploration phase, our algorithm outputs an (cid:15) -optimal policy for any given reward during the planning phase.
D. P ROOF OF T HEOREM
The proof of Theorem 5 follows the proof framework in Wang et al.
(2020) with a slight modiﬁcation.
Therefore, we only sketch the proof and mainly focus on explaining the differences.
Firstly, we introduce the value function ˜
V ∗ h ( s, R ) , which is recursively deﬁned from step H + 1 to step 1 :
[Equation]
Compared with the deﬁnition of V ∗ h ( s, R ) , the main difference is that we take minimization over the value and H at each step.
We can similarly deﬁne ˜
Q ∗ h ( s, a, R ) , ˜ V πh ( s, R ) and ˜ Q πh ( s, a, R ) .
Lemma 17.
With probability 1 − δ/ 2 , for all k ∈ [ K ] ,
[Equation]
and
[Equation]
(211) for some constant c > 0 where V k, 1 is as deﬁned in Algorithm 3.
This lemma corresponds to Lemma 3.1 in Wang et al.
The main difference is that we replace
V ∗ 1 with ˜ V ∗ 1 .
Note that ˜ V ∗ h ( s, R ) ≤ H by the deﬁnition of ˜ V ∗ h ( s, R ) .
Lemma 17 can be similarly proved following the proof of Lemma 3.1 in Wang et al.
and replacing V ∗ h by ˜ V ∗ h during the proof.
Lemma 18.
With probability 1 − δ/ 4 , for the function u h ( · , · ) deﬁned in Line 5 in Algorithm 4, we have
[Equation]
Compared with Lemma 3.2 in Wang et al.
(2020), we replace the term V ∗ 1 ( s, u h /H ) with ˜
V ∗ 1 ( s, u h ) .
Note that with our choice of exploration-driven reward in the exploration phase (i.e.
R k,h = u k,h ), we replace the term u h /H with u h in the expectation.
This lemma can be proved by following the proof of Lemma 3.2 in Wang et al.
Lemma 19.
With probability 1 − δ/ 2 , for any reward function satisfying Assumption 1 and all h ∈ [ H ] , we have
[Equation]
Since our algorithm in the planning phase is exactly the same with that of Wang et al.
(2020), this lemma shares the same idea of Lemma 3.3 in Wang et al.
Now we can prove Theorem 5 with the help of Lemma 18 and Lemma 19.
Proof.
(Proof of Theorem 5)
We condition on the events deﬁned in Lemma 18 and Lemma 19, which hold with probability at least 1 − δ .
By Lemma 19, we have for any s ∈ S ,
[Equation]
[Equation]
which implies
[Equation]
V 1 ( s 1 ) − V π 1 ( s 1 , R )] .
[Equation]
Note that 0 ≤ V h ( s ) ≤ H and 0 ≤ V πh ( s, R ) ≤ H since 0 ≤ R ( s, a ) ≤ 1 .
Therefore, we always have V h ( s ) − V πh ( s, R ) ≤ H .
For any s h , we have
[Equation]
[Equation] By recursively decomposing V h ( s h ) − V πh ( s h , R ) from step H to 1 , we have
[Equation]
By deﬁnition of ˜ V ∗ 1 ( s, u ) , we have E s ∼ µ (cid:104)
˜
V π 1 ( s, u ) (cid:105) ≤ 2 E s ∼ µ (cid:104) ˜
V
[Equation]
[Equation]
s, u ) (cid:105) .
By Lemma 18,
[Equation]
By taking K = c K d 3 H 4 log( dHδ − 1 (cid:15) − 1 ) /(cid:15) 2 for a sufﬁciently large constant c K > 0 , we have
[Equation]
[Equation]
[Equation]
E A UXILIARY L EMMAS
Lemma 20.
(Self-Normalized Bound for Vector-Valued Martingales, Theorem 1 and 2 in AbbasiYadkori et al.
Let { F t } ∞ t
=
be a ﬁltration.
Let { η t } ∞ t =1 be a real-valued stochatic process such that η t is F t -measurable and η t is conditionally R -sub-Gaussian for some R ≥ 0 , i.e.
[Equation]
[Equation]
Let { X t } ∞ t =1 be an R d -valued stochastic process such that X t is F t − 1 -measurable.
Assume that V is a d × d positive deﬁnite matrix.
For any t ≥ 0 , deﬁne
[Equation]
Then, for any δ > 0 , with probability at least 1 − δ , for all t ≥ 0 ,
[Equation] [Equation]
Further, let V =
Iλ, λ > 0 .
Deﬁne Y t = (cid:104)
X t , θ ∗ (cid:105) + η t and assume that (cid:107) θ ∗ (cid:107) 2 ≤ S , (cid:107) X t (cid:107) 2 ≤
L, ∀ t ≥ 1 .
Then for any δ > 0 ,with probability at least 1 − δ , for all t ≥ 0 , θ ∗ satisﬁes
[Equation]
where ˆ θ t is the l 2 -regularized least-squares estimation of θ ∗ with regularization parameter λ >based on history samples till step t .
Lemma 21.
(Lemma 11 in Abbasi-Yadkori et al.
Let { X t } ∞ t =1 be a sequence in R d , V a d × d positive deﬁnite matrix and deﬁne ¯ V t
= V + (cid:80) ts =
1 X s X (cid:62) s .
Then, we have that
[Equation]
Further , if (cid:107) X t (cid:107) 2 ≤ L for all t , then n (cid:88) t =
[Equation]
and ﬁnally, if λ min ( V ) ≥ max(1 , L 2 ) , then
[Equation]
Lemma 22.
(Bernstein inequality for vector-valued martingales, Theorem 4.1 in Zhou et al.
(2020a))
Let {G t } ∞ t =1 be a ﬁltration, { x t , η t } t ≥ 1 a stochastic process so that x t ∈ R d is G tmeasurable and η t ∈ R is
G t
+1 -measurable.
Fix R, L, σ, λ > 0 , µ ∗ ∈ R d .
For t ≥ 1 , let y t =
(cid:104) µ ∗ , x t (cid:105) + η t and suppose that η t , x t also satisfy
[Equation]
Then, for any 0 < δ < 1 , with probability at least 1 − δ , we have
[Equation]
229) where for t ≥ 1 , µ t =
Z − 1 t b t , Z t = λI + (cid:80) ti =
1 x i x (cid:62) i , b t = (cid:80) ti =
y
i x i and β t = 8 σ (cid:112)
[Equation]