T OWARDS D EPLOYMENT -E FFICIENT R EINFORCEMENT L EARNING : L OWER B OUND AND O PTIMALITY
A BSTRACT
Deployment efﬁciency is an important criterion for many real-world applications of reinforcement learning (RL).
Despite the community’s increasing interest, there lacks a formal theoretical formulation for the problem.
In this paper, we propose such a formulation for deployment-efﬁcient RL (DE-RL) from an “optimization with constraints” perspective: we are interested in exploring an MDP and obtaining a near-optimal policy within minimal deployment complexity , whereas in each deployment the policy can sample a large batch of data.
Using ﬁnite-horizon linear
MDPs as a concrete structural model, we reveal the fundamental limit in achieving deployment efﬁciency by establishing information-theoretic lower bounds, and provide algorithms that achieve the optimal deployment efﬁciency.
Moreover, our formulation for DE-RL is ﬂexible and can serve as a building block for other practically relevant settings; we give “Safe DE-RL” and “Sample-Efﬁcient DE-RL” as two examples, which may be worth future investigation.
I NTRODUCTION
In many real-world applications, deploying a new policy to replace the previous one is costly, while generating a large batch of samples with an already deployed policy can be relatively fast and cheap.
For example, in recommendation systems (Afsar et al., 2021), education software (Bennane et al., 2013), and healthcare (Yu et al., 2019), the new recommendation, teaching, or medical treatment strategy must pass several internal tests to ensure safety and practicality before being deployed, which can be time-consuming.
On the other hand, the algorithm may be able to collect a large amount of samples in a short period of time if the system serves a large population of users.
Besides, in robotics applications (Kober et al., 2013), deploying a new policy usually involves operations on the hardware level, which requires non-negligible physical labor and long waiting periods, while sampling trajectories is relatively less laborious.
However, deployment efﬁciency was neglected in most of existing RL literatures.
Even for those few works considering this important criterion (Bai et al., 2020; Gao et al., 2021; Matsushima et al., 2021), either their settings or methods have limitations in the scenarios described above, or a formal mathematical formulation is missing.
We defer a detailed discussion of these related works to Section 1.1.
In order to close the gap between existing RL settings and real-world applications requiring high deployment efﬁciency, our ﬁrst contribution is to provide a formal deﬁnition and tractable objective for Deployment-Efﬁcient Reinforcement Learning (DE-RL) via an “optimization with constraints” perspective.
Roughly speaking, we are interested in minimizing the number of deployments K under two constraints: (a) after deploying K times, the algorithm can return a near-optimal policy, and (b) the number of trajectories collected in each deployment, denoted as N , is at the same level across
K deployments, and it can be large but should still be polynomial in standard parameters.
Similar to the notion of sample complexity in online RL, we will refer to K as deployment complexity .
To provide a more quantitative understanding, we instantiate our DE-RL framework in ﬁnite-horizon linear MDPs 1 (Jin et al., 2019) and develop the essential theory.
The main questions we address are:
Q1:
What is the optimum of the deployment efﬁciency in our DE-RL setting?
Q2:
Can we achieve the optimal deployment efﬁciency in our DE-RL setting?
When answering these questions, we separately study algorithms with or without being constrained to deploy deterministic policies each time.
While deploying more general forms of policies can be practical (e.g., randomized experiments on a population of users can be viewed as deploying a mixture of deterministic policies), most previous theoretical works in related settings exclusively focused on upper and lower bounds for algorithms using deterministic policies (Jin et al., 2019; Wang et al., 2020b; Gao et al., 2021).
As we will show, the origin of the difﬁculty in optimizing deployment efﬁciency and the principle in algorithm design to achieve optimal deployment efﬁciency are generally different in these two settings, and therefore, we believe both of them are of independent interests.
As our second contribution, in Section 3, we answer Q1 by providing information-theoretic lower bounds for the required number of deployments under the constraints of (a) and (b) in Def 2.1.
We establish Ω p dH q and r Ω p H q lower bounds for algorithms with and without the constraints of deploying deterministic policies, respectively.
Contrary to the impression given by previous empirical works (Matsushima et al., 2021), even if we can deploy unrestricted policies, the minimal number of deployments cannot be reduced to a constant without additional assumptions, which sheds light on the fundamental limitation in achieving deployment efﬁciency.
Besides, in the line of work on “horizon-free RL” (e.g., Wang et al., 2020a), it is shown that RL problem is not signiﬁcantly harder than bandits (i.e., when H “ 1 ) when we consider sample complexity.
In contrast, the H dependence in our lower bound reveals some fundamental hardness that is speciﬁc to long-horizon RL, particularly in the deployment-efﬁcient setting.
2 Such hardness results were originally conjectured by Jiang & Agarwal (2018), but no hardness has been shown in sample-complexity settings.
After identifying the limitation of deployment efﬁciency, as our third contribution, we address Q2 by proposing novel algorithms whose deployment efﬁciency match the lower bounds.
In Section 4.1, we propose an algorithm deploying deterministic policies, which is based on Least-Square Value
Iteration with reward bonus (Jin et al., 2019) and a layer-by-layer exploration strategy, and can return an ε -optimal policy within O p dH q deployments.
As part of its analysis, we prove Lemma 4.2 as a technical contribution, which can be regarded as a batched ﬁnite-sample version of the well-known “Elliptical Potential Lemma”(Carpentier et al., 2020) and may be of independent interest.
Moreover, our analysis based on Lemma 4.2 can be applied to the reward-free setting (Jin et al., 2020; Wang et al., 2020b) and achieve the same optimal deployment efﬁciency.
In Section 4.2, we focus on algorithms which can deploy arbitrary policies.
They are much more challenging because it requires us to ﬁnd a provably exploratory stochastic policy without interacting with the environment.
To our knowledge, Agarwal et al.
(2020b) is the only work tackling a similar problem, but their algorithm is model-based which relies on a strong assumption about the realizability of the true dynamics and a sampling oracle that allows the agent to sample data from the model, and how to solve the problem in linear MDPs without a model class is still an open problem.
To overcome this challenge, we propose a model-free layer-by-layer exploration algorithm based on a novel covariance matrix estimation technique, and prove that it requires Θ p H q deployments to return an ε -optimal policy, which only differs from the lower bound r Ω p H q by a logarithmic factor.
Although the per-deployment sample complexity of our algorithm has dependence on a “reachability coefﬁcient” (see Def. 4.3), similar quantities also appear in related works (Zanette et al., 2020; Agarwal et al., 2020b; Modi et al., 2021) and we conjecture that it is unavoidable and leave the investigation to future work.
Finally, thanks to the ﬂexibility of our “optimization with constraints” perspective, our DE-RL setting can serve as a building block for more advanced and practically relevant settings where optimizing the number of deployments is an important consideration.
In Appendix F, we propose two potentially interesting settings: “Safe DE-RL” and “Sample-Efﬁcient DE-RL”, by introducing constraints regarding safety and sample efﬁciency, respectively.
1. C LOSELY R ELATED W ORKS
We defer the detailed discussion of previous literatures about pure online RL and pure ofﬂine RL to Appendix A, and mainly focus on those literatures which considered deployment efﬁciency and more related to us in this section.
To our knowledge, the term “deployment efﬁciency” was ﬁrst coined by Matsushima et al.
, but they did not provide a concrete mathematical formulation that is amendable to theoretical investigation.
In existing theoretical works, low switching cost is a concept closely related to deployment efﬁciency, and has been studied in both bandit (Esfandiari et al., 2020; Han et al., 2020;
Gu et al., 2021;
Ruan et al., 2021) and RL settings (Bai et al., 2020; Gao et al., 2021; Kong et al., 2021).
Another related concept is concurrent RL, as proposed by Guo & Brunskill (2015).
We highlight the difference with them in two-folds from problem setting and techniques.
As for the problem setting, existing literature on low switching cost mainly focuses on sub-linear regret guarantees, which does not directly implies a near-optimal policy after a number of policy deployments 3 .
Besides, low switching-cost RL algorithms (Bai et al., 2020; Gao et al., 2021; Kong et al., 2021) rely on adaptive switching strategies (i.e., the interval between policy switching is not ﬁxed), which can be difﬁcult to implement in practical scenarios.
For example, in recommendation or education systems, once deployed, a policy usually needs to interact with the population of users for a fair amount of time and generate a lot of data.
Moreover, since policy preparation is time-consuming (which is what motivates our work to begin with), it is practically difﬁcult if not impossible to change the policy immediately once collecting enough data for policy update, and it will be a signiﬁcant overhead compared to a short policy switch interval.
Therefore, in applications we target at, it is more reasonable to assume that the sample size in each deployment (i.e., between policy switching) has the same order of magnitude and is large enough so that the overhead of policy preparation can be ignored.
More importantly, on the technical side, previous theoretical works on low switching cost mostly use deterministic policies in each deployment, which is easier to analyze.
This issue also applies to the work of Guo & Brunskill (2015) on concurrent PAC RL.
However, if the agent can deploy stochastic (and possibly non-Markov) policies (e.g., a mixture of deterministic policies), then intuitively— and as reﬂected in our lower bounds—exploration can be done much more deployment-efﬁciently, and we provide a stochastic policy algorithm that achieves an r O p H q deployment complexity and overcomes the Ω p dH q lower bounds for deterministic policy algorithms (Gao et al., 2021).
P RELIMINARIES
Notation Throughout our paper, for n P Z ` , we will denote r n s “ t 1 , 2 , ..., n u .
r ¨ s denotes the ceiling function.
Unless otherwise speciﬁed, for vector x P R d and matrix X
P R d ˆ d , } x } denotes the vector l 2 -norm of x and } X } denotes the largest singular value of X .
We will use standard big-oh notations O p¨q , Ω p¨q , Θ p¨q , and notations such as r O p¨q to suppress logarithmic factors.
2. E PISODIC R EINFORCEMENT L EARNING
We consider an episodic Markov Decision Process denoted by M p S , A , H, P, r q , where S is the state space, A is the ﬁnite action space, H is the horizon length, and P “ t P h u
Hh “ 1 and r “ t r h u
Hh “denote the transition and the reward functions.
At the beginning of each episode, the environment will sample an initial state s 1 from the initial state distribution d 1 .
Then, for each time step h P r H s , the agent selects an action a h P A , interacts with the environment, receives a reward r h p s h , a h q , and transitions to the next state s h ` 1 .
The episode will terminate once s H ` 1 is reached.
A (Markov) policy π h p¨q at step h is a function mapping from S Ñ ∆ p
A q , where ∆ p A q denotes the probability simplex over the action space.
With a slight abuse of notation, when π h p¨q is a deterministic policy, we will assume π h p¨q : S Ñ A .
A full (Markov) policy π “ t π 1 , π 2 , ..., π H u speciﬁes such a mapping for each time step.
We use V πh p s q and Q πh p s, a q to denote the value function and Q-function at step h P r H s , which are deﬁned as:
[Equation]
We also use V ˚ h p¨q and Q ˚ h p¨ , ¨q to denote the optimal value functions and use π ˚ to denote the optimal policy that maximizes the expected return J p π q : “ E r
ř
Hh “ 1 r p s h , a h q| π s .
In some occasions, we use V πh p s ; r q and Q πh p s, a ; r q to denote the value functions with respect to r as the reward function for disambiguation purposes.
The optimal value functions and the optimal policy will be denoted by V ˚ p s ; r q , Q ˚ p s, a ; r q , π ˚ r , respectively.
Non-Markov Policies
While we focus on Markov policies in the above deﬁnition, some of our results apply to or require more general forms of policies.
For example, our lower bounds apply to non-Markov policies that can depend on the history (e.g., S 1 ˆ A 1 ˆ R ...
ˆ S h
´
1 ˆ A h
´ 1 ˆ R ˆ S
h Ñ A for deterministic policies); our algorithm for arbitrary policies deploys a mixture of deterministic Markov policies, which corresponds to choosing a deterministic policy from a given set at the initial state, and following that policy for the entire trajectory.
This can be viewed as a non-Markov stochastic policy.
2. L INEAR MDP S ETTING
We mainly focus on the linear MDP (Jin et al., 2019) satisfying the following assumptions:
Assumption A (Linear MDP Assumptions) .
An MDP M “ p S , A , H, P, r q is said to be a linear
MDP with a feature map φ : S ˆ A Ñ R d if the following hold for any h P r H s :
• There are d unknown signed measures µ h “ p µ p 1 q h , µ p 2 q h , ..., µ p d q h q over S such that for any p s, a, s 1 q P S ˆ A ˆ S , P h p s
|
s, a q “ x µ h p s 1 q , φ p s, a qy .
• There exists an unknown vector θ h P R d such that for any p s, a q P S ˆ A , r h p s, a q “ x φ p s, a q , θ h y .
Similar to Jin et al
and
Wang
et
al
(2020b), without loss of generality, we assume for all p s, a q P S ˆ A and h P r H s , } φ p s, a q} ď 1 , } µ h } ď ?
d , and } θ h } ď ?
d .
In Section 3 we will refer
to
linear MDPs
with
stationary dynamics,
which is a
special
case
when
µ 1 “ µ 2 “ . . .
µ H and θ 1 “ θ 2 “ . . .
“ θ H .
2. A C ONCRETE D EFINITION OF DE-RL
In the following, we introduce our formulation for DE-RL in linear MDPs.
For discussions of comparison to existing works, please refer to Section 1.1.
Deﬁnition 2.1 (Deployment Complexity in Linear MDPs) .
We say that an algorithm has a deployment complexity K in linear MDPs if the following holds: given an arbitrary linear MDP under
Assumption A, for arbitrary ε and
ă δ ă 1 , the algorithm will return a policy π K after K deployments and collecting at most N trajectories in each deployment, under the following constraints:
(a) With probability 1 ´ δ , π K is ε -optimal, i.e. J p π K q ě max π J p π q ´ ε .
(b) The sample size N is polynominal, i.e. N “ poly p d, H, 1 ε , log 1 δ q .
Moreover, N should be ﬁxed a priori and cannot change adaptively from deployment to deployment.
Under this deﬁnition, the goal of Deployment-Efﬁcient RL is to design algorithms with provable guarantees of low deployment complexity.
Polynomial Size of N
We emphasize that the restriction of polynomially large N is crucial to our formulation, and not including it can result in degenerate solutions.
For example, if N is allowed to be exponentially large, we can ﬁnish exploration in 1 deployment in the arbitrary policy setting, by deploying a mixture of exponentially many policies that form an ε -net of the policy space.
Alternatively, we can sample actions uniformly, and use importance sampling (Precup, 2000) to evaluate all of them in an off-policy manner.
None of these solutions are practically feasible and are excluded by our restriction on N .
L OWER B OUND FOR D EPLOYMENT C OMPLEXITY IN RL
In this section, we provide information-theoretic lower bounds of the deployment complexity in our
DE-RL setting.
We defer the lower bound construction and the proofs to Appendix B.
As mentioned in Section 2, we consider non-Markov policies when we refer to deterministic and stochastic policies in this section, which strengthens our lower bounds as they apply to very general forms of policies.
We ﬁrst study the algorithms which can only deploy deterministic policy at each deployment.
Theorem 3.1.
[Lower bound for deterministic policies, informal] For any d ě 4 , H and any algorithm ψ that can only deploy a deterministic policy at each deployment, there exists a linear MDP
M satisfying Assumption A, such that the deployment complexity of ψ in M is K “ Ω p dH q .
The basic idea of our construction and the proof is that, intuitively, a linear MDP with dimension d and horizon length H has Ω p dH q “independent directions”, while deterministic policies have limited exploration capacity and only reach Θ p 1 q direction in each deployment, which result in Ω p dH q deployments in the worst case.
In the next theorem, we will show that, even if the algorithm can use arbitrary exploration strategy (e.g. maximizing entropy, adding reward bonus), without additional assumptions, the number of deployments K still has to depend on H and may not be reduced to a constant when H is large.
Theorem 3.2.
[Lower bound for arbitrary policies, informal] For any d ě 4 , H, N and any algorithm ψ which can deploy arbitrary policies, there exists a linear MDP M satisfying Assumption A, such that the deployment complexity of ψ in M is K “ Ω p H { r log d p NH q s q “ r Ω p H q .
The origin of the difﬁculty can be illustrated by a recursive dilemma: in the worst case, if the agent does not have enough information at layer h , then it cannot identify a good policy to explore till layer h ` Ω p log d p NH qq in 1 deployment, and so on and so forth.
Given that we enforce N to be polynomial, the agent can only push the “information boundary” forward by Ω p log d p NH qq “ r Ω p 1 q layers per deployment.
In many real-world applications, such difﬁculty can indeed exist.
For example, in healthcare, the entire treatment is often divided into multiple stages.
If the treatment in stage h is not effective, the patient may refuse to continue.
This can result in insufﬁcient samples for identifying a policy that performs well in stage h ` 1 .
Stationary vs. non-stationary dynamics Since we consider non-stationary dynamics in Assump.
A, one may suspect that the H -dependence in the lower bound is mainly due to such nonstationarity.
We show that this is not quite the case, and the H -dependence still exists for stationary dynamics.
In fact, our lower bound for non-stationary dynamics directly imply one for stationary dynamics: given a ﬁnite horizon non-stationary MDP Ă M “ p r S , A , H, r P, r r q , we can construct a stationary MDP M “ p S , A , H, P, r q by expanding the state space to S “ r S ˆ r H s so that the new transition function P and reward function r are stationary across time steps.
As a result, given arbitrary d ě 4 and H ě 2 , we can construct a hard non-stationary MDP instance Ă M with dimension r d “ max t 4 , d { H u and horizon r h “ d { r d “ min t H, d { 4 u , and convert it to a stationary MDP
M with dimension d and horizon h “ r h “ min t H, d { 4 u ď H .
If there exists an algorithm which can solve M in K deployments, then it can be used to solve Ă M in no more than K deployments.
Therefore, the lower bounds for stationary MDPs can be extended from Theorems 3.1 and 3.2, as shown in the following corollary:
Corollary 3.3 (Extension to Stationary MDPs) .
For stationary linear MDP with d ě 4 and H ě 2 , suppose N “ poly p d, H, 1 ε , log 1 δ q , the lower bound of deployment complexity would be Ω p d q for deterministic policy algorithms, and Ω p [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
for algorithms which can deploy arbitrary policies.
As we can see, the dependence on dimension and horizon will not be eliminated even if we make a stronger assumption that the MDP is stationary.
The intuition is that, although the transition function is stationary, some states may not be reachable from the initial state distribution within a small number of times, so the stationary MDP can effectively have a “layered” structure.
For example, in
Atari games (Bellemare et al., 2013) (where many algorithms like DQN (Mnih et al., 2013) model the environments as inﬁnite-horizon discounted MDPs) such as Breakout, the agent cannot observe states where most of the bricks are knocked out at the initial stage of the trajectory.
Therefore, the agent still can only push forward the “information frontier” a few steps per deployment.
That said, it is possible reduce the deployment complexity lower bound in stationary MDPs by adding more assumptions, such as the initial state distribution providing good coverage over the entire state space, or all the states are reachable in the ﬁrst few time steps.
However, because these assumptions do not always hold and may overly trivialize the exploration problem, we will not consider them in our algorithm design.
Besides, although our algorithms in the next section are designed for nonstationary MDPs, they can be extended to stationary MDPs by sharing covariance matrices, and we believe the analyses can also be extended to match the lower bound in Corollary 3.3.
T OWARDS O PTIMAL D EPLOYMENT E FFICIENCY
In this section we provide algorithms with deployment-efﬁciency guarantees that nearly match the lower bounds established in Section 3.
Although our lower bound results in Section 3 consider nonMarkov policies, our algorithms in this section only use Markov policies (or a mixture of Markov policies, in the arbitrary policy setting), which are simpler to implement and compute and are already near-optimal in deployment efﬁciency.
Inspiration from Lower Bounds: a Layer-by-Layer Exploration Strategy
The linear dependence on H in the lower bounds implies a possibly deployment-efﬁcient manner to explore, which we call a layer-by-layer strategy: conditioning on sufﬁcient exploration in previous h ´ 1 time steps, we can use poly p d q deployments to sufﬁciently explore the h -th time step, then we only need
H ¨ poly p
d
q deployments to explore the entire MDP.
If we can reduce the deployment cost in each layer from poly p d q to Θ p d q or even Θ p 1 q , then we can achieve the optimal deployment efﬁciency.
Besides, as another motivation, in Appendix C.4, we will brieﬂy discuss the additional beneﬁts of the layer-by-layer strategy, which will be useful especially in “Safe DE-RL”.
In Sections 4.1 and 4.2, we will introduce algorithms based on this idea and provide theoretical guarantees.
4. D EPLOYMENT -E FFICIENT RL WITH D ETERMINISTIC P OLICIES Algorithm
: Layer-by-Layer Batch Exploration Strategy for Linear MDPs Given Reward Function
1 Input : Failure probability δ ą 0 , and target accuracy ε ą 0 , β Ð c β ¨ dH a log p dHδ ´ 1 ε ´ 1 q for some c β ą 0 , total number of deployments K , batch size N , 2 h 1 Ð// h k denotes the layer to explore in iteration k , for all k P r K s 3 for k “ 1 , 2 , ...,
K do
Q kh k ` 1 p¨ , ¨q Ð 0 and V kh k ` 1 p¨q “ for h “ h k , h k ´ 1 , ..., 1 do Λ k h Ð I ` ř k ´ 1 τ “ 1 ř N n “ 1 φ τn h p φ τn h q J , u k h p¨ , ¨q [Equation] min t β ¨[-b φ p¨ , ¨q J p Λ k h q ´ 1 φ p¨ , ¨q , H u w kh Ð p Λ kh q ´ 
-]
[Equation] p
[Equation] u [Equation] 1 q
Q kh p¨ , ¨q Ð min tp w kh q J φ p¨ , ¨q ` r h p¨ , ¨q ` u kh p¨ , ¨q , H u and V kh p¨q “ max a P A Q kh p¨ , a q π kh p¨q Ð arg max a P A Q kh p¨ , a q end
Deﬁne π k “ π k 1 ˝ π k 2 ...
˝
π kh k ˝ unif r h k ` 1: H s for n “ 1 , ..., N do
Receive initial state s kn 1 „ d for h “ 1 , 2 , ..., H do Take action a knh Ð π kh p s
end
20 end
In this sub-section, we focus on the setting where each deployed policy is deterministic.
In Alg 1, we propose a provably deployment-efﬁcient algorithm built on Least-Square Value Iteration with
UCB (Jin et al., 2019) 4 and the “layer-by-layer” strategy.
Brieﬂy speaking, at deployment k , we focus on exploration in previous h k layers, and compute π k 1 , π k 2 , ..., π kh k by running LSVI-UCB in an MDP truncated at step h k .
After that, we deploy π k to collect N trajectories, and complete the trajectory after time step h k with an arbitrary policy.
(In the pseudocode we choose uniform, but the choice is inconsequential.)
In line 19, we compute ∆ k with samples and use it to judge whether we should move on to the next layer till all H layers have been explored.
The theoretical guarantee is listed below, and the missing proofs are deferred to Appendix C.
Theorem 4.1 (Deployment Complexity) .
For arbitrary ε, δ ą 0 , and arbitrary c K ě 2 , as long as
N ě c
[Equation]
, where c is an absolute constant, by choosing
[Equation]
Algorithm 1 will terminate at iteration k ď K and return us a policy π k , and with probability 1 ´ δ ,
[Equation]
As an interesting observation, Eq (1) reﬂects the trade-off between the magnitude of K and N when K is small.
To see this, when we increase c K and keep it at the constant level, K deﬁnitely increases while N will be lower because its dependence on d, H, ε, δ decreases.
Moreover, the beneﬁt of increasing c K is only remarkable when c K is small (e.g. we have N “ O p H 9 d 6 ε ´ 4 q if c K “ 2 , while N “ O p H 5 d 3 .
6 ε ´ 2 .
4 q if c K “ 6 ), and even for moderately large c K , the value of N quickly approaches the limit lim c K Ñ8 N “ c H 4 d 3 ε 2 log 2 p Hdδε q .
It is still an open problem that whether the trade-off in Eq.1 is exact or not, and we leave it for the future work.
Another key step in proving the deployment efﬁciency of Alg. 1 is Lem. 4.2 below.
In fact, by directly applying Lem. 4.2 to LSVI-UCB
(Jin et al., 2019) with large batch sizes, we can achieve O p dH q deployment complexity in deterministic policy setting without exploring in a layer-by-layer manner. We defer the discussion and the additional beneﬁt of layer-by-layer strategy to Appx. C.4.
Lemma 4.2.
[Batched Finite Sample Elliptical Potential Lemma] Consider a sequence of matrices
A 0 , A N , ..., A p K ´ 1 q N P R d ˆ d with A 0 “ I d ˆ d and A kN “ A p k ´ 1 q N ` Φ k ´ 1 , where Φ k ´ 1 “ ř kNt “p k ´ 1 q N ` 1 φ t φ J t and max t ď KN } φ t } ď 1 .
We deﬁne:
K ` : “ !
[Equation]
[Equation]
1 q N Φ k ´ 1 q ě
Nε ) .
For arbitrary ε ă 1 , and arbitrary c K ě 2 , if K “ c K dH ` 1 , by choosing N ě c
[Equation]
q ¯ 1 cK ´ 1 , where c is an absolute constant independent with c K , d, H, ε , we have | K ` | ď c K d ă K { H.
Extension to Reward-free setting Based on the similar methodology, we can design algorithms for reward-free setting (Wang et al., 2020b) and obtain O p dH q deployment complexity.
We defer the algorithms and proofs to Appx. D, and summarize the main result in Thm. D.4.
4. D EPLOYMENT -E FFICIENT RL WITH A RBITRARY P OLICIES
From the discussion of lower bounds in Section 3, we know that in order to reduce the deployment complexity from Ω p dH q to r Ω p H q , we have to utilize stochastic (and possibly non-Markov) policies and try to explore as many different directions as possible in each deployment (as opposed todirection in Algorithm 1).
The key challenge is to ﬁnd a stochastic policy—before the deployment starts—which can sufﬁciently explore d independent directions.
In Alg. 2, we overcome this difﬁculty by a new covariance matrix estimation method (Alg. 6 in Appx. E).
The basic idea is that, for arbitrary policy π 5 , the covariance matrix Λ π h : “ E π r φφ J s can
Here we mainly focus on evaluating deterministic policy or stochastic policy mixed from a ﬁnite number of deterministic policies, because for the other stochastic policies, exactly computing the expectation over policy distribution may be intractable.
1 Input : Accuracy level ε ; Iteration number i max ; Resolution ε 0 ; Reward r ; Bonus coefﬁcient β .
2 for h “ 1 , 2 , ..., H do
Initialize π h, 1 with an arbitrary deterministic policy ; r Σ h, 1 “ 2 I , Π h “ tu .
for i “ 1 , 2 , ..., i max do p Λ π h,i h Ð EstimateCovMatrix p h, D r 1: h ´ 1 s , Σ r 1: h ´ 1 s , π h,i q
# Alg 6, Appx E r Σ h,i ` 1 “ r Σ h,i ` p Λ π h,i h
V h,i ` 1 , ¯ π h,i ` 1 Ð SolveOptQ p h, D r 1: h ´ 1 s , Σ r 1: h ´ 1 s , β, r Σ h,i ` 1 , ε 0 q
# Alg 5, Appx E if V h,i ` 1 ď 3 ν 2min { 8 then break ; Π h “ Π h Ť t ¯ π h,i ` 1 u end
Σ h
“ I , D h “ tu , π h, mix : “ unif p Π h q for n “ 1 , 2 , ..., N do
Sample trajectories with π h, mix Σ h “ Σ h ` φ p s h,n , a h,n q φ p s h,n , a h,n q J ,
D h “ D h Ť t s h,n , a h,n , r h,n , s h ` 1 ,n u end 16 end 17 return p π r Ð Alg 4 p H, t D 1 , ..., D H u , r q be estimated element-wise by running policy evaluation for π with φ i φ j as a reward function, where i, j P r d s and φ i denotes the i -th component of vector φ .
However, a new challenge emerging is that, because the transition is stochastic, in order to guarantee low evaluation error for all possible policies ¯ π h,i ` 1 , we need an union bound over all policies to be evaluated, which is challenging if the policy class is inﬁnite.
To overcome this issue, we discretize the value functions in Algorithm 5 (see Appendix E) to allow for a union bound over the policy space: after computing the Q-function by LSVI-UCB, before converting it to a greedy policy, we ﬁrst project it to an ε 0 -net of the entire Q-function class.
In this way, the number of policy candidates is ﬁnite and the projection error can be controlled as long as ε 0 is small enough.
Using the above techniques, in Lines 3-10, we repeatedly use Alg 6 to estimate the accumulative covariance matrix r Σ h,i ` 1 and further eliminate uncertainty by calling Alg 5 to ﬁnd a policy (approximately) maximizing uncertainty-based reward function r R : “ } φ } r Σ ´ 1 h,i ` 1 .
For each h P r H s , inductively conditioning on sufﬁcient exploration in previous h ´ 1 layers, the errors of Alg 6 and
Alg 5 will be small, and we will ﬁnd a ﬁnite set of policies Π h to cover all dimensions in layer h .
(This is similar to the notion of “policy cover” in Du et al.
Agarwal et al.
(2020a).)
Then, layer h can be explored sufﬁciently by deploying a uniform mixture of Π and choosing N large enough (Lines 11-15).
Also note that the algorithm does not use the reward information, and is essentially a reward-free exploration algorithm.
After exploring all H layers, we obtain a dataset t D 1 , ..., D H u and can use Alg 4 for planning with any given reward function r satisfying Assump.
A to obtain a near-optimal policy.
Deployment complexity guarantees We ﬁrst introduce a quantity denoted as ν min , which measures the reachability to each dimension in the linear MDP.
In Appendix E.8, we will show that the ν min is no less than the “explorability” coefﬁcient in Deﬁnition 2 of Zanette et al.
and ν 2min is also lower bounded by the maximum of the smallest singular value of matrix E π r φφ J s .
Deﬁnition 4.3 (Reachability Coefﬁcient) .
[Equation]
Now, we are ready to state the main theorem of this section, and defer the formal version and its proofs to Appendix E.
Our algorithm is effectively running reward-free exploration and therefore our results hold for arbitrary linear reward functions.
Theorem 4.4.
[Informal] For arbitrary 0 ă ε, δ ă 1 , with proper choices of i max , ε 0 , β , we can choose N “ poly p d, H, 1 ε , log 1 δ , 1 ν min q , such that, after K “ H deployments, with probability 1 ´ δ ,
Algorithm 2 will collect a dataset D “ t D 1 , ..., D H u , and if we run Alg 4 with D and arbitrary reward function satisfying Assump.
A, we will obtain p π r such that V p π r 1 p s 1 ; r q ě V ˚ 1 p s 1 ; r q ´ ε .
Proof Sketch Next, we brieﬂy discuss the key steps of the proof.
Since ε 0 can be chosen to be very small, we will ignore the bias induced by ε 0 when providing intuitions.
Our proof is based on the induction condition below.
We ﬁrst assume it holds after h ´ 1 deployments (which is true when h “ 1 ), and then we try to prove at the h -th deployment we can explore layer h
well enough so that the condition holds for h .
Condition 4.5.
[Induction Condition] Suppose after h ´ 1 deployments, we have the following induction condition for some ξ ă 1 { d , which will be determined later:
[Equation]
The l.h.s. of Eq.(2) measures the uncertainty in previous h ´ 1 layers after exploration.
As a result, with high probability, the following estimations will be accurate:
[Equation]
where } ¨ } 8 , 8 denotes the entry-wise maximum norm.
This directly implies that:
[Equation]
where Σ h,i ` 1 : “ 2 I ` ř ii 1 “ 1 E π h,i 1 r φ p s h , a h q φ p s h , a h q J s is the target value for r Σ h,i ` 1 to approximate.
Besides, recall that in Algorithm 5, we use
[Equation]
i ` 1 φ as the reward function, and the induction condition also implies that:
[Equation]
As a result, if ξ and the resolution ε 0 are small enough, ¯ π h,
i ` 1 would gradually reduce the uncertainty and V h,i ` 1 (also max π E π r} φ p s h , a h q} r Σ ´ 1 h,i ` 1 s ) will decrease.
However, the bias is at the level
O p ξ q , and therefore, no matter how small ξ is, as long as ξ ą 0 , it is still possible that the policies in Π h do not cover all directions if some directions are very difﬁcult to reach, and the error due to such a bias will be at the same level of the required accuracy in induction condition, i.e. O p ξ q .
This is exactly where the “reachability coefﬁcient” ν min deﬁnition helps.
The introduction of ν min provides a threshold, and as long as ξ is small enough so that the bias is lower than the threshold, each dimension will be reached with substantial probability when the breaking criterion in Line 9 is satisﬁed.
As a result, by deploying unif p Π h q and collecting a sufﬁciently large dataset, the induction condition will hold till layer H .
Finally, combining the guarantee of Alg 4, we complete the proof.
C ONCLUSION AND F UTURE W ORK
In this paper, we propose a concrete theoretical formulation for DE-RL to ﬁll the gap between existing RL literatures and real-world applications with deployment constraints.
Based on our framework, we establish lower bounds for deployment complexity in linear MDPs, and provide novel algorithms and techniques to achieve optimal deployment efﬁciency.
Besides, our formulation is ﬂexible and can serve as building blocks for other practically relevant settings related to DE-RL.
We conclude the paper with two such examples, defer a more detailed discussion to Appendix F, and leave the investigation to future work.
Sample-Efﬁcient DE-RL
In our basic formulation in Deﬁnition 2.1, we focus on minimizing the deployment complexity K and put very mild constraints on the per-deployment sample complexity
N .
In practice, however, the latter is also an important consideration, and we may face additional constraints on how large N can be, as they can be upper bounded by e.g. the number of customers or patients our system is serving.
Safe DE-RL In real-world applications, safety is also an important criterion.
The deﬁnition for safety criterion in Safe DE-RL is still an open problem, but we believe it is an interesting setting since it implies a trade-off between exploration and exploitation in deployment-efﬁcient setting.
A CKNOWLEDGEMENTS
JH’s research activities on this work were completed by December 2021 during his internship at
MSRA.
NJ acknowledges funding support from ARL Cooperative Agreement W911NF-17-2-0196,
NSF IIS-2112471, and Adobe Data Science Research Award
A E XTENDED R ELATED W ORK
Online RL Online RL is a paradigm focusing on the challenge of strategic exploration.
On the theoretical side, based on the “Optimism in Face of Uncertainty”(OFU) principle or posterior sampling techniques, many provable algorithms have been developed for tabular MDPs (Jin et al., 2018; Azar et al., 2017; Zanette & Brunskill, 2019; Agrawal & Jia, 2017;
Agrawal et al., 2021), linear MDPs (Jin et al., 2019; Agarwal et al., 2020a), general function approximation (Wang et al., 2020c; Russo & Van Roy, 2013), or MDPs with structural assumptions (Jiang et al., 2017b; Du et al., 2019).
Moreover, there is another stream of work studying how to guide exploration by utilizing state occupancy (Hazan et al., 2019; Zhang et al., 2021).
Beyond the learning in MDPs with pre-speciﬁed reward function, recently, Jin et al.
; Wang et al.
(2020b)
; Zanette et al.
(2020) provide algorithms for exploration in the scenarios where multiple reward functions are of interest.
On the practical side, there are empirical algorithms such as intrinsically-motivated exploration (Bellemare et al., 2016;
Campero et al., 2020), exploration with hand-crafted reward bonus (RND) (Burda et al., 2018), and other more sophisticated strategies (Ecoffet et al., 2019).
However, all of these exploration methods do not take deployment efﬁciency into consideration, and will fail to sufﬁciently explore the MDP and learn near-optimal policies in DE-RL setting where the number of deployments is very limited.
Ofﬂine RL Different from the online setting, where the agents are encouraged to explore rarely visited states to identify the optimal policy, the pure ofﬂine RL setting serves as a framework for utilizing historical data to learn a good policy without further interacting with the environment.
Therefore, the core problem of ofﬂine RL is the performance guarantee of the deployed policy, which motivated multiple importance-sampling based off-policy policy evaluation and optimization methods (Jiang & Li, 2016; Liu et al., 2018; Uehara et al., 2020; Yang et al., 2020; Nachum et al., 2019; Lee et al., 2021), and the “Pessimism in Face of Uncertainty” framework (Liu et al., 2020;
Kumar et al., 2020; Fujimoto & Gu, 2021; Yu et al., 2020; Jin et al., 2021b;
Xie et al., 2021a) in contrast with OFU in online exploration setting.
However, as suggested in Matsushima et al.
(2021), pure ofﬂine RL can be regarded as constraining the total number of deployments to be 1 .
Bridging Online and Ofﬂine RL; Trade-off between Pessimism and Optimism As pointed out by Xie et al.
(2021b)
; Matsushima et al.
(2021), there is a clear gap between existing online and ofﬂine RL settings, and some efforts have been made towards bridging them.
For example, Nair et al.
(2021); Pertsch et al.
; Bai et al.
(2020) studied how to leverage pre-collected ofﬂine datasets to learn a good prior to accelerate the online learning process.
Moskovitz et al.
(2021) proposed a learning framework which can switch between optimism and pessimism by modeling the selection as a bandit problem.
None of these works give provable guarantees in our deployment-efﬁciency setting.
Conversion from Linear Regret in Gao et al.
(2021) to Sample Complexity Gao et al.
(2021) proposed an algorithm with the following guarantee: after interacting with the environments for r
K times (we use r K to distinguish with K in our setting), there exists a constant c such that the algorithm’s regret is
[Equation]
where we denote T “ r KH , and use ι to refer to the log terms.
Besides, in π 1 , ..., π K there are only
O p dH log r K q
policy switching.
As discussed in Section 3.1 by Jin et al.
(2018), such a result can be convert to a PAC guarantee that, by uniformly randomly select a policy π from π 1 , ..., π K , with probability at least 2/3, we should have:
[Equation]
In order to make sure the upper bound in the r.h.s.
will be ε , we need:
[Equation]
and the required policy switching would be:
[Equation]
In contrast with our results in Section 4.
[Table]
there is an additional logarithmic dependence on d, H and ε .
Moreover
[Table]
since their algorithm only deploys deterministic policies, their deployment complexity has to depend on d
[Table]
which is much higher than our stochastic policy algorithms in Section 4.2 when d is large.
[Table]
in our setting deﬁned in Def. 2.1, where R-F is the short note for Reward-Free.
The total number of trajectories cost by our methods is computed by K ¨ N .
We omit log terms in r O .
For algorithm (Jin et al., 2019), we report the sample complexity after the conversion from regret.
For our deterministic policy algorithms, we report the asymptotic results when c K Ñ `8 , which can be achieved approximately when c K is a large constant (e.g. c K “ 100 ).
Investigation on Trade-off between Sample Complexity and Deployment Complexity In Table 1, we compare our algorithms and previous online RL works which did not consider deployment efﬁciency to shed light on the trade-off between sample and deployment complexities.
Besides algorithms that are specialized to linear MDPs, we also include results such as Zanette et al.
(2020), which studied a more general linear approximation setting and can be adapted to our setting.
As stated in Def. 2 of Zanette et al.
, they also rely on some reachability assumption.
To avoid ambiguity, we use r ν min to refer to their reachability coefﬁcient (as discussed in Appx E.8, r ν min is no larger than and can be much smaller than our ν min ).
Because they also assume that ε ď r
O p r ν min {?
d q (see Thm 4.1 in their paper), their results have an implicit dependence on r ν ´ 2min .
In addition, by using the class of linear functions w.r.t.
φ , Q-type OLIVE (Jiang et al., 2017a; Jin et al., 2021a) has r O p
d 3 H 6 ε 2 q sample complexity and O p dH log p 1 { ε qq deployment complexity.
Its deployment complexity is close to our deterministic algorithm, but with additional dependence on ε .
We also want to highlight that OLIVE is known to be computationally intractable (Dann et al., 2018), while our algorithms are computationally efﬁcient.
With the given feature φ in linear MDPs and additional reachability assumption (not comparable to us), we can use a simpliﬁed version of MOFFLE (Modi et al., 2021) by skipping their LearnRep subroutine.
Though this version of
MOFFLE is computationally efﬁcient and its deployment complexity does not depend on ε , it has much worse sample complexity ( η min is their reachability coefﬁcient) and deployment complexity.
On the other hand, PCID (Du et al., 2019) and HOMER (Misra et al., 2020) achieve H deployment complexity in block MDPs.
However, block MDPs are more restricted than linear MDPs and these algorithms have worse sample and computational complexities.
It is worth to note that all our algorithms achieve the optimal dependence on ε (i.e., ε ´ 2 ) in sample complexity.
For algorithms that deploy deterministic policies, we can see that our algorithm has higher dependence on d and H in the sample complexity in both reward-known and reward-free setting, while our deployment complexity is much lower.
Our stochastic policy algorithm (last row) is naturally a reward-free algorithm.
Comparing with Wang et al.
(2020b) and Zanette et al.
(2020), our sample complexity has higher dependence on d and the reachability coefﬁcient ν min , while our algorithm achieves the optimal deployment complexity and better dependence on H .
B O N THE L OWER B OUND OF D EPLOYMENT -E FFICIENT R EINFORCEMENT L EARNING B. A H ARD MDP I NSTANCE T EMPLATE
In this sub-section, we ﬁrst introduce a hard MDP template that is used in further proofs.
As shown in Figure 1, we construct a tabular MDP (which is a special case of linear MDP) where the horizon length is H ` 1 and in each layer except the ﬁrst one, there are d ` 2 states and 2 d ` 1 different state action pairs.
The initial state is ﬁxed as s 0 and there are d ` 2 different actions.
It is easy to see that we can represent the MDP by linear features with at most 2 d ` 1 dimensions, and construct reward and transition function satisfying Assumption A.
As a result, it is a linear MDP with dimension
2 d ` 1 and horizon length H ` 1 .
Since there is only a constant-level blow up of dimension, the dimension of these MDPs is still Θ p d q , and we will directly use d instead of Θ p d q in the rest of the proof.
The states in each layer h ě 1 can be divided into three groups and we introduce them one-by-one in the following.
Group 1: Absorbing States (Green Color)
The ﬁrst group G 1 h “ t u 1 h , u 2 h u consists of two ab
- sorbing states u 1 h and u 2 h , which can only take one action at each state ¯ a 1 h and ¯ a 2 h and transit to u 1 h ` 1 and u 2 h ` 1 with probability 1, respectively.
The reward function is deﬁned as r h p u 1 h , ¯ a 1 h q “ r h p u 2 h , ¯ a 2 h q “ 0 . 5 for all h ď H ´ 2 and r H p u 1 H , ¯ a 1 H q “ 0 . 0 , r H p u 2 H , ¯ a 2 H q “ 1 . 0 .
Group 2: Core States (Red Color)
The second group G 2 h “ t s ˚ h u only contains one state, which we call it core state and denote it as s ˚ h .
For example, in Figure 1, we have s ˚ 1 “ s d 1 , s ˚ 2 “ s 12 and s ˚ 3 “ s 23 .
In the “core state”, the agent can take d actions r a 1 h , r a 2 h , . . .
, r a dh and transit deterministically to s 1 h ` 1 , s 2 h ` 1 , . . .
, s dh ` 1 .
Besides, the reward function is r h p s ˚ h , r a ih q “ 0 . 5 for all i P r d s .
Group 3: Normal States (Blue Color)
The third group G 3 h “ t s ih | i P r d s , s ih ‰ s ˚ h u is what we call “normal states”, and each state s ih P G 3 h can only take one action r a ih and will transit randomly to one of the absorbing states in the next layer, i.e. G 1 h ` 1 .
Besides, the reward function is r h p
s i h , r a i h q “ 0 . 5 for arbitrary s ih P G 3 h , and the transition function is P p u 1 h ` 1 | s ih , r a ih q “ P p u 1 h ` 1 | s ih , r a ih q “ 0 . 5 , except for a state action pair s # : “ s i # h # , a # : “ a i # h # at layer h
# P r H ´ 1 s with index
i # , such that s # R G 2 h # and P p
u 1 h ` 1 | s i
# h # , a i # h # q “ 0 .
5 ´ ε and P p
u 2 h ` 1 | s i
# h # , a i # h # q “ 0 .
5 ` ε .
In the following, we will call s # , a # the “optimal state” and “optimal action” in this MDP.
Note that the “optimal state” can not be the core state.
We will use M p h # , i # , I core “ t i 1 , i 2 , ..., i H uq with i
h
# ‰ i # to denote the MDP whose optimal state is at layer
h # and indexed by i # , and the core states in each layer are
s i 1 h , s i 2 h , ..., s i H H .
As we can see, the only optimal policy should be the one which can generate the following sequence of states before transiting to absorb states at layer
h # ` 1 :
[Equation]
and the optimal value function would be 12 H ` ε .
In order to achieve ε -optimal policy, the algorithm should identify s i
# h # , which is the main origin of the difﬁculty in exploration.
Remark B.1 (Markov v.s. Non-Markov Policies) .
As we can see, the core states in each layer are the only states with #actions ą 1, and for each core state, there exists and only exists one
… … … … … … deterministic path (a sequence of states, actions and rewards) from initial state to it, which implies that for arbitrary non-Markov policy, there exists an equivalent Markov policy.
Therefore, in the rest of the proofs in this section, we only focus on Markov policies.
B. L OWER BOUND FOR ALGORITHMS WHICH CAN DEPLOY DETERMINISTIC POLICIES ONLY
In the following, we will state the formal version of the lower bound theorem for deterministic policy setting and its proof.
The intuition of the proof is that we can construct a hard instance, which can be regarded as a Ω p dH q multi-arm bandit problem, and we will show that in expectation the algorithm need to “pull Ω p dH q arms” before identifying the optimal one.
Theorem B.2 (Lower bound for number of deployments in deterministic policy setting) .
For the linear MDP problem with dimension d and horizon H , given arbitrary algorithm ψ ( ψ can be deterministic or stochastic), which can only deploy a deterministic policy but can collect arbitrary number of samples in each deployment, there exists a MDP problem where the optimal deterministic policy π ˚ is ε better than all the other deterministic policies, but the estimate policy p π (which is also a deterministic policy) of the best policy output by ψ after K deployments must have P p π ˚ ‰
p π q ě 1 { 10 unless the number of deployments K ą p d ´ 1 qp H ´ 1 q{ 2 “ Ω p dH q .
Proof.
First of all, we introduce how we construct hard instances.
Construction of Hard Instances We consider a set of MDPs ¯ M , where for each MDP in that set, the core states (red color) in each layer are ﬁxed to be s 11 , s 12 , ..., s 1 H ´ 1 and the only optimal states which has different probability to transit to absorbing states are randomly selected from p d ´ 1 qp H ´ 1 q normal states (blue color).
Easy to see that, | ¯ M | “ p d ´ 1 qp H ´ 1 q .
Because of the different position of optimal states, the optimal policies for each MDP in ¯ M (i.e. the policy which can transit from s 0 to optimal state) is different.
We will use π 1 , π 2 , ..., π p d ´ 1 qp H ´ 1 q to refer to those different policies and use M π i with 1 ď i ď p d ´ 1 qp H ´ 1 q to denote the MDP in ¯ M where π i is the optimal policy.
For convenience, we will use M 0 to denote the MDP where all the normal states have equal probability to transit to different absorbing states, i.e., all states are optimal states.
Based on the introduction above, we deﬁne M : “ ¯ M Ť t M 0 u and use the MDPs in
M as hard instances.
Lower Bound for Average Failure Probability Next, we try to lower bound the average failure probability, which works as a lower bound for the maximal failure probability among MDPs in
M .
Since any randomized algorithm is just a distribution over deterministic ones, and it therefore sufﬁces to only consider deterministic algorithms ψ (Krishnamurthy et al., 2016).
Given an arbitrary algorithm ψ and k P r K s , we use ψ p k q to denote the policy taken by ψ at the k -th deployment (which is a random variable).
Besides, we denote ψ p K ` 1 q as the output policy.
For arbitrary k P r K s , we use P M πi ,ψ p ψ p k q “ π j q with 1 ď i, j ď p d ´ 1 qp H ´ 1 q to denote the probability that ψ takes policy π j at deployment k when running ψ on M π
i , and use P M πi ,ψ p ψ p K ` 1 q “ π i q to denote the probability that the algorithm ψ returns policy π i as optimal arm after running with K deployments under MDP M π i .
We are interested in providing an upper bound for the expected success rate:
[Equation]
where we assume that all the policies in M 0 are optimal policies.
In the following, we use E k,π i to denote the event that the policy π i has been deployed at least once in the ﬁrst k deployments and P M πj ,ψ p¨q to denote the probability of an event when running algorithm ψ under MDP M π j .
Next, we prove that, for arbitrary M π i ,
[Equation]
First of all, it holds for k “ 1 , because at the beginning ψ has’t observe any data, and all its possible behavior should be the same in both M 0 and M i , and therefore P M πi ,ψ p E
A 1 ,π i q “ P M 0 ,ψ p E
A 1 ,π i q .
Next, we do induction.
Suppose we already know it holds for 1 , 2 , ..., k , then consider the case for k ` 1 .
Because ψ behave the same if the pre-collected episodes are the same, which is the only information it will use for decision, we should have:
[Equation]
[Equation]
The second equality is due to each trajectory τ P ψ p k ` 1 q “ π i X E
A k,π i has the same probability under M 0 and M i by the construction.
Notice that in this induction step, we only consider the trajectory with ﬁrst p k ` 1 q N episodes because deﬁne the whole sample space and event only based on the ﬁrst p k ` 1 q N episodes.
This implies that,
[Equation]
Now we are ready to bound the failure rate.
Suppose K ă p d ´ 1 qp H ´ 1 q{ 2 ă | M |{ 2 , we have:
[Equation]
p E K ` 1 ,π
i q ´
P M πi ,ψ p ψ p K ` 1 q “ π i | E K ` 1 ,π i q ´ P M 0 ,ψ p ψ p K ` 1 q “ π i | E K ` 1 ,π i q ¯ (Eq.(5) and P p
A X B q “ P p B q P p
A | B q ) ď| M | | ¯ M | ÿ i “
P M πi ,ψ p E K ` 1 ,π i q ( P M πi ,ψ p ψ p K ` 1 q “ π i | E K ` 1 ,π i q ´ P M 0 ,ψ p ψ p K ` 1 q “ π i | E K ` 1 ,π i q ď 1 ) “| M | | ¯ M | ÿ i “
P M 0 ,ψ p E K ` 1 ,π i q ( Eq.
ď K | M | where the last step is because:
[Equation]
r 1 t π i is selected us ď
[Equation] [Equation]
1 t π i is selected at deployment k us “
[Equation]
E
[Equation]
ÿ
[Equation]
|
[Equation]
ÿ
[Equation] [Equation]
1 t π i is selected at deployment k us “
[Equation]
| M
[Equation]
( ψ deploy deterministic policy each time)
[Equation]
(Deployment
[Equation]
As a result, | M | ` | M | | ¯ M | ÿ i “
P M πi ,ψ p ψ p K ` 1 q “ π i q ď | M | ` | M | | ¯ M | ÿ i “
P M 0 ,ψ p ψ p K ` 1 q “ π i q ` 1 2 ď | M | ` 2 .
As long as d, H ě 3 , we have | M | “ p d ´ 1 qp H ´ 1 q ` 1 ě 5 the failure rate will be higher than 1 { 10 , which ﬁnishes the proof.
˝ B. P ROOF FOR L OWER B OUND IN A RBITRARY S ETTING
In the following, we provide a formal statement of the lower bound theorem for the arbitrary policy setting and its proof.
Theorem B.3 (Lower bound for number of deployments in arbitrary setting) .
For the linear MDP problem with given dimension d ě 2 and horizon H ě 3 , N “ poly p d, H, 1 ε , log 1 δ q , and arbitrary given algorithm ψ .
Unless the number of deployments K ą
H
[Equation]
log d NH s “ Ω p H { log
[Equation]
NH qq “ r Ω p H q , for any ε , there exists an MDP such that the output policy is not ε -optimal with probability at least 1 2 e .
Here ψ can be deterministic or stochastic.
The algorithm can deploy arbitrary policy but can only collect N “ poly p d, H, 1 ε , log 1 δ q samples in each deployment.
Proof.
Since any randomized algorithm is just a distribution over deterministic ones, it sufﬁces to only consider deterministic algorithms ψ in the following proof (Krishnamurthy et al., 2016).
The crucial part here is notice that a deployment means we have a ﬁxed distribution (occupancy) over the state space and such distribution only depends on the prior information.
Construction of Hard Instances We have d H ´ 2 ˆ d instances by enumerating the location of core state from level 1 to H ´ 2 and the optimal normal state at level H ´ 1 .
We assign s p
[Table]
as the core state at level H ´ 1 if s iH ´ 1 is the optimal state.
Notice that for this hard instance class, we only consider the case that the optimal state is in level H ´ 1 .
We use M to denote this hard instance class.
We make a few claims and later prove these claims and the theorem.
We will use the notation E i p j q to denote the event that at least one state at level j is reached by the i -th deployment.
Also we notice that in all the discussion an event is just a set of trajectories.
For all related discussion, the state at level L does not include the state in the absorbing chain.
In addition, we will use P M ,ψ to denote the distribution of trajectories when executing algorithm ψ and uniformly taking an instance from the hard instance class.
Claim 1.
Assume L ď H ´ 2 .
Then for any deterministic algorithm ψ , we have
[Equation]
Claim 2.
Assume L ` L 1 ď H ´ 2 .
We have that for any deterministic algorithm ψ ,
[Equation]
Proof of Claim 1 By the nature of the deterministic algorithm, we know that for any deterministic algorithm ψ , the deployment is the same at the ﬁrst time for all instances.
The reason is that the agent hasn’t observed anything, so the deployed policy has to be the same.
Let p p i, j, h q denote the probability of the ﬁrst deployment policy to choose action j at node i at level h under the ﬁrst deployment policy.
We know that p p
i, j, h q for ψ is the same under all instances.
Note that there is a one to one correspondence between an MDP in the hard instance class and the speciﬁed locations of core states in the layer 1 , . . .
, H ´ 2 and the optimal state at level H ´ 1 .
Therefore, we can use p i 1 , . .
, i H ´ 2 , s H ´ 1 q to denote any instance in the hard instance class, where i 1 , . .
, i H ´ 2 refers to the location of the core states and s H ´ 1 refers to the location of the optimal state.
From the construction, we know that for instance p i 1 , . . .
, i H ´ 2 , s H ´ 1 q , to arrive at level s L at level L , the path as to be s 0 , i 1 , . .
, i L ´ 1 , s L .
Therefore the probability of a trajectory sampled from ψ to reach state s L is
[Equation]
Here we use p p
i L ´ 1 , s L , L ´ 1 q denotes the probability of taking action at i L ´
to transit to s L and similarly for others.
In the deployment, ψ draws N episodes, so the probability of executing ψ to reach any state s L at level L during the ﬁrst deployment is no more than
[Equation]
[Equation]
Calculating the sum over i 1 , . .
, i L ´ 1 and s L gives us ÿ
[Equation]
Therefore we have the following equation about P M ,ψ p E 1 p L qq
[Equation]
Proof of Claim 2 Let τ denote any possible concatenation of the ﬁrst kN episodes we get in the ﬁrst k deployments.
In this claim, it sufﬁces to consider the kN episodes because the event
E k p L 1 ` L q X E A k ´ 1 p L 1 q only depends on the ﬁrst kN episodes.
Therefore the sample space and the event will be deﬁned on any trajectory with kN episodes.
For any τ , we know that ψ will output the k -th deployment policy solely based on the τ r 0 , k ´ 1 s and this map is deterministic (we use τ r i, j s to denote the iN ` 1 to jN episodes in τ ).
In other words, ψ will map τ r 0 , k ´ 1 s to a ﬁxed policy ψ p τ r 0 , k ´ 1 sq to deploy at the k -th time
[Equation]
We have the following equation for any I P M
[Equation]
Notice that this equality does not generally hold for probability distribution P M ,ψ .
Then we ﬁx τ r 0 , k ´ 1 s , such that τ r 0 , k ´ 1 s
P
E
[Equation]
p L 1 q .
We also ﬁx p i 1 , . .
i L 1 ´ 1 q , p i L
[Equation]
` L , . .
, i H ´ 2 , s H ´ 1 q and consider two instances
[Equation] [Equation]
and
I
[Equation] [Equation]
Therefore, we have that
P I 1 ,ψ p τ r 0 , k ´ 1 sq “ P I 2 ,ψ p τ r 0 , k ´ 1 sq (from the construction of I 1 , I 2 and the property of deterministic algorithm ψ ).
We use I p i 1 , . .
i L 1 ´ 1 , i L 1 ` L , . .
, i H ´ 2 , s H ´ 1 q to denote the instance class that has ﬁxed p i 1 , . .
, i L 1 ´ 1 q , p i L 1 ` L , . .
, i H ´ 2 , s H ´ 1 q , but different p i L 1 , . . .
i L 1 ` L ´ 1 q .
In addition, we use I p i 1 , . .
i L 1 ´ 1 q to denote the instance class that has ﬁxed p i 1 , . . .
i L 1 ´ 1 q , but different p i L 1 , . . .
i L 1 ` L ´ 1 q and p i
L 1 ` L , . .
, i H ´ 2 , s H ´ 1 q .
Since we have already ﬁxed τ r 0 , k ´ 1 s P E
A k ´ 1 p L 1 q here, ψ p τ r 0 , k ´ 1 sq is also ﬁxed (for all I P
I p i 1 , . . .
i L 1 ´ 1 , i L 1 ` L , . .
, i H ´ 2 , s H ´ 1 q ).
Also notice that we are considering the probability of N episodes τ 1 r k ´ 1 : k s .
Therefore, we can follow Claim 1 and deﬁne p p i, j, h q for 0 ď h ď L 1 ` L ´ 1 , which represents the probability of choosing action j at node i at level h under the k -th deployment policy.
In the k -th deployment, ψ draws N episodes, so the probability of executing ψ to reach any state s L 1 ` L at level L 1 ` L under instance I “ p i 1 , . .
, i H ´ 2 , s H ´ 1 q is
[Equation]
Following the same step in Eq (6) by summing over i L 1 , . . .
, i L 1 ` L ´ 1 and s L 1 ` L gives us ÿ
[Equation]
ÿ
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
ď N.
Now, we sum over all possible p i L 1 ` L , . .
, i H ´ 2 , s H ´ 1 q and take the average.
For any ﬁxed τ r 0 , k ´ 1 s P
E
A k ´ 1 p L 1 q we have
[Equation]
Moreover, summing over all τ r 0 , k ´ 1 s P E
A k ´ 1 p L 1 q , gives us i 1 , . .
, i L 1 ´
P M ,ψ p E k p L 1 ` L q X E A k ´ 1 p L 1 qq “| M | ÿ i 1 ,...,i L 1´ÿ
I P I p
i 1 ,...,i L 1´
[Equation]
ÿ
[Equation]
ÿ
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Equation]
[Table]
ÿ
[Table]
ÿ
[Table]
ÿ
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
ÿ
[Table]
[Equation]
In the second equality, we use P i 1 ,...,i L 1´ 1 ,ψ because for any ﬁxed τ r 0 , k ´ 1 s P E
A k ´ 1 p L 1 q and all
I P I p
i 1 , . .
i L 1 ´ 1 q , P i 1 ,...,i L 1´ 1 ,ψ p τ r 0 , k ´ 1 sq are the same.
Finally, we have
[Equation]
Proof of the Theorem If KL ď H ´ 2 , then applying Claim 2 for K ´ 1 times and applying
Claim 1 tells us
[Equation]
We can set L “ r log d NH s ` 1 and K ď
H ´ 2 2 r log d NH s .
Then for H ě 3 , we get KL ď H ´ 2 and
P M ,ψ p does not hit any state at level H ´
2 q ě p 1 ´
[Equation]
Let event F denote the event (a set of length KN episodes trajectories) that any state at level H ´is not hit.
Then we have P M ,ψ p F q ě 1 ´ 1 e .
We use I p i 1 , . .
, i H ´ 2 q to denote the instance class that has ﬁxed core states p i 1 , . .
, i H ´ 2 q but different optimal states s H ´ 1 .
Consider any ﬁxed τ P F .
Similar as the proof in the prior claims, by the property of deterministic algorithm, we can deﬁne p p i, j, h q for h “ H ´ 2 , which represents the probability of the output policy ψ τ p K ` 1 q under trajectory τ to choose action j at node i at level H ´ 2 .
Then we have ÿ
I
[Equation]
p
[Equation]
q chooses optimal state)
“ ÿ
[Equation]
“ ÿ
[Equation] [Equation]
Summing over τ P F gives us ÿ
[Equation]
P
[Equation]
p F X the output policy chooses optimal state q “ ÿ
[Equation]
ÿ
[Equation]
p K ` 1 q chooses optimal state) “ ÿ
[Equation]
ÿ
[Equation]
q chooses optimal state) “ ÿ
[Equation]
ÿ
[Equation]
q chooses optimal state) “ ÿ
[Equation]
In the second equality, we notice that for all instance I P I “ p i 1 , . .
, i H ´ 2 , s H ´ 1 q , P I,ψ p τ q are the same, so this probability distribution essentially depends on i 1 , . .
, i H ´
In the third inequality, we change the order of the summation.
Finally, summing over i 1 , . .
, i H ´ 2 and taking average yields that
[Equation]
optimal
[Equation]
does not depend on the optimal state)
[Equation]
Therefore, we get the probability of not choosing the optimal state is
P M ,ψ p the output policy does not choose the optimal state q ě P
[Equation]
p F X the output policy does not choose the optimal state q
[Equation]
the output policy chooses the optimal state q
[Equation]
From the construction, we know that any policy that does not choose optimal state (thus also does not choose the optimal action associated with the optimal state) is ε sub-optimal.
This implies that with probability at least 1 2 e , the output policy is at least ε sub-optimal.
˝
C D EPLOYMENT -E FFICIENT RL WITH D ETERMINISTIC P OLICIES AND GIVEN R EWARD F UNCTION C. A DDITIONAL N OTATIONS
In the appendix, we will frequently consider the MDP truncated at r h ď H , and we will use:
V πh p s | r h q “ E r r h ÿ h 1 “ h r h 1 p s h 1 , a h 1 q| s h “ s, π s ,
Q πh p s, a | r h q “ E r r h ÿ h 1 “ h r h 1 p s h 1 , a h 1 q| s h “ s, a h “ a, π s to denote the value function in truncated MDP for arbitrary h ď r h , and also extend the deﬁnition in
Section 2 to V ˚ h p¨| r h q , Q ˚ h p¨ , ¨| r h q , π ˚| r h for optimal policy setting and V ˚ h p¨ , r | r h q , Q ˚ h p¨ , ¨ , r | r h q , π ˚ r | h for reward-free setting.
C. A UXILIARY L EMMA
Lemma C.1 (Elliptical Potential Lemma; Lemma 26 of Agarwal et al. (2020b)) .
Consider a sequence of d ˆ d positive semi-deﬁnite matrices X 1 , ..., X T with max t
Tr p X t q ď 1 and deﬁne
M 0 “ λI, ..., M t “ M t ´ 1 ` X t .
Then
T ÿ t “
Tr p X t M ´ 1 t ´ 1 q ď p 1 ` 1 { λ q d log p 1 ` T { d q .
Lemma C.2 (Abbasi-yadkori et al.
Suppose A , B P R d ˆ d are two positive deﬁnite matrices satisfying A ľ B , then for any x P R d , we have:
[Equation]
Next, we prove a lemma to bridge between trace and determinant, which is crucial to prove our key technique in Lemma 4.2.
Lemma C.3.
[Bridge between Trace and Determinant] Consider a sequence of matrices
A 0 , A N , ..., A p K ´ 1 q N with A“ I and A kN “ A p k ´ 1 q N ` Φ k ´ 1 , where Φ k ´“ ř kNt “p k ´ 1 q N ` 1 φ t φ J t .
We have
[Equation]
Proof.
Consider a more general case, given matrix Y
ľ
I , we have the following inequality
[Equation]
By replacing Y with I ` A ´ 1 X in the above inequality, we have:
[Equation]
By assigning A “ A p k ´ 1 q N and X “ Φ k ´ 1 , and applying Lemma C.2, we have:
[Equation] [Equation]
which ﬁnished the proof.
˝
Lemma 4.2.
[Batched Finite Sample Elliptical Potential Lemma] Consider a sequence of matrices
A 0 , A N , ..., A p K ´ 1 q N P R d ˆ d with A 0 “ I d ˆ d and A kN “ A p k ´ 1 q N ` Φ k ´ 1 , where Φ k ´ 1 “ ř kNt “p k ´ 1 q N ` 1 φ t φ J t and max t ď KN } φ t } ď 1 .
We deﬁne:
K ` : “
[Equation]
Nε ) .
For arbitrary ε ă 1 , and arbitrary c K ě 2 , if K “ c K dH ` 1 , by choosing N ě c ´
[Equation]
q ¯ 1 cK ´ 1 , where c is an absolute constant independent with c K , d, H, ε , we have | K ` | ď c K d ă K { H.
Proof.
Suppose we have Tr p A ´ 1 p k ´ 1 q N Φ k ´ 1 q ě Nε , by applying Lemma C.3 we must have:
[Equation]
which implies that,
Nε
[Equation]
Therefore,
[Equation]
which implies that, conditioning on N ě dε log p 1 ` KN { d q , we have:
[Equation]
Now, we are interested in ﬁnd the mimimum N , under the constraint that |
K ` | ď c K d .
To solve this problem, we ﬁrst choose an arbitrary p ď c K , and ﬁnd a N such that,
[Equation]
In order to guarantee the above, we need:
[Equation]
The ﬁrst constraint can be satisﬁed easily with N ě c 1 dε log dHε for some constant c 1 .
Since usually
KN { d ą 1 , the second constraint can be directly satisﬁed if:
[Equation]
Recall K “ c K dH ` 1 , it can be satisﬁed by choosing
[Equation]
[Equation]
where c 2 is an absolute constant.
Therefore, we can ﬁnd an absolute number c such that,
[Equation]
to make sure that
[Equation]
Since in Eq.(7), it’s required that 1 {p p ´ 1 q ă 8 , we should constraint that p ą 1 and therefore, c K ě 2 .
Because the dependence of d, H, 1 ε , log dHε are decreasing as p increases, by assigning p “ c K and 1 ă p ď c K , N will be minimized when p “ c K .
Then, we ﬁnished the proof.
˝ C. A NALYSIS FOR A LGORITHMS
Next, we will use the above lemma to bound the difference between J p π K q and J p π ˚ q .
We ﬁrst prove a lemma similar to Lemma B.3 in (Jin et al., 2019) and Lemma A.1 in (Wang et al., 2020b).
Lemma C.4 (Concentration Lemma) .
We use E 1 to denote the event that, when running Algorithm 1, the following inequality holds for all k P r K s and h P r h k s and arbitrary V kh ` 1 occurs in Alg 1.
››› k ´ 1 ÿ
[Equation]
Under Assumption A, there exists some absolute constant c ě 0 , such that P p E 1 q ě 1 ´ δ { 2 .
Proof.
The proof is almost identical to Lemma B.3 in (Jin et al., 2019), so we omit it here.
The only difference is that we have an inner summation from n “ 1 to N
and we truncate the horizon at h k in iteration k .
˝
Lemma C.5 (Overestimation) .
On the event E 1 in Lemma C.4, which holds with probability 1 ´ δ { 2 , for all k P r K s and n P r N s ,
[Equation]
where recall that V k 1 is the function computed at iteration k in Alg.1 and
V ˚ 1 p¨| h
k q “
E r ř h k h “ 1 r h p s h , a h q| π ˚r 1: h k s s denote the optimal value function in the MDP truncated at layer h k and π ˚r 1: h k s is the optimal policy in the truncated MDP.
Besides, we also have:
[Equation]
Proof.
First of all, by applying Lemma C.4 above, after a similar discussion to the proof of Lemma 3.1 in (Wang et al., 2020b), we can show that
[Equation]
and the overestimation
[Equation]
As a result,
[Equation]
[Table]
[Equation] [Equation]
where in the second inequality
[Equation]
we use the following fact
[Equation]
| s 1 , a 1 q V k 2 p s 2 q “ min tp w k 1 q J φ p s 1 , a 1 q ´ ÿ s 2 P S
P 1 p s 2 | s 1 , a 1 q V k 2 p s 2 q , H ´ r k 1 p s 1 , a 1 q ´ u k 1 p s 1 , a 1 q ´ ÿ s 2 P S
P 1 p s 2 | s 1 , a 1 q V k 2 p s 2 qu ď min t β } φ p s 1 , a 1 q} p Λ k 1 q ´ 1 , H u “ u k 1 p s 1 , a 1 q ˝
Now we are ready to prove the following theorem restated from Theorem 4.1 in a more detailed version, where we include the guarantees during the execution of the algorithm.
Theorem C.6.
[Deployment Complexity] For arbitrary ε, δ ą 0 , and arbitrary c K ě 2 , as long as
N ě c
[Equation]
1 , where c is an absolute constant and independent with c K , d, H, ε, δ , by choosing
[Equation]
Algorithm 1 will terminate at iteration k H ď K and return us a policy π k H , and with probability 1 ´ δ , (1) E s 1 „ d 1 r V ˚ 1 p s 1 q ´ V π kHp s 1 qs ď ε .
(2) for each h P r H ´ 1 s , there exists an iteration k h , such that h k h “ h
but h k h ` 1 “ h ` 1 , and π k h is an ε -optimal policy for the MDP truncated at step h ;
Proof.
As stated in the theorem, we use k h to denote the number of deployment after which the algorithm switch the exploration from layer h to layer h ` 1 , i.e. h k h “ h and h k h ` 1 “ h ` 1 .
According to the deﬁnition and the algorithm, we must have ∆ k h ď εh kh 2 H , and for arbitrary k h ´ 1 ` 1 ď k ď k h ´ 1 , ∆ k ě εh k 2 H (if k h ´ 1 ` 1 ą k h ´ 1 , then it means ∆ k h ´ 1 ` 1 is small enough and the algorithm directly switch the exploration to the next layer, and we can skip the discussion below ).
Therefore, for arbitrary k h ´ 1 ` 1 ď k ď k h ´ 1 , during the k -th deployment, there exists h P r h k s , such that,
[Equation]
where the second inequality is because the average is less than the maximum.
The above implies that
[Equation]
According to Lemma 4.2, there exists constant c, c 1 , such that by choosing N according to Eq.(10) below, the event in Eq.(9) will not happen more than dc K times at each layer
h P r h k s .
[Equation]
Recall that ε ă 1 and the covariance matrices in each layer is initialized by I d ˆ d .
Therefore, at the ﬁrst deployment, although the computation of π 1 does not consider the layers h ě 2 , Eq.(9) happens in each layer
h P r H s .
We use ζ p k, j q to denote the total number of times events in Eq.(9) happens for layer j previous to deployment k , as a result,
[Equation]
where we minus h ´ 1 because such event must happen at the ﬁrst deployment for each h P r H
s
and we should remove the repeated computation; and we add another h back is because there are h times we waste the samples (i.e. for those k such that ∆ k ă εh k 2 H ).
Therefore, we must have k H ď c K dH ` 1 “ K .
Moreover, because at iteration k “ k h , we have ∆ k h ď ε { 2 , according to Hoeffding inequality, with probability 1 ´ δ { 2 , for each deployment k , we must have:
[Equation]
Therefore, by choosing
[Equation]
we must have,
[Equation]
s
Therefore, after a combination of Eq.(10) and Eq.(12)
[Equation]
we can conclude that, for arbitrary c K ě 2 , there exists absolute constant c , such that by choosing
[Equation]
the algorithm will stop at k H ď K , and with probability 1 ´ δ (on the event of E 1 in C.4 and the
Hoeffding inequality above), we must have:
[Equation]
and an additional beneﬁts that for each h P r H ´ 1 s , π k h is an ε -optimal policy at the MDP truncated at h step, or equivalently,
[Equation]
˝
C. A DDITIONAL S AFETY G UARANTEE B ROUGHT WITH L AYER - BY -L AYER S TRATEGY
The layer-by-layer strategy brings another advantage that, if we ﬁnish the exploration of the ﬁrst h layers, based on the samples collected so far, we can obtain a policy p π | h , which is an ε -optimal in the MDP truncated at step h , or equivalently:
[Equation]
| h q ě H
[Equation]
O p
[Equation]
q , @
h P r H s
We formally state these guarantees in Theorem C.6 (a detailed version of Theorem 4.1), Theorem
D.4 and Theorem E.9 (the formal version of Theorem 4.4).
Such a property may be valuable in certain application scenarios.
For example, in “Safe DE-RL”, which we will discuss in Appendix
F, p π | h can be used as the pessimistic policy in Algorithm 7 and guarantee the monotonic policy improvement criterion.
Besides, in some real-world settings, we may hope to maintain a sub-optimal but gradually improving policy before we complete the execution of the entire algorithm.
If we replace Line 7-8 in LSVI-UCB (Algorithm 1) in Jin et al.
with Line 13-18 in our
Algorithm 1, the similar analysis can be done based on Lemma 4.2, and the same Θ p dH q deployment complexity can be derived.
However, the direct extension based on LSVI-UCB does not have the above safety guarantee.
It is only guaranteed to return a near-optimal policy after K “ Θ p dH q deployments, but if we interrupt the algorithm after some k ă K deployments, there is no guarantee about what the best possible policy would be based on the data collected so far.
D R EWARD -F REE D EPLOYMENT -E FFICIENT RL WITH D ETERMINISTIC P OLICIES D. A LGORITHM
Similar to other algorithms in reward-free setting (Wang et al., 2020b; Jin et al., 2020), our algorithm includes an “Exploration Phase” to uniformly explore the entire MDP, and a “Planning Phase” to return near-optimal policy given an arbitrary reward function.
The crucial part is to collect a wellcovered dataset in the online “exploration phase”, which is sufﬁcient for the batch RL algorithm
(Antos et al., 2008; Munos & Szepesv ´ ari, 2008; Chen & Jiang, 2019) in the ofﬂine “planning phase” to work.
Our algorithm in Alg.3 and Alg.4 is based on (Wang et al., 2020b) and the layer-by-layer strategy.
The main difference with Algorithm 1 is in two-folds.
First, similar to (Wang et al., 2020b), we replace the reward function with 1 { H of the bonus term.
Secondly, we use a smaller threshold for ∆ k comparing with Algorithm 1.
D. A NALYSIS FOR A LG 3 AND A LG
We ﬁrst show a lemma adapted from Lemma C.4 for Alg 3.
Since the proof is similar, we omit it here.
Lemma D.1 (Concentration for DE-RL in Reward-Free Setting) .
We use E 2 to denote the event that, when running Algorithm 3, the following inequality holds for all k P r K s and h P r h k s and all
V “ V kh ` 1 occurs in Alg 3 or V “ V h occurs in Alg 4:
[Equation]
Under Assumption A, there exists some absolute constant c ě 0 , such that P p E 2 q ě 1 ´ δ { 2 .
Proof.
The proof is almost identical to Lemma 3.1 in (Wang et al., 2020b), so we omit it here.
The only difference is that we have an inner summation from n “ 1 to N
and we truncate the horizon at h k in iteration k .
˝
Next, we prove a lemma simlar to Lemma C.5 based on Lemma D.1.
Lemma D.2 (Overestimation) .
On the event E 2 in Lemma D.1, which holds with probability 1 ´ δ { 2 , in Algorithm 3, for all k P r K s and n P r N s ,
[Equation]
and
[Equation]
Proof.
We ﬁrst prove the overestimation inequality.
1 Input : Failure probability δ ą 0 , and target accuracy ε ą 0 , β Ð c β ¨ dH a log p dHδ ´ 1 ε ´ 1 q for some c β ą 0 , total number of deployments K , batch size N 2 Initialize h 1 “3 D 1 “ tu , D 2 “ tu , ..., D H “ tu 4 for k “ 1 , 2 , ..., K do
Q kh k ` 1 p¨ , ¨q Ð 0 and V kh k ` 1 p¨q “ for h “ h k , h k ´ 1 , ..., 1 do Λ kh Ð
I
` ř kτ “ 1 ř Nn “ 1 φ τnh p φ τnh q J u kh p¨ , ¨q Ð min t β ¨ b φ p¨ , ¨q J p Λ kh q ´ 1 φ p¨ , ¨q , H u
Deﬁne the exploration-driven reward function r kh p¨ , ¨q Ð u kh p¨ , ¨q{ H w kh Ð p Λ kh q ´
ř k ´ 1 τ “ 1 ř Nn “ 1 φ τnh ¨ V kh ` 1 p s τnh ` 1 q
Q kh p¨ , ¨q Ð min tp w kh q J φ p¨ , ¨q ` r kh p¨ , ¨q ` u kh p¨ , ¨q , H u and V kh p¨q “ max a P A Q kh p¨ , a q π kh p¨q Ð arg max a P A Q kh p¨ , a q end
Deﬁne π k “ π k 1 ˝
π k 2 ˝ ...
π kh k
˝ unif r h k ` 1: H s for n “ 1 , ..., N do
Receive initial state s kn 1 „ d for h “ 1 , 2 , ..., H do
Take action a knh Ð π k p s knh q and observe s knh ` 1 „ P h p s kh , a kh q
D h “ D h Ť tp s knh , a knh qu end end
Compute ∆ k Ð 2 βN
ř Nn “ 1 ř h k h “b φ p s knh , a knh q J p Λ kh q ´ 1 φ p s knh , a knh q .
if ∆ k ă εh k p 4 H ` 2 q H then if h k “ H then return D “ t D 1 , D 2 , ..., D H u ; else h k Ð h k ` 1 ; end 27 end
1 Input : Horizon length r h ; Dataset D “ tp s knh , a knh q k,n,
h Pr K sˆr N sˆr r h s u , reward function r “ t r h u
h Pr r h s 2 Q r h ` 1 p¨ , ¨q Ð 0 and V r h ` 1 p¨q Ð3 for h “ r h, r h ´ 1 , ..., 1 do Λ h Ð I ` ř Kτ “ 1 ř Nn “ 1 φ p s τnh , a τnh q φ p s τnh , a τnh q J
Let u planh p¨ , ¨q Ð min t β a φ p¨ , ¨q J p Λ h q ´ 1 φ p¨ , ¨q , r h u w h Ð p Λ h q ´
ř
Kτ
“ 1 φ p s τnh , a τnh q ¨ V h ` 1 p s τnh ` 1 , a q
Q h p¨ , ¨q Ð min t w J h φ p¨ , ¨q ` r h p¨ , ¨q ` u planh p¨ , ¨q , r h u and V h p¨q “ max a P A Q h p¨ , a q π h p¨q Ð arg max a P A Q h p¨ , a q 9 end 10 return π r | r h “ t π h u h Pr r h
[Equation]
Overestimation First of all, similar to the proof of Lemma 3.1 in (Wang et al., 2020b), on the event of E 2 deﬁned in Lemma C.4, which holds with probability 1 ´ δ { 2 , we have: | φ p s, a
[Equation]
Then, we can use induction to show the overestimation.
For h “ h k ` 1 , we have:
[Equation]
Suppose for some h P r h k s , we have
V
[Equation]
Then, @ s P S , we have
[Equation]
where in the last inequality, we apply Eq.(14).
Relationship between V k 1 p¨q and V π k p¨ , r k q
[Equation]
where in the ﬁrst inequality, we add and subtract ř s 2 P S P 1 p s
| s 1 , a 1 q V k 2 p s 2 q , and in the second inequality, we use the following fact
[Equation]
˝
Next, we provide some analysis for Algorithm 4, which will help us to understand what we want to do in Algorithm
Lemma D.3.
On the event E 2 in Lemma D.1, which holds with probability 1 ´ δ { 2 , if we assign r h “ h k in Algorithm 4 and assign D to be the samples collected till deployment k , i.e.
D “ tp s knh , a knh q k,n,h Pr K sˆr N sˆr h
k s u , then for arbitrary reward function r satisfying the linear Assumption A, the policy π r | r h returned by Alg 4 would satisfy:
[Equation]
where r plan : “ u plan { r h .
Proof.
By applying the similar technique in the analysis of E s 1 „ d 1 r V k 1 p s 1 q ´ V π k p s 1 , r k | r h qs in
Lemma D.2 after replacing r k with r , we have:
[Equation]
where p V 1 denotes the value function returned by Alg 4 Besides,
[Equation]
then, we ﬁnish the proof.
˝
From Eq.(15) in Lemma D.3, we can see that, after exploring with Algorithm 3, the sub-optimality gap between π ˚ and π returned by Alg.4 can be bounded by the value of the optimal policy w.r.t.
r K , which we will further bound in the next theorem.
Now we are ready to prove the main theorem.
Theorem D.4.
For arbitrary ε, δ ą 0 , by assigning K “ c K dH ` 1 for some c K ě 2 , as long as
[Equation]
where c is an absolute constant and independent with c K , d, H, ε, δ , then, Alg 3 will terminate at iteration k H ď K and return us a dataset D “ t D 1 , D 2 , ..., D H u , such that given arbitrary reward function r satisfying Assumption A, by running Alg 4 with D and r , with probability 1 ´ δ , we can obtain a policy π r satisfying E s „ d 1 r V ˚ 1 p s, r q ´ V π p s, r qs ď ε .
Moreover, for each h P r H ´ 1 s , there exists iteration k h , such that h k h “ h but h k h ` 1 “ h ` 1 , and if we run Alg 4 with the reward function r and the dataset Alg 3 has collected till k “ k h , we can obtain a policy π r | h , which is an ε -optimal policy for MDP truncated at step h .
Proof.
The proof is similar to Theorem 4.1.
As stated in theorem, we use k h to denote the number of deployment when the algorithm switch the exploration from layer h to layer h ` 1 , i.e. h k h “ h and h k h ` 1 “ h ` 1 .
According to the deﬁnition and the algorithm, we must have ∆ k h ď εh kh p 4 H ` 2 q H , and for arbitrary k h ´ 1 ` 1 ď k ď k h ´ 1 , we must have ∆ k ě εh k p 4 H ` 2 q H
(if k h ´ 1 ` 1 ą k
[Equation]
, then it means ∆ k h ´ 1 ` 1 is small enough and the algorithm directly switch the exploration to the next layer, and we can skip the discussion below).
Therefore, for arbitrary k h ´ 1 ` 1 ď k ď k h ´ 1 , during the k -th deployment, there exists h P r h k s , such that,
[Equation]
which implies that
[Equation]
According to Lemma 4.2, there exists an absolute constant c , for arbitrary ε ă 1 , by choosing N according to Eq.(18) below, the events in Eq.(17) will not happen more than c K d times at each layer h P r H s .
[Equation]
We use ζ p k, j q to denote the total number of times Eq.(9) happens for layer j till deployment k h .
With a similar discussion as Theorem 4.1, we have:
[Equation]
Moreover, we must have ∆ k h ď ε 4 H ` 2 for each h P r H s , and according to Hoeffding inequality, with probability 1 ´ δ { 2 , for each step k , we must have
[Equation]
Therefore, by choosing
[Equation]
we have,
[Equation]
For arbitrary h P r H s , in Algorithm 4, if we assign r h “ h and D “ tp s knh , a knh q k,n,h Pr k h sˆr
N sˆr
h s u , note that r plan “ r k h , by applying Lemma D.2 and Lemma D.3 we have:
[Equation]
Therefore, after a combination of Eq.(18) and Eq.(19), we can conclude that, for arbitrary c K ě 2 , there exists absolute constant c , such that by choosing
[Equation]
Alg 3 will terminate at k H ď K , and with probability 1 ´ δ (on the event in Lemma D. and Hoeffding inequality above), for each h P r H s , if we feed Alg 4 with r h “ h , D
“ tp s knh , a knh q k,n,h Pr k h sˆr N sˆr
h s u and arbitrary linear reward function r , the policy π r | h returned by Alg 4 should satisfy:
[Equation]
˝
E DE-RL WITH A RBITRARY D EPLOYED P OLICIES
In the proof for this section, without loss of generality, we assume the initial state is ﬁxed, which will makes the notation and derivation simpler without trivialize the results.
For the case where initial state is sampled from some ﬁxed distribution, our algorithms and results can be extended simply by considering the concentration error related to the initial state distribution.
1 Input : Time step h ; Dataset in previous steps t D 1 , ..., D h ´ 1 u ; Unregularized Covariance
Matrices t Σ 1 , ... Σ h ´ 1 u ; Bonus factor β ; Matrix to construct reward function Σ R ; Discretize resolution ε 0 ď
[Table]
R p¨ ,
[Table]
Ð a φ p¨ ,
[Table]
I
[Table] [Table]
Ð Discretization pp 2 I `
[Table] [Table]
, ¯ R p¨ ,
[Table]
Ð a φ p¨ , ¨q J Z h φ p¨ , ¨q
[Table]
p¨ , ¨q “ R p¨ , ¨q ,
V h p¨q “ max
[Table]
Q
[Table]
p¨ , a
[Table] [Table]
[Table] [Table]
R p¨ , ¨q , ¯ V
[Table]
p¨q “
[Table]
p¨ , a
[Table]
“ arg
[Table] [Table]
for
[Table] [Table]
[Table] [Table]
do
[Table]
ř p
[Table]
[Table] [Table]
, a
[Table] [Table]
[Table] [Table]
p¨ ,
[Table]
min
[Table] [Table]
[Table] [Table]
, a
[Table] [Table]
[Table] [Table]
[Table]
Ð min t ¯ w J r h φ p¨ , ¨q ` ¯ u r h , 1 u , ¯ V r h p¨q Ð max a ¯ Q r h p¨ , a q ¯ π r h p¨q Ð arg max a ¯ Q r h p¨ , a q 14 end 15 return V 1 p s 1 q , ¯ π : “ ¯ π 1 ˝
¯ π 2 ˝ ...
¯ π h
1 Input : Time step h ; Dataset in previous steps t D 1 , ...,
D h ´ 1 u ; Covariance Matrices t Σ 1 , ...
Σ h ´ 1 u ; Deterministic Policy to evaluate ¯ π “ t ¯ π 1 , ¯ π 2 , ..., ¯ π h u 2 Initialize a zero matrix r Λ πh “ O 3 for i “ 1 , 2 , ..., d do for j “ i, i ` 1 , ..., d do
Deﬁne r R ij , such that, r R ijh p¨ , ¨q “ 1 ` φ i p¨ , ¨q φ j p¨ , ¨qand r R ij r h “ 0 for all r h P r h ´ 1 s ; p Q ¯ πh p¨ , ¨q “ r R ijh p¨ , ¨q , p V ¯ πh p¨q “ p Q ¯ πh p¨ , ¯ π h p¨qq for r h “ h ´ 1 , ..., 1 do p w ¯ π r h Ð Σ ´ 1 r h
ř p s r h ,a r h ,s r h ` 1 qP D
r h φ p s t , a t q ¨ p V
¯ π r h ` 1 p s r h ` 1 q p Q ¯ π r h p¨ , ¨q Ð min t r R ij r h p¨ , ¨q ` p p w ¯
π r h q J φ p¨ , ¨q , 1 u p V
¯
π r h p¨q “ p Q ¯ π r h p¨ , ¯ π r h p¨qq end p
r
Λ ¯ π r h q ij Ð p V
¯ π 1 p s 1 q ; p r Λ ¯ π r h q ji Ð p V
¯ π 1 p s 1 q end 14 end 15 p Λ ¯ π r h “ 2 p
Λ π r h q ´16 return p Λ π h
E. A LGORITHMS
We ﬁrst introduce the deﬁnition for Discretization function:
Deﬁnition E.1 ( Discretization function) .
Given vector w “ p w 1 , w 2 , ..., w d q J P R d or matrix Σ “ p Σ ij q i,j Pr d
s P R d ˆ d as input, we have:
[Equation]
where r ¨ s is the ceiling function.
In Algorithm 6, we are trying to estimate the expected covariance matrix under policy ¯ π by policy evaluation.
The basic idea is that, the expected covariance matrix can be represented by:
[Equation]
where we use p a ij q ij to denote a matrix whose element indexed by i in row and j in column is a ij .
In another word, the element in the covariance matrix indexed by ij is equal to the value function of policy ¯ π with R ijh p s h , a h q : “ φ i p s h , a h q φ j p s h , a h q as reward function at the last layer (and use zero reward in previous layers), where φ i denotes the i -th elements of vector φ .
Because the techniques rely on the reward is non-negative and bounded in r 0 , 1 s , by leveraging the fact that | φ i p¨ , ¨q| ď } φ p¨ , ¨q} ď 1 , we shift and scale R ij to obtain r R ij and use it for policy evaluation.
In Alg 5, we maintance two Q functions Q h and ¯ Q h .
The learning of Q h is based on LSVI-UCB, while ¯ Q h is a “discretized version” for Q h computed by discretizing w h , β 2 Σ ´ 1 h (or Σ
´ 1 R at layer h ) elementwisely with resolution ε 0 , and ¯ Q h will be used to compute ¯ π h for deployment.
The main reason why we discretize Q h is to make sure the number of greedy policies ¯ π is bounded, so that we can use union bound and upper bound the error when using Alg6 to estimate the covariance matrix.
In Section E.5, we will analyze the error resulting from discretization, and we will upper bound the estimation error Algorithm 6.
[Equation]
AND ε 0 -C OVER
We ﬁrst introduce some useful function classes and their ε 0 -cover.
Notation for Value Function Classes and Policy Classes
We ﬁrst introduce some new notations for value and policy classes.
Similar to Eq.(6) in (Jin et al., 2019), we deﬁne the greedy value function class
[Equation]
and the Q function class:
[Equation]
Besides, suppose we have a deterministic policy class Π with ﬁnite candidates (i.e. | Π | ď 8 ), we use V L,B ˆ Π to denote:
[Equation]
Recall that in Alg.6, we will use a special reward function, and we need to consider it in the union bound.
We denote:
[Equation]
and easy to check |
V φ ˆ Π | “ d 2 | Π | .
Moreover, if we have a Q function class Q , we will use Π Q to denote the class of greedy policies induced from Q , i.e.
[Equation]
Discretization with Resolution ε
In the following, we will use C w,L,ε 0 to denote the ε 0 -cover for w P R d with } w } ď L , concretely,
[Equation]
where r ¨ s is the ceiling function.
Similarly, we will use C Σ ,B,ε 0 to denote the ε 0 -cover for matrix Σ P R d ˆ d with max i,j | Σ ij | ď B
[Equation]
Easy to check that:
[Equation]
Recall the deﬁnition of Discretize function in Def.
E.1, easy to check that: } Discretize p w, ε 0 q ´ w } ď dε 0 , } Discretize p Σ , ε 0 q ´ Σ } ď } Discretize p Σ , ε 0 q ´ Σ } F ď dεε 0 -cover
Before we introduce our notations for ε 0 -net, we ﬁrst show a useful lemma:
Lemma E.2.
For arbitrary w, Σ , denote ¯ w “ Discretize p w, ε 0 2 d q and ¯Σ “ Discretize p Σ , ε 02 4 d q .
Consider the following two functions and their greedy policies, where } φ p¨ , ¨q} ď
[Equation]
then we have:
[Equation]
Proof.
After similar derivation as Eq.(28) in (Jin et al., 2019), we can show that
[Equation]
Because π and ¯ π are greedy policies, we have:
[Equation]
˝
Now, we consider the following Q function class and V function class,
[Equation]
¯
[Equation]
L
[Equation] [Equation]
based on Lemma E.2, and another important fact that max i,j | a ij | ď } A } F ď d } A } , we know that ¯
Q L,B,ε 0 is an ε 0 -cover of Q L,B , i.e. for arbitrary Q P Q L,B , there exists ¯ Q P ¯ Q L,B,ε 0 , such that } Q ´ ¯ Q } ď ε 0 .
Similarly, ¯ V ˚ L,B,ε 0 is also an ε 0 -cover of V ˚ L,B .
Besides, we will use Π ¯ Q L,B,ε 0 to denote the collection of greedy policy induced from elements in ¯
Q L,B,ε 0 .
We also deﬁne ¯ V L,B,ε 0 ˆ Π , which is an ε 0 cover for V L,B ˆ Π .
¯ V L,B,ε 0 ˆ Π : “ t V | V p¨q “ min t φ p¨ , π p¨qq J w ` b φ p¨ , π p¨qq J Σ φ p¨ , π p¨qq , 1 u , w P C w,L, ε 02 d , Σ P C Σ ,dB, ε 024 d , π P Π u
Obviously, | ¯ V L,B,ε 0 ˆ Π | “ | Π | ¨ | C w,L, ε 02 d | ¨ | C Σ ,dB, ε 024 d | .
Besides, because V φ ˆ Π is already a ﬁnite set, it is an ε 0 -cover of itself.
E. C ONSTRAINTS IN A DVANCE
Induction Condition Related to Accumulative Error Recall the induction condition in 4.5, and we restate it here.
Condition 4.5.
[Induction Condition] Suppose after h ´ 1 deployments, we have the following induction condition for some ξ ă 1 { d , which will be determined later:
[Equation]
Constraints for the Validity of the Algorithm Besides, in order to make sure the algorithm can run successfully, we add the following constaints:
[Equation]
where constraint (22) for Z h is to make sure the reward ¯ R locates in r 0 , 1 s interval.
Recall the deﬁnition of Σ r h “ I `
[Equation]
therefore,
[Equation]
According to Lemma E.11, to make sure Z r h ľ 0 , we need the following constraint on ε
[Equation]
which is equivalent to ε 0 ď β {?
N ` 1 .
As for Z h , the constraint is equivalent to:
[Equation]
and can be rewritten to
[Equation]
C ONCENTRATION B OUND
Based on the notations above, we are already to claim that:
Claim 3.
By choosing L “ ? dN, B “ β 2 ` dε 0 for
some ε 0 ď 1 { d , during the running of Algorithm
• In Alg 5, for each h P r H s and r h P r h s , and the ¯ Q r h and ¯ π r h generated while running the algorithm, we must have ¯ Q r h P ¯ Q L,B,ε 0 and ¯ π r h P Π ¯ Q L,B,ε 0 .
Besides, Q r h P Q L,B and V r h P V ˚ L,B • In Alg 6, for all r h P r h s , we have ¯ π r h P Π ¯ Q L,B,ε 0 , and for arbitrary value function p V ¯ π r h generated in it, we have p V ¯ π r h P p V L,
B ˆ Π ¯ Q L,B,ε 0 q Y p V
φ ˆ Π ¯ Q L,B,ε 0 q
and therefore, there exists V P
p ¯ V L,B,ε
ˆ Π ¯ Q L,B,ε 0 q Y p V
φ ˆ Π ¯ Q L,B,ε 0 q such that } V ´ p V ¯ π r h } ď ε
Proof.
Algorithm 5 : First we bound the norm of the weights w h in Algorithm 5.
For arbitrary v P R d and } v } “ 1 , we have:
[Equation]
ď
[Equation]
p s r h ,a r h ,s r h ` 1 qP D r h v J Σ ´ 1 r h v
||
ÿ p s r h ,a r h ,s r h ` 1 qP D
r h φ p s r h , a r h q J Σ ´ 1 r h φ p s r h , a r h q|
[Equation]
therefore, } w r h } ď ? dN .
Besides, according to Lemma E.11 and constraint (24), we have:
[Equation]
Recall B “ β 2 ` dε 0 and β ą 1 , the claim about Alg 5 is true.
Algorithm 6 : The discussion about the value range of p w ¯ π r h is similar to above.
Therefore, all the value functions occurred in the previous h ´ 1 layers would belong to V L,B ˆ Π , except that the last layer should belong to V φ ˆ Π .
Besides, since Alg 6 is only used to estimate the policies returned by Alg 5, we should have Π “ Π ¯ Q L,B,ε 0 .
As a result, the claim for Alg 6 is correct.
˝
Recall Lemma D.4 from (Jin et al., 2019), which holds for arbitrary V with covering number N εand sup s | V p s q| ď H .
Next, we state a slightly generalized version by replacing sup s | V p s q| ď H with sup s | V p s q| ď 1 , since in our Alg. 5 and Alg.
6 V max “ 1 .
This is also the main reason why we only need the coefﬁcient of the bonus term β “ r O p d q instead of r O p dH q in Alg.
5 so that we can achieve better dependence on H .
Lemma E.3 ((Revised) Lemma D.4 in (Jin et al., 2019)) .
Let t s τ u 8 τ “ 1 , be a stochastic process on state space S with corresponding ﬁltration t F τ u 8 τ “ 1 .
Let t
φ
τ u 8 τ “ 0 be an R d -valued stochastic process where φ τ P F τ ´ 1 and } φ τ } ď 1 .
Let Λ t “ λI ` ř tτ “ 1 φ τ φ J τ .
Then for any δ ą 0 , with probability at least 1 ´ δ , for all t ě 0 , and any V P V so that sup s | V p s q| ď 1 , we have:
[Equation]
λ where N ε 0 is the ε 0 -covering number of V w.r.t.
the distance dist p V, V 1 q “ sup s | V p s q ´ V p s q 1 |
Now we are ready to prove the main concentration result for Algorithm 2:
Theorem E.4.
Consider value function class V : “ V ˚ L,B Y p V L,B ˆ Π ¯ Q L,B,ε 0 q Y p V
φ ˆ Π ¯ Q L,B,ε 0 q with L “ 1 , B “ β 2 ` dε 0 .
According to Claim 3, V covers all possible value functions occurs when running Alg 2.
We use E to denote the even that the following inequality holds for arbitrary
V P V and arbitrary k P r K s “ r H s , h P r H s :
[Equation]
As long as
[Equation]
there exists some constant c , such that P p E q ě 1 ´ δ { 2 .
Proof.
We consider the value function class:
[Equation]
and we have an ε 0 -cover for it, which we denote as:
[Equation]
Besides
[Equation]
there exists c 1 ą
[Equation]
s
[Equation]
t.
[Equation] [Equation]
By plugging into Lemma E.3 and considering the union bound over k P r K s and h P r H s (note that
K “ H ), we have with probability 1 ´ δ { 2 ,
[Equation]
When ε 0 ď 1 N , the ﬁrst term will dominate, and there must exists c such that
[Equation]
E.
[Equation]
B IAS A NALYSIS
Lemma E.5 (Overestimation in Alg 5) .
Suppose we choose
[Equation]
(26) for some c β ą 0 .
During the running of Alg 2, on the condition 4.5 and on the event of E in Theorem
E.4, which holds with probability 1 ´ δ { 2 , for arbitrary h P r H s and r h ď h ´ 1 , the parameter w r h and value function V r h ` 1 occurs in Algorithm 5 should satisfy:
[Equation]
and
[Equation]
Proof.
The proof is mainly based on Theorem E.4, and the steps are similar to Lemma B.3 in (Jin et al., 2019) and Lemma 3.1 in (Wang et al., 2020b) and we omit here.
˝
Lemma E.6 (Bias Accumulation in Alg 5) .
On the induction condition 4.5 and on the events in
Theorem E.4 which holds with probability 1 ´ δ { 2 , if
[Equation]
(27) in Algorithm 5, for arbitrary ¯ R generated, we have:
[Equation]
where recall that we use V π p s ; ¯ R q to denote the value function with ¯ R as reward function.
Proof.
We will use π h p¨q : “ arg max a Q h p¨ , a q to denote the optimal policy w.r.t.
the Q function without discretization, although we do not deploy it in practice.
According to Lemma E.2, we should have max s,h | Q h p s, π p s qq
´ Q h p s, ¯ π p s qq| ď
ε 0 for arbitrary h P r h s .
Recall that ¯ π “ ¯ π 1 ˝ ¯ π 2 ...
˝
¯ π h
V ˚
[Equation]
p
[Equation]
(Lemma E.5; Overestimation)
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Equation]
h ´
[Equation]
h ´
[Equation]
¯
[Equation]
r h ´ 1 ÿ r h “
[Equation]
} φ p s r h , a r h q} Σ ´ 1 r h s ` 2 p h ´ 1 q ε 0 ` E s h ,a h „ ¯ π r V h p s h q ´ V ¯ πh p s h qs
[Equation]
(Condition 4.5)
˝
Lemma E.7 (Bias of Linear Regression in Alg 6) .
During the running of Alg 2, on the event of E in
Theorem E.4, which holds with probability 1 ´ δ { 2 , for arbitrary h P r H s , r h P r h ´ 1 s , and arbitrary π , p w π r h and p V π r h ` 1 occurs in Alg 6, we have:
[Equation]
where β is the same as Lemma E.5.
The proofs for the above Lemma is based on Theorem E.4 and Claim 3 and is similar to Lemma 3.
in (Wang et al., 2020b), so we omit it here.
Lemma E.8 (Policy Evaluation Error in Alg 6) .
During the running of Algorithm 2, on the events of E in Theorem E.4, which holds with probability 1 ´ δ { 2 , and on the induction condition in 4.5, for arbitrary h P r H s and i, j P r d s , and arbitrary policy π and their evaluation results p V π Algorithm 6, we have:
[Equation]
where we use r R ij to denote the reward function used in Algorithm 6.
Proof.
As a result of Lemma.
E.7 , for arbitrary r R ij , we have:
[Equation]
V π 1 p s 1 ; r R ij q| “| p
Q π 1 p s 1 , a 1 q ´ Q π 1 p s 1 , a 1 ; r R ij q| “| φ p s 1 , a 1 q J p w π 1 ´ ÿ s
P h p s 2 | s 1 , a 1 q V π p s 2 ; r R ij q| ď| φ p s 1 , a 1 q J p w π 1 ´ ÿ s
P h p s 2 | s 1 , a 1 q p V π 2 p s 2 q| ` E π | p V π 2 p s 2 q ´ Q π 2 p s 2 , π p s 2 q ; r R ij q| ď β } φ p s 1 , a 1 q} Σ ´ 1 1 ` E s 2 | V π p s 2 q ´ Q π 2 p s 2 , π p s 2 q ; r R ij q| ...
[Equation]
˝
E. M AIN T HEOREM AND P ROOF
Now, we restate Theorem 4.4 in a formal version below:
Theorem E.9 (Formal Version of Theorem 4.4) .
For arbitrary 0 ă ε, δ ă 1 , there exists absolute constants
c i , c β and c N , such that by choosing
[Equation]
with probability 1 ´ δ , after K “ H deployments, by running Alg 4 with the collected dataset
D “ t D 1 , ..., D H u and arbitrary r satisfying the linear assumption in A (in Line of Algorithm 2), we will obtain a policy p π such that V π 1 p s 1 ; r q ě
[Table]
[Table] [Table]
As additional guarantees, after h deployments, by running Alg 4 with the collected dataset t D 1 , D 2 ..., D h u and reward function r , we will obtain a policy π | h which is ε -optimal in the MDP truncated at step h .
Proof.
We will use Σ h,i : “ 2 I `
[Equation] [Equation]
q
[Equation]
s to denote the matrix which r Σ h,i approximates in Alg. 2, and use R h,i : “
[Equation] [Equation]
1 h,i φ p¨ , ¨q to denote the reward function used in
Alg 5 if the covariance matrix estimation is perfect (i.e. r Σ h,i “ Σ h,i ).
The proof consists of three steps.
In step 1, we try to show that the inner loop of Alg 2 will terminate and Π h will contain a set of exploratory policies.
In step 2, we will analyze the samples generated by a mixture of policies in Π h .
In the last step, we determine the choice of hyper-parameters and ﬁll the gaps of pre-assumed constraints and induction conditions.
Step 1: Exploration Ability for Policies in Π h
In the inner loop (line 5 - 12) in Algorithm 2, our goal is to ﬁnd a set of policies Π h , such that if the algorithm stops at iteration i , the following uncertainty measure is as small as possible
[Equation]
To achieve this goal, we repeatedly use Alg 6 to estimate the covariance matrix of the policy and append it to r Σ h,i as an approximation of Σ h,i , and use Alg 5 to ﬁnd a near-optimal policy to maximizing the uncertainty-based reward function r R , by sampling trajectories with which we can reduce the uncertainty Q ˚ h,i in Eq.(28).
First, we take a look at the estimation error of the accumulative covariance matrix when running
Algorithm 2.
On the conditions in Lemma E.8, we can bound the elementwise estimation error of Σ h,i :
[Equation]
As a result of Lemma E.
[Equation]
we have:
[Equation]
where the last but two step is because we at most repeat it i max iterations at each layer
h
[Equation]
and we introduce the following constraint for ξ during the derivation
[Equation]
to make sure the bias is small and all the iterms occurs in the derivation is well deﬁned:
[Equation]
Next, we want to ﬁnd a good choice of i max to make sure V h,i ` 1 will not always be large and the for-loop will break for some i ď i max .
We ﬁrst provide an upper bound for V h,i ` 1 :
[Equation] [Equation]
Eq.(30))
Next, we try to show that V ¯ π h,i ` 1 p s 1 ; R h,i q can not always be large.
According to Elliptical Potential
Lemma in Lemma C.1, we have:
[Equation]
ď
[Equation]
ď
[Equation]
where in the last step, we use the deﬁnition of Σ h,i .
Therefore,
[Equation]
In order to guarantee min i V ¯ π h,i ` 1 p s 1 ; R h,i q ď ν 2 min { 8 , we require:
[Equation]
which can be satisﬁed by:
[Equation]
(31) for some absolute constant c i .
Combining the above results, we can conclude that the inner loop in Alg 2 will break at some i ă i max , such that V h,i ` 1 ď 3 ν 2min { 8 , and guarantee that:
[Equation]
estimation
[Equation]
Eq
[Equation] [Equation]
(Overestimation in Lemma E.5)
[Equation]
Step 2: Policy Deployment and Concentration Error For uniform mixture policy π h,mix : “
Unif p Π h q , by applying Lemma E.13, Lemma E.14 and the results above, we must have:
[Equation]
and this is the motivation of breaking criterion in Line 8 in Alg 2.
In the following, we will use Σ ´ h : “ Σ h
´ I “ ř Nn “ 1 φ p s h,n a h,n q φ p s h,n , a h,n q J to denote the matrix of sampled feature without regularization terms, according to Lemma E.10, with probability 1 ´ δ { 2 , we have:
[Equation]
Follow the same steps in the proof of Lemma E.
[Equation]
we know that
[Equation]
As a result,
[Equation]
which implies that, as long as
[Equation]
we have
[Equation]
N
Therefore
[Equation]
for arbitrary π
[Equation]
we have:
[Equation]
δ
As a result
[Equation]
max π
[Equation]
In order to make sure the induction conditions holds
[Equation]
we need c
[Equation]
c
[Equation]
As long as we tighten the constraint in 32 to:
[Equation]
the induction conditions can be satisﬁed when
[Equation]
or equivalently,
[Equation]
Step 3: Determine Hyper-parameters
[Equation]
Recall that we still have a constraint for Z h in (24)
[Equation]
Since we already determined i max in Eq.(31), also recall our constraints on ξ in (30) the above constraints for ε 0 can be satisﬁed as long as:
[Equation]
Combining all the constraints of ε 0 , including (23), (25), (27) and (34), we conclude that:
[Equation]
(2) Induction error ξ
Besides the constraint in (30), we need another one to make sure the quality of the ﬁnal output policy.
By applying Lemma D.3 for planning algorithm Alg. 2, if the induction condition (4.5) holds till h P r H s , Alg.
4 will return us a policy p π such that:
[Equation]
To make sure
V ˚
´ V p π ď ε , we require ξ ď ε 2 β , which implies that
[Equation]
Choice of N and β
Since ε 0 “ 1 N , by plugging it into Eq.(26), we may choose β to be:
[Equation]
Now, we are ready to compute N .
When ξ “ ε 2 β ď
[Equation]
i
[Equation]
dβ , we have:
[Equation]
and otherwise, we have:
[Equation]
Combining the additional constraint to control the concentration error in Eq.(33), the total number of complexity would at the level:
[Equation]
and therefore,
[Equation]
for some c β .
Near-Optimal Guarantee Under the events in Theorem E.4, considering the failure rate of concentration inequality in Step 2, we can conclude that the induction condition holds for h P r H s with probability 1 ´ δ .
Combining our discussion about choice of ξ above, the probability that Alg 2 will return us an ε -optimal policy would be 1 ´ δ .
The additional guarantee in Theorem E.9 can be directly obtained by considering the induction condition at layer h
P r H s .
˝
E.
T ECHNICAL L EMMA
Lemma E.10 (Matrix Bernstein Theorem (Theorem 6.1.1 in (Tropp, 2015))) .
Consider a ﬁnite sequence t S k u of independent, random matrices with common dimension d 1 ˆ d 2 .
Assume that
[Equation]
and
[Equation]
ď L for each index k
Introduce the random matrix
[Equation]
Let v p
Z q be the matrix variance statistic of the sum:
[Equation]
Then,
[Equation]
Furthermore, for all t ě
[Equation]
Lemma E.11 (Matrix Perturbation) .
Given a positive deﬁnite matrix A ą I and ∆ satisfying | ∆ ij | ď ε ă 1 { d , deﬁne matrix A ` “ A ` ∆ , then for arbitrary φ P R d with } φ } ď 1 , we have:
[Equation]
which implies that
[Equation]
Moreover,
[Equation]
Proof.
First of all, easy to see that
[Equation]
and therefore we have
[Equation]
where we use σ min and σ max to denote the smallest and the largest sigular value
[Equation]
respectively
[Equation]
and use } ¨ } F to refer to the Frobenius norm.
Therefore,
[Equation]
and
[Equation]
Moreover,
[Equation]
Next, we will try to prove that, with a proper choice of N , Algorithm 2 will explore layer h to satisfy the recursive induction condition.
Lemma E.12 (Random Matrix Estimation Error) .
Denote Λ πh “ E π r φφ J s .
Based on the same induction condition 2, we have:
[Equation]
Proof.
Based on Lemma E.8, we have:
[Equation]
and as a result of Lemma E.11, we ﬁnish the proof.
˝
Lemma E.13.
Given a matrix A ľ λI with λ ą 0 , and φ satisﬁes } φ } ď 1 , then we have:
[Equation]
Proof.
Because A ľ λI , we have
[Equation]
¯
Therefore,
[Equation]
˝
Lemma E.14.
Given a matrix A ľ 0 , suppose max π E π r} φ } p cI ` A q ´ 1 s ď r ε ď ν 2min {p 2 ?
c q , where c ą 0 is a constant, where ν min is deﬁned in Deﬁnition 4.3, we have:
[Equation]
Proof.
Because } φ } p cI
[Equation]
c , we must have:
[Equation]
Consider the SVD of A “ U J Σ U with Σ “ p σ ii q i “ 1 ,...,d
and U “ r u 1 , u 2 ..., u d s , then we have: @ π,
[Equation]
According to the Deﬁnition 4.
[Equation]
we have:
[Equation]
which implies that
[Equation]
By applying Lemma E.13 and assign λ “ c
[Equation]
we have:
[Equation]
E. M ORE ABOUT OUR REACHABILITY COEFFICIENT
Recall the deﬁnition of reachability coefﬁcient in (Zanette et al., 2020) is:
[Equation]
Easy to see that, for arbitrary θ with } θ } “ 1 , we have
[Equation]
Therefore,
[Equation]
Besides, according to the min-max theorem, ν h is also lower bounded by
[Equation]
to see this, max
[Equation]
In fact, the value of max π σ min p E π r φ h
φ J h sq is also related to the ”Well-Explored Dataset” assumption in many previous literature in ofﬂine setting (Jin et al., 2021b), where it is assumed that there exists a behavior policy such that the minimum singular value of the covariance matrix is lower bounded.
Therefore, we can conclude that our reachability coefﬁcient ν min is also lower bounded by, e.g. c in Corollary 4.6 in (Jin et al., 2021b).
F E XTENDED D EPLOYMENT -E FFICIENT RL S ETTING F. S AMPLE -E FFICIENT DE-RL
In applications such as recommendation systems, the value of N cannot exceed the number of users our system serves during a period of time.
Therefore, as an interesting extension to our framework, we can revise the constraint (b) in Deﬁnition 2.1 and explicitly assign an upper bound for N .
Concretely, we may consider the following alternatives: (b’)
The sample size
N ď d c 1 H c 2 ε ´ c
3 log c 4 dHεδ , where c 1 , c 2 , c 3 , c 4 ą 0 are some constant ﬁxed according to the real situation.
Under these revised constraints, the lower bound for K may be different.
In fact, given constraints in the form of N ď N 0 , our results in Section 4 already implies an upper bound for K , since we can emulate 1 deployment of our algorithm that uses a large N ą N 0 by deploying the same policy for r N { N 0 s times.
However, this may result in sub-optimal deployment complexity since we are not adaptively updating our policy within those N { N 0 deployments.
It would be an interesting open problem to identify the ﬁne-grained trade-off between K and N 0 in such a setting.
F. S AFE DE-RL
Monotonic Policy Improvement Constraint
In many applications, improvement of service quality after policy update is highly desired, and can be incorporated in our formulation by adding an additional constraint into Def 2.1:
[Equation]
Because we require the deployed policy has substantial probability to visit unfamiliar states so that the agent can identify the near-optimal policy as shown in Def 2.1-(a), we relax the strict policy improvement with an small budget term ε ą 0 for exploration.
Trade-off between Pessimism and Optimism
The balance between satisfying two contradictory constraints: (a) and (c), implies that a proper algorithm should leverage both pessimism and optimism in face of uncertainty.
In Algorithm 7, we propose a simple but effective mixture policy srategy, where we treat pessimistic and optimistic algorithm as black boxes and mix the learned policies with a coefﬁcient α .
One key property of the mixed policy is that:
1 for k=1,2,...,K do
D “ t D 1 , D 2 , ..., D k u π k, pessim Ð PessimismBased OﬄineAlgorithm p D q π k, optim Ð OptimismBased BatchExplorationStrategy p D q // Mix policy in trajectory level, i.e. w.p.
´ α , τ „ π pessim ; w.p. α , τ „ π optim π k Ð p 1 ´ α q π k, pessim ` απ k, optim
D k “ t τ n „ π k , @ n P r N su 8 end
Property F.1 (Policy improvement for mixture policy) .
[Equation]
As a result, as long as the ofﬂine algorithm (which we treat as a black box here) has some policy improvement guarantee, such as (Kumar et al., 2020; Liu et al., 2020; Laroche et al., 2019), then
Eq.(36) implies a substantial policy improvement if α is chosen appropraitely.
Besides, if we use
Algorithms in Section 4.1 or 4.2, and collecting r N “ Θ p N { α q samples, the guarantees in Theorem 4.1 and Theorem 4.4 can be extended correspondingly.
Therefore, Alg 7 will return us a near-optimal policy after K deployments while satisfying the safety constraint (c) in (35).