T OWARDS D EPLOYMENT -E FFICIENT R EINFORCEMENT L EARNING : L OWER B OUND AND O PTIMALITY
Anonymous authors
Paper under double-blind review
A BSTRACT
Deployment efﬁciency is an important criterion for many real-world applications of reinforcement learning (RL).
Despite the community’s increasing interest, there lacks a formal theoretical formulation for the problem.
In this paper, we propose such a formulation for deployment-efﬁcient RL (DE-RL) from an “optimization with constraints” perspective: we are interested in exploring an MDP and obtaining a near-optimal policy within minimal deployment complexity , whereas in each deployment the policy can sample a large batch of data.
Using ﬁnite-horizon linear
MDPs as a concrete structural model, we reveal the fundamental limit in achieving deployment efﬁciency by establishing information-theoretic lower bounds, and provide algorithms that achieve the optimal deployment efﬁciency.
Moreover, our formulation for DE-RL is ﬂexible and can serve as a building block for other practically relevant settings; we give “Safe DE-RL” and “Sample-Efﬁcient DE-RL” as two examples, which may be worth future investigation.
I NTRODUCTION
In many real-world applications, deploying a new policy to replace the previous one is costly, while generating a large batch of samples with an already deployed policy can be relatively fast and cheap.
For example, in recommendation systems (Afsar et al., 2021), education software (Bennane et al., 2013), and healthcare (Yu et al., 2019), the new recommendation, teaching, or medical treatment strategy must pass several internal tests to ensure safety and practicality before being deployed, which can be time-consuming.
On the other hand, the algorithm may be able to collect a large amount of samples in a short period of time if the system serves a large population of users.
Besides, in robotics applications (Kober et al., 2013), deploying a new policy usually involves operations on the hardware level, which requires non-negligible physical labor and long waiting periods, while sampling trajectories is relatively less laborious.
However, deployment efﬁciency was neglected in most of existing RL literatures.
Even for those few works considering this important criterion (Bai et al., 2020; Gao et al., 2021; Matsushima et al., 2021), either their settings or methods have limitations in the scenarios described above, or a formal mathematical formulation is missing.
We defer a detailed discussion of these related works to Section 1.1.
In order to close the gap between existing RL settings and real-world applications requiring high deployment efﬁciency, our ﬁrst contribution is to provide a formal deﬁnition and tractable objective for Deployment-Efﬁcient Reinforcement Learning (DE-RL) via an “optimization with constraints” perspective.
Roughly speaking, we are interested in minimizing the number of deployments K under two constraints: (a) after deploying K times, the algorithm can return a near-optimal policy, and (b) the number of trajectories collected in each deployment, denoted as N , is at the same level across K deployments, and it can be large but should still be polynomial in standard parameters.
Similar to the notion of sample complexity in online RL, we will refer to K as deployment complexity .
To provide a more quantitative understanding, we instantiate our DE-RL framework in ﬁnite-horizon linear MDPs 1 (Jin et al., 2019) and develop the essential theory.
The main questions we address are:
Q1:
What is the optimum of the deployment efﬁciency in our DE-RL setting?
Q2:
Can we achieve the optimal deployment efﬁciency in our DE-RL setting?
When answering these questions, we separately study algorithms with or without being constrained to deploy deterministic policies each time.
While deploying more general forms of policies can be practical (e.g., randomized experiments on a population of users can be viewed as deploying a mixture of deterministic policies), most previous theoretical works in related settings exclusively focused on upper and lower bounds for algorithms using deterministic policies (Jin et al., 2019; Wang et al., 2020b; Gao et al., 2021).
As we will show, the origin of the difﬁculty in optimizing deployment efﬁciency and the principle in algorithm design to achieve optimal deployment efﬁciency are generally different in these two settings, and therefore, we believe both of them are of independent interests.
As our second contribution, in Section 3, we answer Q1 by providing information-theoretic lower bounds for the required number of deployments under the constraints of (a) and (b) in Def 2.1.
We establish Ω p dH q and r Ω p H q lower bounds for algorithms with and without the constraints of deploying deterministic policies, respectively.
Contrary to the impression given by previous empirical works (Matsushima et al., 2021), even if we can deploy unrestricted policies, the minimal number of deployments cannot be reduced to a constant without additional assumptions, which sheds light on the fundamental limitation in achieving deployment efﬁciency.
Besides, in the line of work on “horizon-free RL” (e.g., Wang et al., 2020a), it is shown that RL problem is not signiﬁcantly harder than bandits (i.e., when H “ 1 ) when we consider sample complexity.
In contrast, the H dependence in our lower bound reveals some fundamental hardness that is speciﬁc to long-horizon RL, particularly in the deployment-efﬁcient setting.
2 Such hardness results were originally conjectured by Jiang & Agarwal (2018), but no hardness has been shown in sample-complexity settings.
After identifying the limitation of deployment efﬁciency, as our third contribution, we address Q2 by proposing novel algorithms whose deployment efﬁciency match the lower bounds.
In Section 4.1, we propose an algorithm deploying deterministic policies, which is based on Least-Square Value
Iteration with reward bonus (Jin et al., 2019) and a layer-by-layer exploration strategy, and can return an ε -optimal policy within O p dH q deployments.
As part of its analysis, we prove Lemma 4.2 as a technical contribution, which can be regarded as a ﬁnite-sample version of the well-known “Elliptical Potential Lemma”(Carpentier et al., 2020) and may be of independent interest.
Moreover, our analysis based on Lemma 4.2 can be applied to the reward-free setting (Jin et al., 2020; Wang et al., 2020b) and achieve the same optimal deployment efﬁciency.
In Section 4.2, we focus on algorithms which can deploy arbitrary policies.
They are much more challenging because it requires us to ﬁnd a provably exploratory stochastic policy without interacting with the environment.
To our knowledge, Agarwal et al.
(2020b) is the only work tackling a similar problem, but their algorithm is model-based which relies on a strong assumption about the realizability of the true dynamics and a sampling oracle that allows the agent to sample data from the model, and how to solve the problem in linear MDPs without a model class is still an open problem.
To overcome this challenge, we propose a model-free layer-by-layer exploration algorithm based on a novel covariance matrix estimation technique, and prove that it requires Θ p H q deployments to return an ε -optimal policy, which only differs from the lower bound r Ω p H q by a logarithmic factor.
Although the per-deployment sample complexity of our algorithm has dependence on a “reachability coefﬁcient” (see Def. 4.3), similar quantities also appear in related works (Zanette et al., 2020; Agarwal et al., 2020b; Modi et al., 2021) and we conjecture that it is unavoidable and leave the investigation to future work.
Finally, thanks to the ﬂexibility of our “optimization with constraints” perspective, our DE-RL setting can serve as a building block for more advanced and practically relevant settings where optimizing the number of deployments is an important consideration.
In Appendix F, we propose two potentially interesting settings: “Safe DE-RL” and “Sample-Efﬁcient DE-RL”, by introducing constraints regarding safety and sample efﬁciency, respectively.
1. C LOSELY R ELATED W ORKS
We defer the detailed discussion of previous literatures about pure online RL and pure ofﬂine RL to Appendix A, and mainly focus on those literatures which considered deployment efﬁciency and more related to us in this section.
To our knowledge, the term “deployment efﬁciency” was ﬁrst coined by Matsushima et al.
, but they did not provide a concrete mathematical formulation that is amendable to theoretical investigation.
In existing theoretical works, low switching cost is a concept closely related to deployment efﬁciency, and has been studied in both bandit (Esfandiari et al., 2020; Han et al., 2020;
Gu et al., 2021;
Ruan et al., 2021) and RL settings (Bai et al., 2020;
Gao et al., 2021; Kong et al., 2021).
Another related concept is concurrent RL, as proposed by Guo & Brunskill (2015).
We highlight the difference with them in two-folds from problem setting and techniques.
As for the problem setting, existing literature on low switching cost mainly focuses on sub-linear regret guarantees, which does not directly implies a near-optimal policy after a number of policy deployments 3 .
Besides, low switching-cost RL algorithms (Bai et al., 2020; Gao et al., 2021; Kong et al., 2021) rely on adaptive switching strategies (i.e., the interval between policy switching is not ﬁxed), which can be difﬁcult to implement in practical scenarios.
For example, in recommendation or education systems, once deployed, a policy usually needs to interact with the population of users for a fair amount of time and generate a lot of data.
Moreover, since policy preparation is time-consuming (which is what motivates our work to begin with), it is practically difﬁcult if not impossible to change the policy immediately once collecting enough data for policy update, and it will be a signiﬁcant overhead compared to a short policy switch interval.
Therefore, in applications we target at, it is more reasonable to assume that the sample size in each deployment (i.e., between policy switching) has the same order of magnitude and is large enough so that the overhead of policy preparation can be ignored.
More importantly, on the technical side, previous theoretical works on low switching cost mostly use deterministic policies in each deployment, which is easier to analyze.
This issue also applies to the work of Guo & Brunskill (2015) on concurrent PAC RL.
However, if the agent can deploy stochastic (and possibly non-Markov) policies (e.g., a mixture of deterministic policies), then intuitively— and as reﬂected in our lower bounds—exploration can be done much more deployment-efﬁciently, and we provide a stochastic policy algorithm that achieves an r O p H q deployment complexity and overcomes the Ω p dH q lower bounds for deterministic policy algorithms (Gao et al., 2021).
P RELIMINARIES
Notation Throughout our paper, for n P Z ` , we will denote r n s “ t 1 , 2 , ..., n u .
r ¨ s denotes the ceiling function.
Unless otherwise speciﬁed, for vector x P R d and matrix X
P R d ˆ d , } x } denotes the vector l 2 -norm of x and } X } denotes the largest singular value of X .
We will use standard big-oh notations O p¨q , Ω p¨q , Θ p¨q , and notations such as r O p¨q to suppress logarithmic factors.
2. E PISODIC R EINFORCEMENT L EARNING
We consider an episodic Markov Decision Process denoted by M p S , A , H, P, r q , where S is the state space, A is the ﬁnite action space, H is the horizon length, and P “ t P h u
Hh “ 1 and r “ t r h u
Hh “denote the transition and the reward functions.
At the beginning of each episode, the environment will sample an initial state s 1 from the initial state distribution d 1 .
Then, for each time step h P r H s , the agent selects an action a h P A , interacts with the environment, receives a reward r h p s h , a h q , and transitions to the next state s h ` 1 .
The episode will terminate once s H ` 1 is reached.
A (Markov) policy π h p¨q at step h is a function mapping from S Ñ ∆ p
A q , where ∆ p A q denotes the probability simplex over the action space.
With a slight abuse of notation, when π h p¨q is a deterministic policy, we will assume π h p¨q : S Ñ A .
A full (Markov) policy π “ t π 1 , π 2 , ..., π H u speciﬁes such a mapping for each time step.
We use V πh p s q and Q πh p s, a q to denote the value function and Q-function at step h P r H s , which are deﬁned as:
[Equation]
We also use V ˚ h p¨q and Q ˚ h p¨ , ¨q to denote the optimal value functions and use π ˚ to denote the optimal policy that maximizes the expected return J p π q : “ E r
ř
Hh “ 1 r p s h , a h q| π s .
In some occa- sions, we use V πh p s ; r q and Q πh p s, a ; r q to denote the value functions with respect to r as the reward function for disambiguation purposes.
The optimal value functions and the optimal policy will be denoted by V ˚ p s ; r q , Q ˚ p s, a ; r q , π ˚ r , respectively.
Non-Markov Policies
While we focus on Markov policies in the above deﬁnition, some of our results apply to or require more general forms of policies.
For example, our lower bounds apply to non-Markov policies that can depend on the history (e.g., S 1 ˆ A 1 ˆ R ...
ˆ S h
´
1 ˆ A h
´ 1 ˆ R ˆ S
h Ñ A for deterministic policies); our algorithm for arbitrary policies deploys a mixture of deterministic Markov policies, which corresponds to choosing a deterministic policy from a given set at the initial state, and following that policy for the entire trajectory.
This can be viewed as a non-Markov stochastic policy.
2. L INEAR MDP S ETTING
We mainly focus on the linear MDP (Jin et al., 2019) satisfying the following assumptions:
Assumption A (Linear MDP Assumptions) .
An MDP M “ p S , A , H, P, r q is said to be a linear
MDP with a feature map φ : S ˆ A Ñ R d if the following hold for any h P r H s :
• There are d unknown signed measures µ h “ p µ p 1 q h , µ p 2 q h , ..., µ p d q h q over S such that for any p s, a, s 1 q P S ˆ A ˆ S , P h p s
|
s, a q “ x µ h p s 1 q , φ p s, a qy .
• There exists an unknown vector θ h P R d such that for any p s, a q P S ˆ A , r h p s, a q “ x φ p s, a q , θ h y .
Similar to Jin et al.
and Wang et al.
(2020b , without loss of generality, we assume for all p s , a q P S ˆ A and h P r H s , } φ p s, a q} ď 1 , } µ h } ď ?
d , and } θ h } ď ?
d .
In Section 3 we will refer to linear MDPs with stationary dynamics, which is a special case when µ 1 “ µ 2 “ . . .
µ H and θ 1 “ θ 2 “ . . .
“ θ H .
A C ONCRETE D EFINITION OF DE-RL
In the following, we introduce our formulation for DE-RL in linear MDPs.
For discussions of comparison to existing works, please refer to Section 1.1.
Deﬁnition 2.1 (Deployment Complexity in Linear MDPs) .
We say that an algorithm has a deployment complexity K in linear MDPs if the following holds: given an arbitrary linear MDP under Assumption A, for arbitrary ε and
ă δ ă 1 , the algorithm will return a policy π K after K deployments and collecting at most N trajectories in each deployment, under the following constraints:
(a) With probability 1 ´ δ , π K is ε -optimal, i e. J p π K q ě max π J p π q ´ ε .
(b) The sample size N is polynominal, i.e. N “ poly p d, H, 1 ε , log 1 δ q .
Moreover, N should be ﬁxed a priori and cannot change adaptively from deployment to deployment.
Under this deﬁnition, the goal of Deployment-Efﬁcient RL is to design algorithms with provable guarantees of low deployment complexity.
Polynomial Size of N
We emphasize that the restriction of polynomially large N is crucial to our formulation, and not including it can result in degenerate solutions.
For example, if N is allowed to be exponentially large, we can ﬁnish exploration in 1 deployment in the arbitrary policy setting, by deploying a mixture of exponentially many policies that form an ε -net of the policy space.
Alternatively, we can sample actions uniformly, and use importance sampling (Precup, 2000) to evaluate all of them in an off-policy manner.
None of these solutions are practically feasible and are excluded by our restriction on N .
L OWER B OUND FOR D EPLOYMENT C OMPLEXITY IN RL
In this section, we provide information-theoretic lower bounds of the deployment complexity in our DE-RL setting.
We defer the lower bound construction and the proofs to Appendix B.
As mentioned in Section 2, we consider non-Markov policies when we refer to deterministic and stochastic policies in this section, which strengthens our lower bounds as they apply to very general forms of policies.
We ﬁrst study the algorithms which can only deploy deterministic policy at each deployment.
Theorem 3.1.
[Lower bound for deterministic policies, informal] For any d ě 4 , H and any algorithm ψ that can only deploy a deterministic policy at each deployment, there exists a linear MDP
M satisfying Assumption A, such that the deployment complexity of ψ in M is K “ Ω p dH q .
The basic idea of our construction and the proof is that, intuitively, a linear MDP with dimension d and horizon length H has Ω p dH q “independent directions”, while deterministic policies have limited exploration capacity and only reach Θ p 1 q direction in each deployment, which result in Ω p dH q deployments in the worst case.
In the next theorem, we will show that, even if the algorithm can use arbitrary exploration strategy (e.g. maximizing entropy, adding reward bonus), without additional assumptions, the number of deployments K still has to depend on H and may not be reduced to a constant when H is large.
Theorem 3.2.
[Lower bound for arbitrary policies, informal] For any d ě 4 , H, N and any algorithm ψ which can deploy arbitrary policies, there exists a linear MDP M satisfying Assumption A, such that the deployment complexity of ψ in M is K “ Ω p H { r log d p NH q s q “ r Ω p H q .
The origin of the difﬁculty can be illustrated by a recursive dilemma: in the worst case, if the agent does not have enough information at layer h , then it cannot identify a good policy to explore layer h ` Ω p log d p NH qq in 1 deployment, and so on and so forth.
Given that we enforce N to be polynomial, the agent can only push the “information boundary” forward by Ω p log d p NH qq “ r Ω p 1 q layers per deployment.
In many real-world applications, such difﬁculty can indeed exist.
For example, in healthcare, the entire treatment is often divided into multiple stages.
If the treatment in stage h is not effective, the patient may refuse to continue.
This can result in insufﬁcient samples for identifying a policy that performs well in stage h ` 1 .
Stationary vs. non-stationary dynamics Since we consider non-stationary dynamics in Assump.
A, one may suspect that the H -dependence in the lower bound is mainly due to such nonstationarity.
We show that this is not quite the case, and the H -dependence still exists for stationary dynamics.
In fact, our lower bound for non-stationary dynamics directly imply one for stationary dynamics: given a ﬁnite horizon non-stationary MDP Ă M “ p r S , A , H, r P, r r q , we can construct a stationary MDP M “ p S , A , H, P, r q by expanding the state space to S “ r S ˆ r H s so that the new transition function P and reward function r are stationary across time steps.
As a result, given arbitrary d ě 4 and H ě 2 , we can construct a hard non-stationary MDP instance Ă M with dimension r d “ max t 4 , d { H u and horizon r h “ d { r d “ min t H, d { 4 u , and convert it to a stationary MDP
M with dimension d and horizon h “ r h “ min t H, d { 4 u ď H .
If there exists an algorithm which can solve M in K deployments, then it can be used to solve Ă M in no more than K deployments.
Therefore, the lower bounds for stationary MDPs can be extended from Theorems 3.1 and 3.2, as shown in the following corollary:
Corollary 3.3 (Extension to Stationary MDPs) .
For stationary linear MDP with d ě 4 and H ě 2 , suppose N “ poly p d, H, 1 ε , log 1 δ q , the lower bound of deployment complexity would be Ω p d q for deterministic policy algorithms, and Ω p [Equation] q
[Equation]
for algorithms which can deploy arbitrary policies.
As we can see, the dependence on dimension and horizon will not be eliminated even if we make a stronger assumption that the MDP is stationary.
The intuition is that, although the transition function is stationary, some states may not be reachable from the initial state distribution within a small number of times, so the stationary MDP can effectively have a “layered” structure.
For example, in
Atari games (Bellemare et al., 2013) (where many algorithms like DQN (Mnih et al., 2013) model the environments as inﬁnite-horizon discounted MDPs) such as Breakout, the agent cannot observe states where most of the bricks are knocked out at the initial stage of the trajectory.
Therefore, the agent still can only push forward the “information frontier” a few steps per deployment.
That said, it is possible reduce the deployment complexity lower bound in stationary MDPs by adding more assumptions, such as the initial state distribution providing good coverage over the entire state space, or all the states are reachable in the ﬁrst few time steps.
However, because these assumptions do not always hold and may overly trivialize the exploration problem, we will not consider them in our algorithm design.
Besides, although our algorithms in the next section are designed for nonstationary MDPs, they can be extended to stationary MDPs by sharing covariance matrices, and we believe the analyses can also be extended to match the lower bound in Corollary 3.3.
T OWARDS O PTIMAL D EPLOYMENT E FFICIENCY
In this section we provide algorithms with deployment-efﬁciency guarantees that nearly match the lower bounds established in Section 3.
Although our lower bound results in Section 3 consider nonMarkov policies, our algorithms in this section only use Markov policies (or a mixture of Markov policies, in the arbitrary policy setting), which are simpler to implement and compute and are already near-optimal in deployment efﬁciency.
Inspiration from Lower Bounds: a Layer-by-Layer Exploration Strategy
The linear dependence on H in the lower bounds implies a possibly deployment-efﬁcient manner to explore, which we call a layer-by-layer strategy: conditioning on sufﬁcient exploration in previous h ´ 1 time steps, we can use poly p d q deployments to sufﬁciently explore the h -th time step, then we only need
H ¨ poly p
d
q deployments to explore the entire MDP.
If we can reduce the deployment cost in each layer from poly p d q to Θ p d q or even Θ p 1 q , then we can achieve the optimal deployment efﬁciency.
Besides, as another motivation, in Appendix C.4, we will brieﬂy discuss the additional beneﬁts of the layer-by-layer strategy, which will be useful especially in “Safe DE-RL”.
In Sections 4.1 and 4.2, we will introduce algorithms based on this idea and provide theoretical guarantees.
4. D EPLOYMENT -E FFICIENT RL WITH D ETERMINISTIC P OLICIES
1 Input : Failure probability δ ą 0 , and target accuracy ε ą 0 , β Ð c β ¨ dH
a log p dHδ
[Equation] 1 q for some c β ą 0 , total number of deployments K , batch size N , 2 h 1 Ð// h k denotes the layer to explore in iteration k , for all k P r K s 3 for k “ 1 , 2 , ...,
K do
Q kh k ` 1 p¨ , ¨q Ð 0 and V kh k ` 1 p¨q “ for h “ h k , h k ´ 1 , ..., 1 do Λ kh Ð
I
` ř kτ “ 1 ř Nn “ 1 φ τnh p φ τnh q J u kh p¨ , ¨q Ð min t β ¨ b φ p¨ , ¨q J p Λ kh q ´ 1 φ p¨ , ¨q , H u w kh Ð p Λ kh q ´
ř k ´ 1 τ
“ 1 ř Nn “ 1 φ τnh ¨ V kh ` 1 p s τnh ` 1 q
Q kh p¨ , ¨q Ð min tp w kh q J φ p¨ , ¨q ` r h p¨ , ¨q ` u kh p¨ , ¨q , H u and V kh p¨q “ max a P A Q kh p¨ , a q π kh p¨q Ð arg max a P A Q kh p¨ , a q end
Deﬁne π k “ π k 1 ˝ π k 2 ...
˝
π kh k ˝ unif r h k ` 1: H s for n “ 1 , ..., N do
Receive initial state s kn 1 „ d for h “ 1 , 2 , ..., H do
Take action a knh Ð π kh p s knh q and observe s knh ` 1 „ P h p s kh , a kh q end end
Compute ∆ k Ð 2 βN
ř Nn “ 1 ř h k h “b φ p s knh , a knh q J p Λ kh q ´ 1 φ p s knh , a knh q .
if ∆ k ě εh k 2 H then h k ` 1 Ð h k ; else if h k “ H then return π
In this sub-section, we focus on the setting where each deployed policy is deterministic.
In Alg 1, we propose a provably deployment-efﬁcient algorithm built on Least-Square Value Iteration with
UCB (Jin et al., 2019) 4 and the “layer-by-layer” strategy.
Brieﬂy speaking, at deployment k , we focus on exploration in previous h k layers, and compute π k 1 , π k 2 , ..., π kh k by running LSVI-UCB in an MDP truncated at step h k .
After that, we deploy π k to collect N trajectories, and complete the trajectory after time step h k with an arbitrary policy.
(In the pseudocode we choose uniform, but the choice is inconsequential.)
In line 19, we compute ∆ k with samples and use it to judge whether we should move on to the next layer till all H layers have been explored.
The theoretical guarantee is listed below, and the missing proofs are deferred to Appendix C.
Theorem 4.1 (Deployment Complexity) .
For arbitrary ε, δ ą 0 , and arbitrary c K ě 2 , as long as
N ě c
[Equation]
, where c is an absolute constant, by choosing
[Equation]
Algorithm 1 will terminate at iteration k ď K and return us a policy π k , and with probability 1 ´ δ ,
[Equation]
As an interesting observation, Eq (1) reﬂects the trade-off between the magnitude of K and N when K is small.
To see this, when we increase c K and keep it at the constant level, K deﬁnitely increases while N will be lower because its dependence on d, H, ε, δ decreases.
Moreover, the beneﬁt of increasing c K is only remarkable when c K is small (e.g. we have N “ O p H 9 d 6 ε ´ 4 q if c K “ 2 , while N “ O p H 5 d 3 .
6 ε ´ 2 .
4 q if c K “ 6 ), and even for moderately large c K , the value of N quickly approaches the limit lim
[Equation]
[Equation]
2 log 2 p Hdδε q .
Another key step in proving the deployment efﬁciency of Algorithm 1 is Lemma 4.2 below.
In
Appendix C, we will discuss the possibility of applying this technique to LSVI-UCB (Jin et al., 2019) with large batch sizes, and the additional beneﬁt of our layer-by-layer strategy.
Lemma 4.2.
[Finite Sample Elliptical Potential Lemma] Consider a sequence of matrices
A 0 , A N , ..., A p K ´ 1 q N P R d ˆ d with A 0 “ I d ˆ d and A kN “ A p k ´ 1 q N ` Φ k ´ 1 , where Φ k ´ 1 “ ř kNt “p k ´ 1 q N ` 1 φ t φ J t and max t ď KN } φ t } ď 1 .
We deﬁne:
K ` : “ !
[Equation]
Nε ) .
For arbitrary ε ă 1 , and arbitrary c K ě 2 , if K “ c K dH ` 1 , by choosing N ě c ´ c [Equation] log [Equation]
q ¯ 1 cK [Equation] 1 , where c is an absolute constant independent with c K , d, H, ε , we have | K ` | ď c K d ă K { H.
Extension to Reward-free setting Based on the similar methodology, we can design algorithms for reward-free setting (Wang et al., 2020b) and obtain guarantees for deployment complexity based on Lemma 4.2.
We defer the algorithms and proofs to Appendix D, and summarize the main result in Theorem D.
4. D EPLOYMENT -E FFICIENT RL WITH A RBITRARY P OLICIES
From the discussion of lower bounds in Section 3, we know that in order to reduce the deployment complexity from Ω p dH q to r Ω p H q , we have to utilize stochastic (and possibly non-Markov) policies and try to explore as many different directions as possible in each deployment (as opposed todirection in Algorithm 1).
The key challenge is to ﬁnd a stochastic policy—before the deployment starts—which can sufﬁciently explore d independent directions.
In Algorithm 2, we overcome this difﬁculty by a new covariance matrix estimation method (Algorithm 6 in Appendix E).
The basic idea is that, for arbitrary policy π 5 , the covariance matrix Λ πh : “ E π r φ p s h , a h q φ p s h , a h q J s can be estimated element-wise by running policy evaluation for π with φ i p s h , a h q φ j p s h , a h q as a reward function, where i, j P r d s and φ i p¨ , ¨q denotes the i -th component of vector φ p¨ , ¨q .
Here we mainly focus on evaluating deterministic policy or stochastic policy mixed from a ﬁnite number of determnistic policies, because for the other stochastic policies, exactly computing the expectation over policy distribution may be intractable.
1 Input : do
Initialize deterministic
r Σ h, 1 “ 2 I , Π h “ tu for i “ 1 , 2 , ..., i max do p Λ π h,i h Ð EstimateCovMatrix p h, D r 1: h ´ 1 s , Σ r 1: h ´ 1 s , π h,i q
# Alg 6, Appx E r Σ h,i ` 1 “ r Σ h,i ` p Λ π h,i h
V h,i ` 1 , ¯ π h,i ` 1 Ð SolveOptQ p h, D r 1: h ´ 1 s , Σ r 1: h ´ 1 s , β, r Σ h,i ` 1 , ε 0 q
# Alg 5, Appx E if V h,i ` 1 ď 3 ν 2min { 8 then break ; Π h “ Π h Ť t ¯ π h,i ` 1 u end
Σ h
“ I , D h “ tu , π h, mix : “ unif p Π h q for n “ 1 , 2 , ..., N do
Sample trajectories with π h, mix Σ h “ Σ h ` φ p s h,n , a h,n q φ p s h,n , a h,n q J
D h “ D h Ť t s h,n , a h,n , r h,n , s h ` 1 ,n u end 18 end 19 p π r Ð Alg 4 p H, t D 1 , ..., D H u , r q 20 return p π r However, a new challenge emerging is that, because the transition is stochastic, in order to guarantee low evaluation error for all possible policies ¯ π h,i ` 1 , we need an union bound over all policies to be evaluated, which is challenging if the policy class is inﬁnite. To overcome this issue, we discretize the value functions in Algorithm 5 (see Appendix E) to allow for a union bound over the policy space: after computing the Q-function by LSVI-UCB, before converting it to a greedy policy, we ﬁrst project it to an ε 0 -net of the entire Q-function class.
In this way, the number of policy candidates is ﬁnite and the projection error can be controlled as long as ε 0 is small enough.
Using the above techniques, in Lines 3-11, we repeatedly use Alg 6 to estimate the accumulative covariance matrix r Σ h,i ` 1 and further eliminate uncertainty by calling Alg 5 to ﬁnd a policy (approximately) maximizing uncertainty-based reward function r R : “ } φ } r Σ ´ 1 h,i ` 1 .
For each h P r H s , inductively conditioning on sufﬁcient exploration in previous h ´ 1 layers, the errors of Alg 6 and
Alg 5 will be small, and we will ﬁnd a ﬁnite set of policies Π h to cover all dimensions in layer h .
(This is similar to the notion of “policy cover” in Du et al.
Agarwal et al.
(2020a).)
Then, layer h can be explored sufﬁciently by deploying a uniform mixture of Π and choosing N large enough (Lines 12-17).
Also note that the algorithm does not use the reward information, and is essentially a reward-free exploration algorithm.
After exploring all H layers, we obtain a dataset t D 1 , ..., D H u and can use Alg 4 for planning with any given reward function r satisfying Assump.
A to obtain a near-optimal policy.
Deployment complexity guarantees We ﬁrst introduce a quantity denoted as ν min , which measures the reachability to each dimension in the linear MDP.
In Appendix E.8, we will show that the ν min is no less than the “explorability” coefﬁcient in Deﬁnition 2 of Zanette et al.
and ν 2min is also lower bounded by the maximum of the smallest singular value of matrix E π r φφ J s .
Deﬁnition 4.3 (Reachability Coefﬁcient) .
[Equation]
Now, we are ready to state the main theorem of this section, and defer the formal version and its proofs to Appendix E.
Our algorithm is effectively running reward-free exploration and therefore our results hold for arbitrary linear reward functions.
Theorem 4.4.
[Informal] For arbitrary 0 ă ε, δ ă 1 , with proper choices of i max , ε 0 , β , we can choose N “ poly p d, H, 1 ε , log 1 δ , 1 ν min q , such that, after K “ H deployments, with probability 1 ´ δ ,
Algorithm 2 will collect a dataset D “ t D 1 , ..., D H u , and if we run Alg 4 with D and arbitrary reward function satisfying Assump.
A, we will obtain p π r such that V p π r 1 p s 1 ; r q ě V ˚ 1 p s 1 ; r q ´ ε .
Proof Sketch Next, we brieﬂy discuss the key steps of the proof.
Since ε 0 can be chosen to be very small, we will ignore the bias induced by ε 0 when providing intuitions.
Our proof is based on the induction condition below.
We ﬁrst assume it holds after h ´ 1 deployments (which is true when h “ 1 ), and then we try to prove at the h -th deployment we can explore layer h
well enough so that the condition holds for h .
Condition 4.5.
[Induction Condition] Suppose after h ´ 1 deployments, we have the following induction condition for some ξ ă 1 { d , which will be determined later:
[Equation]
The l.h.s. of Eq.(2) measures the uncertainty in previous h ´ 1 layers after exploration.
As a result, with high probability, the following estimations will be accurate:
[Equation]
where } ¨ } 8 , 8 denotes the entry-wise maximum norm.
This directly implies that:
[Equation]
where Σ h,i ` 1 : “ 2 I ` ř
[Equation]
s [Equation] [Equation] q φ p s h , a h q J s is the target value for r Σ h,i ` 1 to approximate.
Besides, recall that in Algorithm 5, we use
[Equation]
φ as the reward function, and the induction condition also implies that:
[Equation]
As a result, if ξ and the resolution ε 0 are small enough, ¯ π h,
i ` 1 would gradually reduce the uncertainty and V h,i ` 1 (also max π E π r} φ p s h , a h q} r Σ ´ 1 h,i ` 1 s ) will decrease.
However, the bias is at the level
O p ξ q , and therefore, no matter how small ξ is, as long as ξ ą 0 , it is still possible that the policies in Π h do not cover all directions if some directions are very difﬁcult to reach, and the error due to such a bias will be at the same level of the required accuracy in induction condition, i.e. O p ξ q .
This is exactly where the “reachability coefﬁcient” ν min deﬁnition helps.
The introduction of ν min provides a threshold, and as long as ξ is small enough so that the bias is lower than the threshold, each dimension will be reached with substantial probability when the breaking criterion in Line 9 is satisﬁed.
As a result, by deploying unif p Π h q and collecting a sufﬁciently large dataset, the induction condition will hold till layer H .
Finally, combining the guarantee of Alg 4, we complete the proof.
C ONCLUSION AND F UTURE W ORK
In this paper, we propose a concrete theoretical formulation for DE-RL to ﬁll the gap between existing RL literatures and real-world applications with deployment constraints.
Based on our framework, we establish lower bounds for deployment complexity in linear MDPs, and provide novel algorithms and techniques to achieve optimal deployment efﬁciency.
Besides, our formulation is ﬂexible and can serve as building blocks for other practically relevant settings related to DE-RL.
We conclude the paper with two such examples, defer a more detailed discussion to Appendix F, and leave the investigation to future work.
Sample-Efﬁcient DE-RL
In our basic formulation in Deﬁnition 2.1, we focus on minimizing the deployment complexity K and put very mild constraints on the per-deployment sample complexity
N .
In practice, however, the latter is also an important consideration, and we may face additional constraints on how large N can be, as they can be upper bounded by e.g. the number of customers or patients our system is serving.
Safe DE-RL In real-world applications, safety is also an important criterion.
The deﬁnition for safety criterion in Safe DE-RL is still an open problem, but we believe it is an interesting setting since it implies a trade-off between exploration and exploitation in deployment-efﬁcient setting.