Regret Bounds for Information-Directed Reinforcement Learning
Anonymous Author(s)
Afﬁliation
Address email
Abstract
Information-directed sampling (IDS) has revealed its potential as a data-efﬁcientalgorithm
[Lu et al., 2021] for reinforcement learning (RL).
However, theoreticalunderstanding of IDS for Markov Decision Processes (MDPs) is still limited.
We develop novel information-theoretic tools to bound the information ratio andcumulative information gain about the learning target.
Our theoretical results shedlight on the importance of choosing the learning target such that the practitionerscan balance the computation and regret bounds.
As a consequence, we derive priorfree Bayesian regret bounds for vanilla-IDS which learns the whole environmentunder tabular ﬁnite-horizon MDPs.
In addition, we propose a computationallyefﬁcient regularized-IDS that maximizes an additive form rather than the ratioform and show that it enjoys the same regret bound as vanilla-IDS .
With the aidof rate-distortion theory, we improve the regret bound by learning a surrogate, lessinformative environment.
Furthermore, we extend our analysis to linear MDPs andprove similar regret bounds for Thompson sampling as a by-product.
Introduction
Information-directed sampling (IDS) is a design principle proposed by [Russo and Van Roy, 2014,2018] that optimizes the trade-off between information and regret .
Comparing with other designprinciples such as UCB and Thompson sampling (TS), IDS can automatically adapt to differentinformation-regret structures.
As a result, IDS demonstrates impressive empirical performance [Russoand Van Roy, 2018] and outperforms UCB and TS in terms of asymptotic optimality [Kirschner et al.
,2021] and minimax optimality in heteroscedastic bandits [Kirschner and Krause, 2018] and sparselinear bandits [Hao et al., 2021].
In the context of full RL mutiple works have examined the empirical performance of IDS [Nikolovet al., 2018, Lu et al., 2021].
However, formal regret guarantee for IDS is still lacking.
IDS minimizesa notion of information ratio that is the ratio of per-episode regret and information gain about thelearning target.
While different choices of the learning target could lead to different regret boundsand computational methods, the most natural choice is the whole environment and we name thecorresponding IDS as vanilla-IDS .
In this work, we prove the ﬁrst prior-free (cid:101) O [Equation] Bayesian regret bound for vanilla-IDS ,where S is the size of state space, A is the size of action space, H is the length of episodes and L isthe number of episodes.
Computationally, vanilla-IDS needs to optimize over the full policy space,which is not efﬁcient in general.
To facilitate the computation, we consider its regularized form, named regularized-IDS , that can be solved by any dynamic programming solver.
By carefullychoosing the tunable parameter, we prove that regularized-IDS enjoys the same regret bound asvanilla-IDS .
Although learning the whole environment offers certain computational advantages, the agent couldtake too much information to learn the whole environment exactly.
A key observation is that differentstates may correspond to the same value function which eventually determines the behavior ofthe optimal policy.
Through the rate-distortion theory, we construct a surrogate environment thatis less informative to learn but enough to identify the optimal policy.
As a result, we proposesurrogate-IDS that takes the surrogate environment as the learning target and prove a sharper(cid:101) O ( √ S
A 2 H 4 L ) bound for tabular MDPs.
In the end, we extend our analysis to linear MDPs where we must learn a surrogate environment dueto potentially inﬁnitely many states and derive a (cid:101) O ( dH 2 √ T ) regret bound that matches the existinglower bound up to a factor of H .
As a by-product of our analysis, we also prove prior-free Bayesianregret bounds for TS under tabular and linear MDPs.
Related work In general, there are two ways to prove Bayesian regret bounds.
The ﬁrst is tointroduce conﬁdence sets such that the Bayesian regret bounds of TS can match the best possiblefrequentist regret bounds by UCB [Russo and Van Roy, 2014] and has been extended to RL by
Osband et al.
[2013], Osband and Van Roy [2014], Osband et al.
However, when the bestpossible bound for UCB is sub-optimal, this technique yields a sub-optimal Bayesian regret bound.
In addition, this technique can only be used to analyze TS but not IDS.
The second is to decompose the Bayesian regret into a information ratio term and a cumulativeinformation gain term and bound them by tools from information theory [Russo and Van Roy, 2016].
This technique can be used to analyze both TS
[Dong and Roy, 2018, Bubeck and Sellke, 2020] and
IDS in bandits setting [Russo and Van Roy, 2014, Liu et al., 2018, Kirschner et al., 2020b, Hao et al.
,2021, 2022], partial monitoring [Lattimore and Szepesv´ari, 2019, Kirschner et al., 2020a, Lattimoreand Gyorgy, 2021] but not in RL as far as we know.
One exception is Lu and Van Roy [2019], Lu[2020] who bounded the information ratio for a speciﬁc Dirichlet prior with additional assumptions.
Frequentist regret bounds in episodic RL have received considerable attention recently.
For tabular
MDPs, several representative works include UCBVI [Azar et al., 2017], optimistic Q-learning [Jinet al., 2018], RLSVI [Russo, 2019], UCB-Advantage [Zhang et al., 2020], UCB-MQ [M ´ enard et al.,2021].
While our regret bounds are not state of the art, the primary goal of this paper is to broadenthe set of efﬁcient RL design principles known to satisfy √ T regret bounds.
For linear or linear mixture MDPs, several representative works include LSVI-UCB [Jin et al., 2020],
OPPO [Cai et al., 2020], UCRL-VTR [Ayoub et al., 2020, Zhou et al., 2021], RLSVI
[Zanette et al.
Notably, Dann et al.
[2021] derived minimax regret bounds for a variant of TS.
Beyond linearcases, several works consider general function approximation based on Bellman rank [Jiang et al. ,2017], eluder dimension [Wang et al., 2020], Bellman-eluder dimension [Jin et al., 2021] and bilinearclass [Du et al., 2021].
It is worth mentioning the recent impressive work by Foster et al.
[2021] who proposed a general
Estimation-to-Decisions (E2D) design principle.
Although motivated by different design principles,
E2D shares the similar form as regularized-IDS .
On one hand, Foster et al.
[2021] mainly focuseson statistical complexity, while we offer a speciﬁc computationally-efﬁcient algorithm thanks to thechain rule of mutual information and independent priors.
On the other hand, while E2D tends to learnthe whole environment, our theory in Section 4 suggests learning a surrogate environment could yieldbetter regret bounds.
Preliminary
Finite-horizon MDPs
The environment is characterized by a ﬁnite-horizon time-inhomogeneous
MDP, which is a tuple E = ( S , A , H, { P h } Hh =1 , { r h } Hh =1 ) , where S is the countable state space with
|S| = S , A is the ﬁnite action space with |A| =
A , H is the episode length, P h : S ×
A → ∆ S isthe transition probability kernel and r h : S
×
A → [0 , 1] is the reward function.
For a ﬁnite set S , let∆ S be the set of probability distributions over S .
We assume S , A , r h are known and deterministicwhile the transition probability kernel is unknown and random.
Throughout the paper, we may write
P h and r h explicitly depend on E when necessary.
Let Θ h
= [0 , 1] S ×
A × S be the parameter space of P h and Θ = Θ 1 × · · · × Θ H be the full parameterspace.
We assume ρ h is the prior probability measure for P h on Θ h with Borel σ -algebra andρ = ρ 1 ⊗ · · · ⊗ ρ H as the product prior probability measure for the whole environment on Θ with
Borel σ -algebra.
This ensures the priors over different layers are independent.
Interaction protocol An agent interacts with a ﬁnite-horizon MDP as follows.
The initial states (cid:96) 1 is assumed to be ﬁxed over episodes.
In each episode (cid:96) ∈ [ L ] and each layer h ∈
[ H ] , the agentobserves a state s (cid:96)h , takes an action a (cid:96)h , and receives a reward r (cid:96)h .
Then, the environment evolves to arandom next state s [-ℓh +
-](cid:96)h +1 according to distribution P h ( ·| s (cid:96)h , a (cid:96)h ) .
The episode terminates when s H +1 isreached and is reset to the initial state.
Denote H (cid:96),h as the history of episode (cid:96) up to layer h , e.g., H (cid:96),h = ( s (cid:96) 1 , a (cid:96) 1 , r (cid:96) 1 , . . .
, s (cid:96)h , a (cid:96)h , r (cid:96)h ) andthe set of such possible history is
Ω h = (cid:81) h
i =1 ( S ×
A × [0 , 1]) .
Let D (cid:96) =
( H 1 ,H , . . .
, H (cid:96) − 1 ,H ) asthe entire history up to episode (cid:96) with D 1 = ∅ .
A policy π is a collection of (possibly randomised)mappings ( π 1 , . . .
, π H ) where each π h maps an element from Ω
h − 1 × S to ∆( A ) and Π is the wholepolicy class.
A stationary policy chooses actions based on only the current state and current layer.
The set of such policies is denoted by Π S where we denote π h ( a | s ) as the probability that the agentchooses action a at state s and layer h .
Value function For each h ∈
[ H ] and a policy π , the value function V E h,π : S → R is deﬁned asthe expected value of cumulative rewards received under policy π when starting from an arbitrarystate at h th layer; that is,
[Equation]
where E E π denotes the expectation over the sample path generated under policy π and environment E .
We adapt the convention that V E H +1 ,π ( · ) = 0 .
There always exists an optimal policy π ∗ which givesthe optimal value V E h,π ∗ ( s ) =
max π ∈ Π S V E h,π ( s ) for all s ∈ S and h ∈
[ H ] .
Note that in the Bayesiansetting, π ∗ is a function of E so it is also a random variable.
In addition, we deﬁne the action-valuefunction as follows:
[Equation]
which satisﬁes the Bellman equation: Q E h,π ( s, a )
=
r h ( s, a ) + E s (cid:48) ∼
[Equation]
h [Equation] s
[Equation]
[ V E h
+1 , π ( s [Equation]
Furthermore, we denote the state-action occupancy measure as
[Equation]
where we denote P E π as the law of the sample path generated under policy π and environment E .
Bayesian regret The agent interacts with the environment for L episodes and the total number ofsteps is T = LH .
The expected cumulative regret of an algorithm π = { π (cid:96) } L(cid:96) =1 with respect to anenvironment E is deﬁned as
[Equation]
where the expectation is taken with respect to the randomness of π (cid:96) .
The Bayesian regret then isdeﬁned as
[Equation]
where the expectation is taken with respect to the prior distribution of E .
At each episode, TS ﬁnds
[Equation]
, where E (cid:96) is a sample from the posterior distribution of E , e.g., E (cid:96) ∼ P ( E ∈ ·|D (cid:96) ) .
Notations We write P (cid:96) ( · ) =
P ( ·|D (cid:96) ) , E (cid:96) [ · ] = E [ ·|D (cid:96) ] and also deﬁne the conditional mutualinformation I (cid:96) ( X ; Y ) =
D KL ( P (( X, Y ) ∈ ·|D (cid:96) ) || P ( X = ·|D (cid:96) ) ⊗ P ( Y = ·|D (cid:96) )) .
For a randomvariable χ we deﬁne:
[Equation]
where P (cid:96),π is the law of χ and the history induced by policy π interacting with a sample from theposterior distribution of E given D (cid:96) .
For a MDP E , we deﬁne P ¯ E (cid:96) h ( ·| s, a ) = E (cid:96) [ P E h ( ·| s, a )] as themean of posterior measure for each state-action pair ( s, a ) and write ¯ E (cid:96) as the MDP consisted by{ P
[Equation] h } Hh =1 .
Learning the whole environment
The core design of IDS for RL relies on a notion of information ratio .
The information ratio for apolicy π at episode (cid:96) is deﬁned as
[Equation]
where χ is the learning target to prioritize information sought by the agent.
The choice of χ plays acrucial role in designing the IDS and could lead to different regret bounds and computational methods.
We ﬁrst consider the most natural choice of χ which is the whole environment E .
3.1 Vanilla IDS
Vanilla-IDS takes the whole environment E as the learning target and at the beginning of eachepisode, the agent computes a stochastic policy:
[Equation]
Deﬁne the worst-case information ratio Γ ∗ such that Γ (cid:96) ( π (cid:96) IDS ) ≤ Γ ∗ for any (cid:96) ∈ [ L ] almost surely.
The next theorem derives a generic regret bound for vanilla-IDS in terms of Γ ∗ and the mutualinformation between E and the history.
Theorem 3.1.
A generic regret bound for vanilla-IDS is
[Equation]
The proof is deferred to Appendix A.1 and follows standard information-theoretical regret decomposition and the chain rule of mutual information that originally was exploited by Russo and Van Roy[2014].
For tabular MDPs, it remains to bound the E [Γ ∗ ] and I ( E ; D L +1 ) separately.
Lemma 3.2.
The worst-case information ratio for tabular MDPs is upper bounded by
[Equation]
We sketch the main steps of the proof and defer the full proof to Appendix A.2.
Proof sketch.
Since vanilla-IDS minimizes the information ratio over all the policies, we canbound the information ratio of vanilla-IDS by the information ratio of TS.
• Step one.
Our regret decomposition uses the value function based on ¯ E (cid:96) as a bridge: [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Table]
[Table]
[Table] [Table] [Equation] [Table]
• Step two.
Denote [Equation]
[Table]
[Equation]
[Table]
[Equation] ] as
the value function difference.
With the use of state-action occupancy measure and Lemma
D.3, we can derive
[Equation]
Applying the Cauchy–Schwarz inequality and Pinsker’s inequality (see Eqs.
(A.2)-(A.4) inthe appendix for details), we can obtain
[Equation]
where we interchange π (cid:96) TS and π ∗ again and E ¯ E (cid:96) π (cid:96) TS is taken with respect to s (cid:96)h , a (cid:96)h and E (cid:96) istaken with respect to π (cid:96) TS and E .•
Step three.
It remains to establish the following equivalence of above KL-divergence andthe information gain (Lemma A.1):
[Equation]
A crucial step is to use the linearity of the expectation and the independence of priors overdifferent layers (from the product prior as we assumed in Section 2) to show
[Equation]
Combining Steps 1-3, we can reach the conclusion and the bound for I 2 is similar.
The next lemma directly bounds the mutual information for tabular MDPs.
Lemma 3.3.
The mutual information can be bounded by
[Equation]
The proof relies on the construction of Bayes mixture density and a covering set for KL-divergenceand is deferred to Appendix A.3.
Combining Theorem 3.1, Lemmas 3.2 and 3.3 yields the following:
Theorem 3.4 (Regret bound for tabular MDPs) .
Suppose π IDS
=
π (cid:96) IDS } L(cid:96)
=1 is the vanilla IDS policy.
The following Bayesian regret bound holds for tabular MDPs
[Equation]
Although this regret bound is sub-optimal, this is the ﬁrst sub-linear prior-free Bayesian regret boundfor vanilla-IDS .
Remark 3.5.
It is worth mentioning that Lu and Van Roy [2019], Lu [2020] also derived Bayesianregret bound using information-theoretical tools but only hold for a speciﬁc Dirichlet prior aswell other distribution-speciﬁc assumptions.
Their proof heavily exploits the property of Dirichletdistribution and can not easily be extended to prior-free regret bounds.
In the context of ﬁnite-horizon MDPs, Lu et al.
[2021] considered a conditional-IDS such that ateach time step, conditional on s (cid:96)h , conditional-IDS takes the action according to
[Equation]
, where A h is sampled from ν .
Conditional-IDS deﬁned the information ratio per-step rather thanper-episode such that it only needs to optimize over action space rather than the policy space.
Thisoffers great computational beneﬁts but there is no regret guarantee for conditional-IDS .
Recently,
Hao et al.
[2022] has demonstrated the theoretical limitation of conditionar-IDS in contextualbandits.
3.2 Regularized IDS
Computing an IDS policy practically usually involves two steps: 1.
approximating the informationratio ; 2.
optimizing the information ratio .
In bandits where the optimal policy is only a function ofaction space, optimizing Eq.
(3.2) is a convex optimization problem and has an optimal solution withat most two non-zero components (Russo and Van Roy [2018, Proposition 6]).
However in MDPswhere the optimal policy is a mapping from the state space to the action space, vanilla-IDS needsto traverse two non-zero components over the full policy space which suggests the computationaltime might grow exponentially in S and H .
To overcome this obstacle, we propose regularized-IDS that can be efﬁciently computed by anydynamic programming solver and enjoy the same regret bound as vanilla-IDS .
At each episode (cid:96) ,regularized-IDS ﬁnds the policy:
[Equation]
where λ > 0 is a tunable parameter.
To approximate the objective function in Eq.
(3.3), we assume the access to a posterior samplingoracle .
Deﬁnition 3.6 (Posterior sampling oracle) .
Given a prior over E and history D (cid:96) , the posteriorsampling oracle , SAMP, is a subroutine which returns a sample from the posterior distribution P (cid:96) ( E ) .
Multiple calls to the procedure result in independent samples.
Remark 3.7.
SAMP can be exactly obtained when the conjugate prior such as Dirichlet distributionis put on the transition kernel.
When one uses neural nets to estimate the model, SAMP can beapproximated by epistemic neural networks [Osband et al., 2021a], a general framework to quantifyuncertainty for neural nets.
The effectiveness of different epistemic neural networks such as deepensemble, dropout and stochastic gradient MCMC has been examined empirically by Osband et al.[2021b].
We compute π (cid:96) r-IDS in two steps:
• Firstly, we prove an equivalent form of the objective function in Eq.
(3.3) using the chain
rule of mutual information.
Deﬁne r (cid:48) h ( s, a ) as an augmented reward function:
[Equation]
Proposition 3.8.
The following equivalence holds
[Equation]
=
[Equation]
[Equation]
The proof is deferred to Appendix A.4.
• Given SAMP , the augmented reward r (cid:48) h and the MDP ¯ E (cid:96) can be well approximated by Monte
Carlo sampling.
Therefore, at each episode (cid:96) , ﬁnding
π (cid:96) r
IDS is equivalent to ﬁnd an optimal
policy based on a computable and augmented MDP { P ¯ E (cid:96) h , r (cid:48) h
} Hh =1 .
This can be solvedefﬁciently by
any dynamic programming solver such as value iteration or policy iteration.
In the end, we show that π (cid:96) r-IDS enjoys the same regret bound as vanilla-IDS when the tunableparameter is carefully chosen.
Theorem 3.9.
By choosing λ = (cid:112) L E [Γ ∗ ] / I ( E ; D L +1 ) , we have
[Equation]
The proof is deferred to Appendix A.5.
Let M 1 , M 2 be upper bounds of E [Γ ∗ ] and I ( E ; D L +1 )respectively.
In practice, we could conservatively choose λ
= (cid:112) LM 1 /M
such that BR L ( π r-IDS ) ≤ 215 (cid:112) 3 / 2 M 1 M 2 L .
From Lemmas 3.2 and 3.3 for tabular MDPs, we could choose M 1 = 2 SAH 3 and
M 2 = 2 S 2 AH log( SLH ) .
Remark 3.10.
Russo and Van Roy [2018, Section 9.3] also considered a tunable version of IDS (forbandits) but took a square form of E (cid:96) [ V E 1 ,π ( s (cid:96) 1 )] .
While this makes no difference in bandits setting, thisprevented us to use dynamic programming solver in RL setting.
We are also inspired by Foster et al.[2021, Section 9.3] who studied the relationship between information ratio and Decision-Estimation
Coefﬁcient.
Learning a surrogate environment
When the state space is large, the agent could take too much information to learn exactly the wholeenvironment E which is reﬂected through I ( E ; D L +1 ) .
A key observation is that different states maycorrespond to the same value function who eventually determines the behavior of the optimal policy.
Based on the rate-distortion theory developed in Dong and Roy [2018], we reduce this redundancyand construct a surrogate environment that needs less information to learn.
4.1 A rate distortion approach
The rate-distortion theory [Cover and Thomas, 1991] addresses the problem of determining theminimal number of bits per symbol that should be communicated over a channel, so that the source(input signal) can be approximately reconstructed at the receiver (output signal) without exceeding anexpected distortion.
It was recently introduced to bandits community to develop sharper bounds forlinear bandits [Dong and Roy, 2018] and time-sensitive bandits [Russo and Van Roy, 2022].
We takea similar approach to construct a surrogate environment.
Surrogate environment Suppose there exists a partition { Θ k } Kk =1 over Θ such that for any E , E (cid:48) ∈Θ k and any k ∈ [ K ] , we have [Equation]
[Equation]
where ε > 0 is the distortion tolerance and we write the optimal policy explicitly depending on theenvironment.
Let ζ be a discrete random variable taking values in { 1 , . .
, K } that indicates theregion E lies such that ζ
= k
if and only if E ∈ Θ k .
Therefore, ζ can be viewed as a statistic of E andless informative than E if K is small.
The next lemma shows the existence of the surrogate environment based on the partition.
Lemma 4.1.
For any partition { Θ k } Kk =1 and any (cid:96) ∈ [ L ] , we can construct a surrogate environment(cid:101)
E ∗ (cid:96) ∈ Θ which is a random MDP such that the law of (cid:101) E ∗ (cid:96) only depends on ζ and
[Equation]
The concrete form of (cid:101) E ∗ (cid:96) is deferred to Eq.
(B.1) in the appendix.
Surrogate IDS
We refer the IDS based on the surrogate environment (cid:101) E ∗ (cid:96) as surrogate-IDS thatminimizes
[Equation]
for some parameters ε > 0 the will be chosen later.
Denote the surrogate information ratio of TS as
[Equation]
We ﬁrst derive a generic regret bound for surrogate IDS in the following theorem.
Theorem 4.2.
A generic regret bound for surrogate IDS is
[Equation]
We defer the proof to Appendix B.2.
Given ζ , we have (cid:101) E ∗ (cid:96) and H (cid:96),H are independent under the law of P (cid:96),π (cid:96) s-IDS .
By the data processing inequality, the proof uses the fact that
[Equation]
[Equation]
Comparing with regret bound of vanilla-IDS in Lemma 3.1, the regret bound of surrogate-IDSdepends on the information gain about ζ rather than the whole environment E .
If there exists apartition with small covering number K , the agent could pay less information to learn.
The secondterm Lε is the price of distortions.
In the following, we will bound the E [ (cid:101) Γ] and I ( ζ ; D L +1 ) for tabular and linear MDPs separately.
4.2 Tabular MDPs
We ﬁrst show the existence of the partition required in Lemma 4.1 for tabular MDPs and an upperbound of the covering number K .
Lemma 4.3.
There exists a partition { Θ k } Kk =1 over Θ such that for any k ∈ [ K ] and E 1 , E 2 ∈ Θ k ,
[Equation]
and the log covering number satisﬁes log( K ) ≤ SAH log(4 H 2 / [Equation]
The proof is deferred to Lemma B.3.
For tabular MDPs, the mutual information between ζ and thehistory can be bounded by
[Equation]
where H ( · ) is the Shannon entropy.
Comparing with Lemma 3.3 when learning the whole environment,learning the surrogate environment saves a factor of S through the bound of mutual information.
Lemma 4.4.
The surrogate information ratio for tabular MDPs is upper bounded by [Equation]
The proof is the same as Lemma 3.2 and thus is omitted.
Putting Lemmas 4.3-4.4 yields an improvedbound for tabular MDPs using surrogate-IDS .
Theorem 4.5 (Improved regret bound for tabular MDPs) .
By choosing ε = 1 /L , the regret bound ofsurrogate-IDS for tabular MDPs satisﬁes
[Equation]
For tabular MDPs, surrogate-IDS improves the regret bound of vanilla-IDS by a factor of S .
However, it is still away from the minimax lower bound by a factor of √ SAH .
We conjecturesurrogate-IDS can achieve the optimal bound with a price of lower order term but leave it as afuture work.
Remark 4.6.
Although the existence of (cid:101) E ∗ (cid:96) is established using a constructive argument, ﬁnding (cid:101) E ∗ (cid:96)needs a grid search and is not computationally efﬁcient.
4.3 Linear MDPs
We extend our analysis to linear MDPs that is a fundamental model to study the theoretical propertiesof linear function approximations in RL.
All the proofs are deferred to Appendix B.4-B.5.
Deﬁnition 4.7 (Linear MDPs [Yang and Wang, 2019, Jin et al., 2020]) .
Let φ : S ×
A → R d bea feature map which assigns to each state-action pair a d -dimensional feature vector and assume(cid:107) φ ( s, a ) (cid:107) 2 ≤ 1 .
An MDP is called a linear MDP if for any h ∈
[ H ] , there exist d unknown (signed)measures ψ 1 h , . .
. , ψ dh over S , such that for any ( s, a ) ∈ S × A , we have
[Equation]
where ψ h = ( ψ 1 h , . . . , ψ dh ) .
Let us denote Θ
Lin be the parameter space of linear MDPs and assume(cid:107) (cid:80) s (cid:48)
ψ h ( s (cid:48) )
(cid:107) 2 ≤ C ψ .
Note that the degree of freedom of linear MDPs still depends on S which implies that I ( E ; D L +1 )may still scale with S .
Therefore, we must learn a surrogate environment rather than the wholeenvironment for linear MDPs based on the current regret decomposition in Theorem 3.4.
We ﬁrstshow the existence of a partition over linear MDPs with the log covering number only depending onthe feature dimension d .
Lemma 4.8.
There exists a partition { Θ k } Kk =1 over Θ Lin such that for any k ∈ [ K ] and E 1 , E 2 ∈ Θ k ,
[Equation]
and the log covering number satisﬁes log( K ) ≤ Hd log( H 2 C ψ /ε + 1) .
For linear MDPs, the mutual information can be bounded by
[Equation]
Lemma 4.9.
The surrogate information ratio of linear MDPs is upper bounded by E [ (cid:101) Γ] ≤ 4 H 3 d .
Theorem 4.10 (Regret bound for linear MDPs) .
By choosing ε = 1 /L , the regret bound of surrogate
IDS for linear MDPs satisﬁes
[Equation]
This bound improves the O ( d 3 / 2 H 2 √ L ) regret of LSVI-UCB [Jin et al., 2020] by a factor of √ dand matches the existing minimax lower bound O ( √ H 3 d 2 L )
[Zhou et al., 2021] up to a H factor.
4.4 Regret bounds for TS
As a direct application of our rate-distortion analysis, we provide Bayesian regret bounds for Thompson sampling.
Theorem 4.11.
A generic regret bound for TS is
[Equation]
This implies for tabular and linear MDPs, TS has the same regret bound as surrogate-IDS .
Notethat the computation of TS does not need to involve the surrogate environment (cid:101) E ∗ (cid:96) so once theposterior sampling oracle is available, computing the policy is efﬁcient.
Howevern when the worstcase information ratio cannot be optimally bounded by the information ratio of TS, IDS demonstratesbetter regret bounds than TS, such as bandits with graph feedback [Hao et al., 2022] and sparse linearbandits [Hao et al., 2021].
Conclusion
In this paper, we derive the ﬁrst prior-free Bayesian regret bounds for information-directed RL undertabular and linear MDPs.
Theoretically, it will be of great interest to see if any version of IDS canachieve the [Table]
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[N/A]
• Did you include the license to the code and datasets?
[N/A]• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors.
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A](b)
Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[N/A]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
A Proofs of learning the whole environment
A.1 Proof of Theorem 3.
Proof.
We follow the standard information-theoretical regret decomposition:
[Equation]

[Equation]
where the ﬁrst inequality is from the Cauchy–Schwarz inequality and the last equation is due to thechain rule of mutual information
[Equation]
According to the deﬁnition of Γ ∗ , we have Γ (cid:96) ( π (cid:96) IDS ) ≤ Γ ∗ for any (cid:96) ∈ [ L ] .
A.2 Proof of Lemma 3.
Proof.
From the deﬁnition of IDS policy stated in Eq.
(3.2), for any (cid:96) ∈ [ L ] ,
[Equation]
We ﬁrst decompose the one-step regret as follows,
[Equation]
Bounding I
Note that conditional on D (cid:96) , the law of π (cid:96) TS is the same as the law of π ∗ and both π ∗and π (cid:96) TS are independent of ¯ E (cid:96) .
This fact implies that
[Equation]
Denote
[Equation]
Applying Lemma D. [Equation] we have
[Equation]
Here we just divide the element that corresponds to d ¯ E (cid:96) h,π ∗ ( s, a ) > 0 .
Applying Cauchy–Schwarz inequality and using the fact that d ¯ E (cid:96) h,π ∗ ( s, a ) ≤ 1 , we have
I 1 ≤ 
[Equation]
   
[Equation]
 
[Equation]

[Equation]

   1 / [Equation]  
[Equation]

 

[Equation]
 
[Equation]

[Equation]
E [Equation]

 

[Equation]
A
[Equation]
First, note that
[Equation]

[Equation]

[Equation]
Second, since the law of π ∗ is the same as π (cid:96) TS conditional on D (cid:96)
[Equation]
we have
[Equation]
Then we can obtain
[Equation] 
[Equation]
 
[Equation]

[Equation]
A.4) where E ¯ E (cid:96) π (cid:96) TS is taken with respect to s (cid:96)h , a (cid:96)h and E (cid:96) is taken with respect to π (cid:96) TS and E .
Combining Eqs.
(A.2)-(A.4) together and applying Lemma D.2 which is a variant of Pinsker’sinequality,
[Equation]
The next lemma establishes the relationship between the cumulative KL-distance in Eq.
(A.5) andmutual information I π (cid:96) TS (cid:96) ( E ; H (cid:96),H ) .
This lemma will be frequently used later.
Lemma A.1.
For environment E and its corresponding mean of posterior measure ¯ E (cid:96) , the followingholds for any policy π ,
[Equation]
Together with Eq.
(A.5),
[Equation]
Bounding I
The upper bound for I 2 is similar.
Applying Lemma D.3 and repeating the steps ofshowing Eq.
(A.5),
[Equation]
Applying Lemma A.1 again, we obtain
[Equation]
Combining Eqs.
(A.1), (A.6) and (A.7) together, we have
[Equation]
[Table]
[Equation]
[Table]
This ends the proof.
A.3 Proof of Lemma 3.3
Proof.
Now we start to bound I ( E ; D L +1 ) .
Assume the prior probability measure of E is ρ ( E ) thattakes the value in Θ and suppose the Bayes mixture density p
ρ ( D L +1 )
= (cid:82) E∈
Θ p ( D L +1 |E ) dρ ( E ) .
Let { Θ k } Kk =1 be a partition of Θ .
We choose ρ 1 as an uniform distribution over { Θ k } Kk =1 suchthat q ( D L +1 ) =
p ρ 1 ( D L +1 )
= [Equation] θ ∈ Θ p ( D L +1 | θ ) d ρ 1 ( θ ) and we denote Q D L +1 as the correspondingprobability measure.
Based on the equivalent form of mutual information,
[Equation]
where the inequality is due to the fact that Bayes mixture density p ρ ( D L +1 )
minimizes the average
KL divergences over any choice of densities q ( D L +1 ) .
According to the deﬁnition of the KL-divergence term,
[Equation]
By the chain rule of KL-divergence,
[Equation]
It remains to construct a partition of { Θ k } Kk =1 such that for any E ∈ Θ , there always exists (cid:101)
E ∈ Θ ksuch that for any s, a, h ,
[Equation]
The existence of such partition is shown in the following theorem.
Theorem A.2 (Divergence covering number) .
For any 0 < ε < 1 , suppose N ( k, ε ) is a divergencecovering set over k -dimensional probability simplex such that for any P , there exists (cid:101) P ∈ N ( k, ε )such that
[Equation]
Then there exists such set whose covering number can be bounded by
[Equation]
The proof could be found in Tang and Polyanskiy [2021, Theorem 4].
For each s, a, h , we constructsuch divergence covering set N hsa ( S, ε ) and the partition { Θ k } Kk =1 is constructed such that E ∈
Θ k iffor any s, a, h , P E h ( ·| s, a ) ∈ N hsa
( S, ε ) .
According to Theorem A.2,
[Equation]
Therefore, together with Eq.
(A.9),
[Equation]
By choosing ε = 1 /LH ,
[Equation]
This ends the proof.
A.4 Proof of Proposition 3.
According to the linearity of expectation and independence of priors over different layers, we canobtain E (cid:96) [ V E 1 ,π ( s (cid:96) 1 )]
=
V ¯ E (cid:96) 1 ,π ( s (cid:96) 1 ) which implies [Equation] h [Equation]
r
[Equation]
s
[Equation]
(cid:35) .
According to the proof of Lemma A.1 in Appendix [-D.
-]{+C.
+}
[Equation]
Combining them together,
[Equation]
A.5 Proof of Theorem
Proof.
Using the fact that 2 ab ≤ a 2 + b 2 , we have for any policy π
[Equation]
This implies
[Equation]
We follow the regret decomposition as
[Equation]
where Γ ∗ is the worst-case information ratio.
Letting λ
= (cid:112)
L E [Γ ∗ ] / I ( E ; D L +1 ) , we have
BR L ( π r-IDS ) ≤ (cid:114)
[Equation]
E [Γ ∗ ] I ( E
[Equation]
D L +1 ) .
This ends the proof.
B Proofs of learning a surrogate environment
B.1 Proof of Lemma 4.
Proof.
Conditional on D (cid:96) , let E (cid:96) be an independent sample of E .
Note that
[Equation]
where the last equation is due to the independence between E (cid:96) and E .
For each k ∈ [ K ] , according to
Lemma D.1, there exists E k,(cid:96) 1 , E k,(cid:96)∈ Θ k and r k,(cid:96) ∈ [0 , 1] such thatr k
[Equation]
E
[Equation]
The surrogate learning target (cid:101) E ∗ (cid:96) is a random variable such that
[Equation]
This implies the law of (cid:101) E ∗ (cid:96) only depends on ζ and conditional on ζ , (cid:101) E ∗ (cid:96) is independent of E .
We also need to ensure learning towards (cid:101) E ∗ (cid:96) will not occur too much additional regret.
Let (cid:101) E (cid:96) be anindependent sample of (cid:101) E ∗ (cid:96) .
From the law of total expectations,
[Equation]
≤ 0 .
On the other hand, we have E [Equation] [ V [Equation]
π ∗E ( s
[Equation]
=
E
[Equation] [Table] [Equation] )] and this implies
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] When E ∈ Θ k , then ζ = k which implies (cid:101) E ∗ (cid:96) ∈
Θ k either.
That means E and (cid:101) E ∗ are in the samepartition and [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
Putting Eqs.
(B.2)-(B.3) together,
[Table]
[Equation]
[Table]
[Equation]
Noticing that E (cid:96) [ V [Equation] 1 ,π ∗E ( s [Equation]
= E [Table] [ V
[Table]
[Equation]
[Table]
[Equation]
π
[Table]
[Equation] s [Table] 1 )] , this ends the proof.
B.2 Proof of
Theorem
[Table]
Proof.
We decompose
BR
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
From the deﬁnition of π (cid:96) s-IDS in Eq.
BR
[Table]
+
[Table]
According to Lemma 4.1,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
For any (cid:96) ∈ [ L ] , conditional on ζ , we have (cid:101) E ∗ (cid:96) and H (cid:96),H are independent under the law of P (cid:96),π (cid:96) s-IDS .
By the data processing inequality, we have
[Equation]
; H
[Equation]
Therefore,
[Equation]
[Table]
[Equation]

[Table]
This ends the proof
[Table]
B.3 Proof of Lemma 4.3
Proof.
We construct a partition over Θ such that Eq.
(4.1) holds.
For any E 1 , E 2 ∈ Θ k , we use
Lemma D.3,
[Equation]
The construction follows the following steps.
• First, we construct a cover for I 1 .
Since the reward is assumed to be bounded by 1, we have
P E h [Equation] ·| s [Equation]
, a [Equation] [Equation] V E h
+1 ,π ∗E ( · ) ∈ [0 , H ] for any E .
For each ( s, a, h ) , we construct a covering set
{I 1 sah , . .
I msah } for [0 , 1] where m = H/ε .
Thus, each set is of length ε .
• Second, let {C 1 , . . .
, C M } be an ε -covering of a S -dimensional (cid:96) 2 -ball.
• Third, we construct the partition { Θ k } Kk =1 in the way that E ∈ Θ k if for any s, a, h , [Equation] P E h ( ·| s, a ) , V E h [Equation]
1 ,π
[Equation]
I
[Equation]
where k 1 ∈ [ m ] , k 2 ∈
[ M ]
The existence of k 1 , k 2 holds trivially.
Apparently, { Θ k } Kk =1 is a partition of Θ .
For any k ∈ [ K ] and E 1 , E 2 ∈ Θ k , the following holds forany s, a, h ,
[Equation]
and
[Equation]
Therefore, we have constructed a partition { Θ k } Kk =1 over Θ such that for any E 1 , E 2 ∈ Θ k ,
[Equation]
with the covering number bounded by
[Equation]
This ends the proof.
B.4 Proof of Lemma 4.8
Proof.
We construct a partition { Θ k } Kk =1 over Θ Lin such that Eq.
(4.1) holds.
For any E 1 , E 2 ∈ Θ k ,by Lemma D.3,
[Equation]
where the last equation is from the deﬁnition of linear MDP.
Moreover, since the value function isalways bounded by H , we have
[Equation]
ψ E 1 h ( s (cid:48) ) − (cid:88) s (cid:48)
ψ E 2 h ( s (cid:48) ) (cid:33)(cid:35) ≤ H H (cid:88) h =
E
[Equation]
s (cid:96)h
[Equation]
) (cid:13)(cid:13)
[Equation]
(cid:3) (cid:13) (cid:13) (cid:13)(cid:13)(cid:13)(cid:88)
s
[Table]
ψ
[Table]
h
[Table]
s
[Table]
−
[Table]
s
[Table]
h
[Table]
≤ H
H
[Table]
h =
[Table]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88) s
[Table]
ψ
[Table]
h
[Table]
s
[Table]
−
[Table]
s
[Table]
h
[Table]
) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Table] From Deﬁnition 4.7,(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88) s (cid:48) ψ [Table]
h ( s
[Table]
) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ C ψ .
For each h ∈ [ H ] , let {C h 1 , . . .
, C hK } be an ε -covering of a d -dimensional (cid:96) 2 -ball.
We construct thepartition { Θ k } Kk =1 in the way that E ∈ Θ k if
[Equation]
Apparently, { Θ k } Kk =1 is a partition of Θ .
For any k ∈ [ K ] and E 1 , E 2 ∈ Θ k , the following holds
[Equation]
which implies
[Equation]
Letting ε (cid:48)
=
H 2 C ψ ε , the covering number can be bounded by
[Equation]
This ends the proof.
B.5 Proof of Lemma 4.
Proof.
We write ¯ E ∗ (cid:96) as the MDP consisted by the mean of posterior measure of (cid:101) E ∗ (cid:96) .
Noting that
E (cid:96) [ V (cid:101) E (cid:96) 1 ,π ∗E ( s [Equation]
=
E (cid:96) [ V
[Equation] E ∗ [Equation] 1 ,π (cid:96) TS ( s (cid:96) 1 )] , we decompose the regret as
[Equation]
[Table]
According to Lemma D.3,
[Table]
Using the deﬁnition of linear MDPs in Deﬁnition 4.7,
[Table]
Denoting
[Table]
we have
[Table]
By Cauchy–Schwarz inequality, we have
[Table]
[Equation]
[Table] [Equation] [Table] [Table]
[Equation]
Applying Lemma A
[Equation]
[Equation]
we have
[Equation]
• For the second part, we can rewrite
[Equation]
Therefore, putting them together,
[Equation]
The derivation of bounding I 2 is similar so we omit the detailed proof here.
This ends the proof.
Directly using Lemma 4.1, we can decompose
[Table]
By the construction of (cid:101) E ∗ (cid:96) , (cid:101) E ∗ (cid:96) and E are independent conditional on ζ .
Thus (cid:101) E ∗ (cid:96) and H (cid:96),H areindependent under the law P [Equation] ,π [Equation]
TS given ζ .
By the data processing inequality, we have
[Equation]
[Equation]
Therefore,
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
This ends the proof.
C Proofs of technical lemmas
C.1 Proof of Lemma A.
Proof.
Using the chain rule of mutual information,
[Equation]
+ [Equation] For
the ﬁrst term in Eq.
(C.1),
by
the deﬁnition
of conditional mutual information and
Markov property [Equation] we have
[Equation]
where P (cid:96) ( E ) is the posterior distribution of E based on D (cid:96) .
Since the priors of transition kernel are independent over different layers, P (cid:96),π ( s (cid:96)h = ·|H
[Equation]
1 ) is the posterior distributionof P E h ( ·| s (cid:96)h − 1 , a (cid:96)h − 1 ) only based on D (cid:96) but not H (cid:96),h − 1 , or equivalently,
[Equation]
Combining Eqs.
(C.2) and (C.3) together,
[Equation]
h
[Equation]
Therefore,
[Equation]
Using the linearity of expectation and the independence of priors over different layers
[Equation]
wecan show
[Equation]
This implies
[Equation]
where E ¯ E (cid:96) π is taken with respect to s (cid:96)h − 1 , a [Equation] − 1 and E (cid:96) is taken with respect to E .
• For the second term of Eq.
(C.1), we have
[Equation]
When H π(cid:96),h − 1 is ﬁxed, both sides of the KL term are equal to π ( ·| s (cid:96)h ) and thus
I π(cid:96) ( E ; a (cid:96)h | s (cid:96)h , H (cid:96),h − 1 ) = 0 .•
For the third term of Eq.
(C.1), since the reward function is deterministic and known, wehave
[Equation]
This sufﬁces to show
[Equation]
This ends the proof.
D Supporting lemmas
Lemma D.1 (Lemma 1 in Dong and Roy [2018]) .
Let { a i } Ni =1 and { b i } Ni =1 be two sequences of realnumbers, where N < ∞ .
Let { p i } Ni
=1 be such that p i ≥
for all i and (cid:80) Ni =1 p i = 1 .
Then thereexists indices j, k ∈ [ N ] and r ∈ [0 , 1] such that
[Equation]
Lemma D.2 (Fact 9 in Russo and Van Roy [2014]) .
For any distribution P and Q such that P isabsolutely continuous with respect to Q , any random variable X : Ω → X and any g : X → R suchthat sup g − inf g ≤ 1 , we have
[Equation]
which is a variant of Pinsker’s inequality.
Lemma D.3.
For any two environments E , E (cid:48) , any policy π and a ﬁxed set of reward functions{ r h } Hh =1 , we have
[Equation]
where we deﬁne V E H +1 ,π ∗ ( · ) = 0 and the outer expectation E E
[Equation]
π is with respect to s h , a h .
Proof.
Similar proof can be found in Osband et al.
[2013], Foster et al.
For the selfcompleteness, we include a full proof here.
First, we realize
[Equation]
[Equation]
Since V E h,π ( s ) =
E a ∼ π h ( ·| s )
[ Q E h,π ( s, a )] , we have
H
[Equation]
D
[Equation]
[Equation]
Using the Bellman equation, we have
[Equation]
This ends the proof
[Equation]