O NLINE T ARGET Q- LEARNING WITH R EVERSE E XPERIENCE R EPLAY : E FFICIENTLY FINDING THE O PTIMAL P OLICY FOR L INEAR MDP S
BSTRACT
Q-learning is a popular Reinforcement Learning (RL) algorithm which is widely deployed with function approximation (Mnih et al., 2015).
In contrast, existing theoretical results are pessimistic about Q-learning.
For example, Q-learning does not converge even with linear function approximation for linear MDPs ((Baird, 1995)) and even for tabular MDPs with synchronous updates, Q-learning has suboptimal sample complexity (Li et al., 2021; Azar et al., 2013).
The goal of this work is to bridge the gap between practical success of Q-learning and the relatively pessimistic theoretical results.
The starting point of our work is the observation that in practice, Q-learning is used with two important modiﬁcations: (i) training with two networks, called online network and target network simultaneously (online target learning, or OTL) , and (ii) experience replay (ER) (Mnih et al., 2015).
While they play a signiﬁcant role in the practical success of Q-learning, a thorough theoretical understanding of how these two modiﬁcations improve the convergence behavior of Q-learning has been missing in literature.
By carefully combining Q-learning with OTL and reverse experience replay (RER) (a form of experience replay), we present novel methods Q-Rex and Q-RexDaRe (Q-Rex+ data reuse).
We show that Q-Rex efﬁciently ﬁnds the optimal policy for linear MDPs (or more generally for MDPs with zero inherent Bellman error with linear approximation (ZIBEL)) and provide
non-asymptotic bounds on sample complexity – the ﬁrst such result for a Q-learning method for this class of MDPs under standard assumptions.
Furthermore, we demonstrate that Q-RexDaRe in fact achieves near optimal sample complexity in the tabular setting, improving upon the existing results for vanilla Q-learning.
I NTRODUCTION
Reinforcement Learning (RL) has been shown to be highly successful in practice for a variety of long term decision making problems (Mnih et al., 2015).
Several classical works have studied RL methods like TD-learning, Q-learning and their variants (Sutton & Barto, 2018; Bertsekas, 2011; Borkar & Meyn, 2000; Sutton, 1988; Tsitsiklis & Van Roy, 1997; Watkins & Dayan, 1992; Watkins, 1989) but the guarantees are mostly asymptotic and therefore do not sufﬁciently answer important questions that are relevant to practitioners who struggle with constraints on the number of data points and the computation power.
Recent works provide non-asymptotic results for a variety of important settings (Kearns & Singh, 1999; Beck & Srikant, 2012; Qu & Wierman, 2020; Ghavamzadeh et al., 2011; Bhandari et al., 2018; Chen et al., 2020; 2019; Dalal et al., 2018a;b; Doan et al., 2020; Gupta et al., 2019; Srikant & Ying, 2019; Weng et al., 2020; Yang & Wang, 2019; Zou et al., 2019a).
Despite a large body of work, several aspects of fundamental methods like Q-learning (Watkins &
Dayan, 1992) are still ill-understood.
Q-learning’s simplicity and the ability to learn from off-policy data makes it widely applicable.
However, theoretical analyses show that even with linear function approximation and when the approximation is exact , Q-learning can fail to converge even in simple examples (Baird, 1995; Boyan & Moore, 1995; Tsitsiklis & Van Roy, 1996).
Furthermore, even in the simple case of tabular RL with synchronous updates, Q-learning is known to have sub-optimal sample complexity (Wainwright, 2019a; Li et al., 2021).
However, Q-learning has seen tremendous practical success when deployed with “heuristic” modiﬁcations like experience replay (ER) and online target learning (OTL). ER is used to alleviate the issues arising due highly dependent samples in an episode whereas OTL helps stabilize the Q iteration.
Mnih et al.
(2015) conducted extensive experiments to show that both these techniques, are essential for the success of Q-learning.
But, existing analyses for ER with Q-learning either require stringent assumptions (Carvalho et al., 2020) to ensure convergence to a good Q value, or assume that ER provides i.i.d.
samples which might not hold in practice (Fan et al., 2020; Carvalho et al., 2020).
In this paper, we attempt to bridge the gap between theory and practice, by rigorously investigating how Q-learning performs with these practical heuristics.
We thus introduce two model free algorithms: Q-Rex and Q-RexDaRe that combine the standard Q-learning with OTL and reverse experience replay (RER).
RER is a form of ER which was recently studied to unravel spurious correlations present while learning form Markovian data in the context of system identiﬁcation (Jain et al., 2021b).
We show that OTL stabilizes the Q value by essentially serving as a variance reduction technique and RER unravels the spurious correlations present in the Markovian data to remove inherent biases introduced in vanilla Q learning.
These simple modiﬁcations have surprisingly far-reaching consequences.
Firstly, this allows us to show that unlike vanilla Q-learning, Q-Rex ﬁnds the optimal policy for MDPs with an exact linear function representation of the Bellman operator. and allows us to derive non-asymptotic sample complexity bounds.
In the tabular setting , Q-Rex even with asynchronous data is able to match the best known bounds for Q-learning with synchronous data.
Its variant Q-RexDaRe , which reuses old samples, admits nearly optimal sample complexity for recovering the optimal Q-function in the tabular setting.
Previously, only Q-learning methods with explicit variance-reduction techniques (not popular in practice)
(Wainwright, 2019b; Li et al., 2020b) or model based methods (Agarwal et al., 2020; Li et al., 2020a) were known to achieve such a sample complexity bound.
Our experiments show that when the algorithmic parameters are chosen carefully, Q-Rex and its variants outperform both vanilla Q-learning and OTL+ER+Q-learning with the same parameters (see Appendix A).
To summarize, in this work, we study Q-learning with practical heuristics like ER and OTL, and propose two concrete methods Q-Rex and Q-RexDaRe based on OTL and reverse experience replay – a modiﬁcation of the standard ER used in practice.
We show that Q-Rex is able to ﬁnd the optimal policy for ZIBEL MDPs, with a strong sample complexity bound which is the ﬁrst such result for Q-learning.
We also show that Q-RexDaRe obtains nearly optimal sample complexity for the simpler tabular setting despite not using any explicit variance reduction technique.
See Tablefor a comparison of our guarantees against the state-of-the-results for the tabular setting.
Organization We review related works in next subsection.
In Section 2 we develop the MDP problem which we seek to solve and present our algorithm, Q-Rex in Section 3.
The main theoretical results are presented in Section 4.
We present a brief overview of the analysis in Section 5 and present our experiments in Section A.
We provide minimax lower bounds for the asynchronous tabular setting in Section K. Most of the formal proofs are relegated to the appendix.
1. R ELATED W ORKS
Tabular Q-learning Tabular MDPs are the most basic examples of MDPs where the state space ( S ) and the action space ( A ) are both ﬁnite and the Q-values are represented by assigning a unique co-ordinate to each state-action pair.
This setting has been well studied over the last few decades and convergence guarantees have been derived in both asymptotic and non-asymptotic regimes for popular model-free and model-based algorithms.
Azar et al.
(2013) shows that the minimax lower bounds on the sample complexity of obtaining the optimal Q-function up-to (cid:15) error is |S||A| (1 − γ ) 3 (cid:15) 2 , where γ is the discount factor.
Near sample-optimal estimation is achieved by several model-based algorithms (Agarwal et al., 2020; Li et al., 2020a) and model-free algorithms like variance reduced Q-learning (Wainwright, 2019b; Li et al., 2020b).
(Li et al., 2021) also shows that vanilla Q-learning with standard step sizes, even in the synchronous data setting – where transitions corresponding to each state [Table] [Table] [Table]
γ ) 5 (cid:15) 2 . These results seem unsatisfactory since γ [-∼ 0 . 
 (or even 0-]∈ (0 . 99 , 1) in most practical applications.
In contrast, our algorithm Q-Rex with asynchronous data has a sample complexity that matches Q-learning bound with synchronous data and its data-efﬁcient variant Q-RexDaRe has near minimax optimal sample complexity (see Table 1).
For details on model based algorithms, and previous works with sub-optimal guarantees we refer to (Agarwal et al., 2020; Li et al., 2020b).
{+We note that the lower bounds apply only to the synchronous case (i.e, when every state-action pair is sampled at every step). We provide minimax lower-bounds which show that the bound is tight in the asynchronous case too (see Theorem 5 in Section K), where |S||A| in the synchronous case of (Azar et al., 2013) is replaced by 
 µ min .+}
Q-learning with Linear Function Approximation Since tabular Q-learning is intractable in most practical RL problems due to a large state space S , function approximation is deployed.
Linear function approximation is the simplest such case where the Q-function is approximated with a linear function of the ‘feature embedding’ associated with each state-action pair.
However, Q-learning can be shown to diverge even in the simplest cases as was ﬁrst noticed in (Baird, 1995), which also introduced residual gradient methods which converged rather slowly but provably.
We will only discuss recent works closest to our work and refer the reader to (Carvalho et al., 2020; Yang & Wang, 2019) for a full survey of various works in this direction.
SARSA is the on-policy control variant of Q-learning where the challenge is to explore the state-space while learning the optimal policy.
Unlike Q-learning, SARSA is inherently stable due to its on-policy nature (Gordon, 2000).
Therefore, we do not compare our results to the results of on-policy control algorithms like SARSA.
We refer to (Zou et al., 2019b; Perkins & Precup, 2002; Melo et al., 2008) for further details. Yang & Wang (2019) consider MDPs with approximate linear function representation. They require additional assumptions like ﬁnite state-action space and existence of known anchor subsets which might not hold in practice.
Our results on the other hand hold with standard assumptions, with asynchronous updates and can handle inﬁnite state-action spaces (see Theorem 1).
Similarly, Chen et al.
(2019) consider Q-learning with linear function approximation which need not be exact.
But the result requires the restrictive assumption that the ofﬂine policy is close to the optimal policy.
In contrast, we consider the less general but well-studied case of MDPs with zero inherent Bellman error and provide global convergence without restrictive assumptions on the behaviour policy.
Under the most general conditions Maei et al.
(2010) present the Greedy-GQ algorithm which converges to a point asymptotically instead of diverging.
Similar results are obtained by Carvalho et al.
(2020) for Coupled Q-learning, a 2-timescale variant of Q-learning which uses a version of OTL and ER 1 .
This algorithm experimentally resolves the popular counter-examples provided by (Tsitsiklis & Van Roy, 1996; Baird, 1995).
However, the value function guarantees in Carvalho et al.
Theorem 2) (albeit without sample complexity guarantees) requires very stringent assumptions and even in the case of tabular Q-learning might not converge to the optimal policy.
Experience Replay and Reverse Experience Replay Reinforcement learning involves learning on-the-go with highly correlated correlated data.
Iterative learning algorithms like Q-learning can sometimes get coupled to the Markov chain resulting in sub-optimal convergence.
Experience replay (ER) was introduced in order to mitigate this drawback (Lin, 1992) – here a large FIFO buffer of a ﬁxed size stores the streaming data and the learning algorithm samples a data point uniformly at random from this buffer at each step.
This makes the samples look roughly i.i.d., thus breaking the harmful correlations.
Reverse experience replay (RER) is a form of ER which stores data in a buffer but processes the data points in the reverse order as stored in the buffer.
This was introduced in entirely different contexts by (Rotinov, 2019; Jain et al., 2021b;a).
In this work, we note that reverse order traversal endows a super-martingale structure which yields the strong concentration result in Theorem 4, which is not possible with forward order traversal (see (Jain et al., 2021b, Section 3.1) for a brief demonstration of this fact).
We can also look at RER is through the lens of Dynamic programming (Bertsekas, 2011) – where the value function is evaluated backwards starting from time T to time 1 - similar to how RER bootstraps present to the future. In the context of reinforcement learning, works like Bhandari et al.
; Zou et al. (2019b) obtain ﬁnite time convergence guarantees for RL algorithms under the mixing assumptions just like this work. The strategy followed in these works is that if two samples are ˜ O ( τ mix ) time apart, then they are approximately independent and thus analysis for i.i.d. data can be applied.
The sample complexity to obtain (cid:15) error here is O ( τ mix (cid:15) 2 ) . Note that this is no better than keeping one every τ mix samples and throwing away the rest and under general mis-speciﬁed linear function representation, we might not be able to do any better (Bresler et al., 2020). In this work, we show that when the linear approximation is well specifed (ZIBEL), we can use RER to obtain a sample complexity of O ( τ mix + 1 (cid:15) 2 ) , where the mixing time serves as a cut-off.
Online Target Learning OTL (Mnih et al., 2015) maintains two different Q-values (called online
Q-value and target Q-value) where the target Q-value is held constant for some time and only the online Q-value is updated by ‘bootstrapping’ to the target.
After a number of such iterations, the target Q-value is set to the current online Q value.
OTL thus attempts to mitigate the destabilizing effects of bootstrapping by removing the ‘moving target’.
This technique has been noted to allow for an unbiased estimation of the bellman operator (Fan et al., 2020) and when trained with large batch sizes is similar to the well known neural ﬁtted Q-iteration (Riedmiller, 2005).
P ROBLEM S ETTING
Markov Decision Process We consider inﬁnite horizon, time homogenous Markov Decision Processes (MDPs) and we denote an MDP by MDP( S , A , γ, P, R ) where S is the state space,
A is the action space, γ ∈ [0 , 1) is the discount factor, P ( s (cid:48) | s, a ) is the probability of transition to state s (cid:48) from the state s on action a .
We assume that S and A are compact subsets of R n (for some n ∈ N ).
R : S ×
A → [0 , 1] is the deterministic reward associated with every state-action pair.
We will think of an MDP as an agent that is aware of its current state and it can choose the action to be taken.
Suppose the agent takes action π ( s ) , where π : S → A , at state s ∈ S , then P along with the ‘policy’ π induces a Markov chain over S , whose transition kernel is denoted by P π .
We write the γ -discounted value function of the MDP starting at state s to be:
[Equation]
It is well-known that under mild assumptions, there exists at least one optimal policy π ∗ such that the value function V ( s, π ) is maximized for every s and that there is an optimal Q-function, Q ∗ :
S ×
A → R , such that one can ﬁnd the optimal policy as π ∗ ( s ) =
arg max a ∈A Q ∗ ( s, a ) , optimal value function as V ∗ ( s ) = max a ∈A Q ∗ ( s, a ) and it satisﬁes the following ﬁxed point equation.
[Equation]
(2) can be alternately viewed as Q ∗ being the ﬁxed point of the Bellman operator T , where
[Equation]
The basic task at hand is to estimate Q ∗ ( s, a ) from a single trajectory ( s t , a t )
Tt =1 such that s t +1 ∼ P ( ·| s t , a t ) along with rewards ( r t )
Tt =1 , where r 1 , . .
, r T are random variables such that E
[ r t | s t = s, a t = a ]
= R ( s, a ) .
We refer to Section B for rigorous deﬁnitions.
Q-learning Since the transition kernel P (and hence the Bellman operator T ) is often unknown in practice, Equation (2) cannot be directly used to to estimate the optimal Q-function.
To this end, we resort to estimating Q ∗ using observations from the MDP.
An agent traverses the MDP and we obtain the state, action, and the reward obtained at each time step.
We assume the off-policy setting which means that the agent is not in our control, i.e., it is not possible to choose the agent’s actions; rather, we just observe the state, the action, and the corresponding reward.
Further, we assume that the agent follows a time homogeneous policy π ( s ) for choosing its action at state s .
Given a trajectory { s t , a t , r t } Tt =1 generated using some unknown behaviour policy π , we aim to estimate Q ∗ in a model-free manner - i.e, estimate Q ∗ without directly estimating P .
We further assume that the trajectory is given to us as a data stream so we can not arbitrarily fetch the data for any time instant.
A popular method to estimate Q ∗ is using the Q-learning algorithm.
In this online algorithm, we maintain an estimate of Q ∗ ( s, a ) at time t , Q t ( s, a ) and the estimate is updated at time t for ( s, a ) =
( s t , a t ) in the trajectory.
Formally, with step-sizes given as { η t } , Q-learning performs the following update at time t ,
[Equation]
In this work, we focus on two special classes of MDPs which are popular in literature.
Linear Markov Decision Process Linear MDPs (Jin et al., 2020) is a popular example of exact linear approximation for which statistically and computationally tractable algorithms are available.
We use the deﬁnition from Jin et al.
(2020), stated as Deﬁnition 1.
Deﬁnition 1.
An MDP( S , A , γ, P , R ) is a linear MDP with feature map φ : S ×
A → R d , if 1.
there exists a vector θ ∈ R
d
such that R ( s, a ) = (cid:104) φ ( s, a ) , θ (cid:105) , and 2.
there exists d unknown (signed) measures over S β ( · ) = { β 1 ( · ) , . .
. , β d ( · ) } such that the transition probability P ( ·| s, a ) = (cid:104) φ ( s, a ) , β ( · ) (cid:105) .
In the rest of this paper, in the tabular setting we assume that the dimension d = |S × A| and we use a one hot embedding where we map ( s, a ) → φ ( s, a ) =
e s,a , a unique standard basis vector.
It is easy to show that this system is a linear MDP (Jin et al., 2020) and Q-learning in this setting reduces to the standard tabular Q-learning (3).
However, when the assumption of a tabular MDP allows us to obtain stronger results, we will present the analysis separately.
Inherent Bellman Error There is another widely studied class of MDPs which admit a good linear representation (Zanette et al., 2020; Munos & Szepesv´ari, 2008; Szepesv´ari & Smart, 2004). Deﬁnition 2. (ZIBEL MDP) For an M = MDP ( S , A , γ, P , R ) with a feature map φ : S ×A → R d , we deﬁne the inherent Bellman error ( IBE ( M ) ) as:
[Equation] If IBE ( M ) = 0 , then call this MDP a ZIBEL (zero inherent Bellman error with linear function approximation) MDP. The class of ZIBEL MDPs is strictly more general than the class of linear MDPs (Zanette et al., 2020). Both these classes of MDPs have the property that there exists a vector w ∗ ∈ R d such that the optimal Q-function, Q ∗ ( s, a )
= (cid:104) φ ( s, a ) , w ∗ (cid:105) for every ( s, a ) ∈ S × A , which can be explicity expressed as a function of θ, β and Q ∗ .
More generally, they allow us to lift the Bellman iteration to R d exactly and update our estimates for w ∗ values directly (Lemmas 3, 4).
Hence, we can focus on estimating Q ∗ by estimating w ∗ .
To this end, the standard Q-Learning approach to learning the Q function can be extended to the linear case as follows: [Equation] (cid:21) φ
[Equation]
The above update can be seen as a gradient descent step on the loss function f ( w t ) =
( (cid:104) φ ( s t , a t ) , w t (cid:105)− target ) 2 where target =
r t + γ max a (cid:48) (cid:104) φ ( s t +1 , a (cid:48) ) , w t (cid:105) .
This update while heavily used in practice, has been known to be unstable and does not converge to w ∗ in general.
The reason often cited for this phenomenon is the presence of the ‘deadly triad’ of bootstrapping, function approximation, and off-policy learning.
2. A SSUMPTIONS
We make the following assumptions on the MDPs considered through the paper in order to present our theoretical results.
Assumption 1.
The MDP {+M has IBE ( M ) = 
+} (Deﬁnition 2), (cid:107) φ ( s, a ) (cid:107) 2 ≤ 1 . Furthermore, R ( s, a ) ∈ [0 , 1] .
Assumption 2.
Let Φ :=
{ φ ( s, a ) : ( s, a ) ∈ S × A} .
Φ is compact, span (Φ) =
R d and ( s, a ) → φ ( s, a ) is measurable.
Even when span (Φ) (cid:54) =
R d , our results hold after we discard the space orthogonal to the span of embedding vectors in Assumption 4 and note that Q-Rex does not update the iterates along span (Φ) ⊥ .
Deﬁnition 3.
For r > 0 , let N (Φ , (cid:107) · (cid:107) 2 , r ) be the r -covering number under the standard Euclidean norm over R d .
Deﬁne:
[Equation]
[-Return w K +1 , 
-]
Observe that since Φ is a subset of the unit Euclidean ball in R d , C Φ ≤ C √ d .
However, in the case of tabular MDPs it is easy to show that C Φ ≤ C √ log d .
Deﬁnition 4.
We deﬁne the norm (cid:107) · (cid:107) φ over R d by (cid:107) x (cid:107) φ = sup ( s,a ) |(cid:104) φ ( s, a ) , x (cid:105)| .
This is the natural norm of interest for the problem (Lemmas 2 and 4).
We assume the existence of a ﬁxed (random) behaviour policy π : S → ∆( A ) which selects a random action corresponding to each state.
At each step, given ( s t , a t ) =
( s, a ) , s t +1 ∼ P ( ·| s, a ) and a t +1 ∼ π
( s t +1 ) .
This gives us a Markov kernel over S ×A which speciﬁes the law of ( s t +1 , a t +1 ) conditioned on ( s t , a t ) .
We will denote this kernel by P π .
This setting is commonly known as the off-policy asynchronous setting.
We make the following assumption which is standard in this line of work.
Assumption 3.
There exists a unique stationary distribution µ for the kernel P π .
Moreover, this
Markov chain is exponentially ergodic in the total variation distance with mixing time τ mix .
That is, there exist a constant C mix for every t ∈ N
[Equation]
In the tabular setting, we will use the standard deﬁnition of τ mix instead:
[Equation]
Here TV refers to the total variation distance.
Assumption 4.
There exists κ > 0 such that: E ( s,a ) ∼ µ φ ( s, a ) φ (cid:62) ( s, a ) (cid:23) Iκ .
Remark 1.
(Bresler et al., 2020, Theorem 1) shows that even linear regression with Markovian data, zero noise and (cid:96) 2 recovery is hard when the condition number κ or the mixing time τ mix are too large.
Hence, our setup of noisy reinforcement learning with (cid:96) ∞ error also requires these quantities to be small.
Therefore, Assumptions 3 and 4 are necessary in order to obtain non-trivial bounds.
In the tabular setting, Assumption 4 manifests itself as 1 κ = µ min := min ( s,a ) µ ( s, a ) which is also standard (Li et al., 2020b).
Whenever we discuss high probability bounds (i.e, probability at least 1 − δ ), we assume that δ ∈ (0 , 1 / 2) .
Similarly, we will assume that the discount factor γ ∈ (1 / 2 , 1) so that we can absorb poly (1 /γ ) factors into constants.
1: Input: learning rates η , horizon T , discount factor γ , trajectory X t = { s t , a t , r t } , Buffer size B , Buffer gap u , Number of inner loop buffers N 2: Total buffer size: S ← B + u , Outer-loop length: K ← TNS , Initialization w 1 , 1 1 =3: for k = 1 , . . , K do 4: for j = 1 , . . . , N do 5:
Form buffer Buf = { X k,j 1 , . . .
, X k,jS } , where, X k,ji ← X NS ( k − 1)+ S ( j − 1)+ i 6:
Deﬁne for all i ∈ [1 , S ] , φ k,ji (cid:44) φ ( s k,ji , a k,ji ) .
7: for i = 1 , . . .
, B do 8: w
[Table]
[Table]
=
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
(cid:21)
[Table]
,jB
[Table]
Option I : w k,j +1= w k,jB + 10:
Option II : w k,j +1= 1 B (cid:80) Bi
=1 w k,ji + 11:
Option I : w k +1 , 1= w k,N +112:
{+Option II : w k +1 , 1= 1 N (cid:80) N +1 l =2 w k,l13: Return w K +1 , 
  O UR A LGORITHM+} As discussed in the introduction, we incorporate RER and OTL into Q-learning and introduce the algorithms Q-Rex (Online Target Q-learning with reverse experience replay, Algorithm 1), its sample efﬁcient variant Q-RexDaRe (Q-Rex + data reuse, Algorithm 2) and its episodic variant EpiQRex (Episodic Q-Rex, Algorithm 3).
Since Q-RexDaRe and EpiQ-Rex are only minor modiﬁcations of Q-Rex, we refer the reader to the appendix for their pseudocode.
Q-Rex is parametrized by K the number of iterations in the outer-loop, N the number of buffers within an outer-loop iteration, B the size of a buffer and u the gap between the buffers.
The algorithm has a three-loop structure where at the start of every outer-loop iteration (indexed by k ∈ [ K ] ), we checkpoint our current guess of the Q function given by w k, 1 1 .
Each outer-loop iteration corresponds to an inner-loop over the buffer collection with N buffers, i.e. at iteration j ∈
[ N ] , we collect a buffer of size B + u consecutive state-action-reward tuples.
For every collected buffer we consider the ﬁrst B collected experiences and perform the target based Q-learning update in the reverse order for these experiences.
We refer Figure 1 for an illustration of the processing order.
Of note, is the usage of checkpointed target network in the RHS of the Q-learning update through the entirety of the outer-loop iteration, i.e. for a ﬁxed k and for all j, i , our algorithm sets
[Equation]
Figure 1 provides an illustration of the processing order for our updates.
It can be seen that the number of experiences collected through the run of the algorithm is T = KN ( B + u ) .
For the sake of simplicity, we will assume that the initial point, w 1 , 1= 0 .
Essentially the same results hold for arbitrary initial conditions.
Q-RexDaRe is a modiﬁcation of Q-Rex where we re-use the data from the ﬁrst outer-loop iteration (i.e, data from k = 1 ) in every outer-loop iteration (i.e, k > 1 ). [Table]
Here the poly-log factors β i are given by [Table]
Remark 2.
For the sake of clarity, we only analyze the algorithms Q-Rex and Q-RexDaRe for data from a single trajectory with Option I .
Option II involves averaging of the iterates which boosts the convergence – indeed we can obtain much better bounds in this setting by using standard analysis.
M AIN R ESULTS
We will now provide ﬁnite time convergence analysis and sample complexity for the algorithms
Q-Rex and Q-RexDaRe.
Recall that K is the number of outer-loops, N is the number of buffers inside an outer-loop iteration, B is the buffer size and u is the size of the gap.
In what follows, we will take u = ˜
O
( τ mix ) , B = 10 u , K = ˜
O
( 11 − γ ) .
We also note that the total number of samples used is NK ( B + u ) for Q-Rex and N ( B + u ) for Q-RexDaRe since we reuse data in each outer-loop iteration.
In what follows, by Q K +1 , 1( s, a ) , we denote (cid:104) φ ( s, a ) , w K +1 , 1(cid:105) which is our estimate for the optimal Q function.
Here w K +1 , 1is the output of either Q-Rex or Q-RexDaRe at the end of K outer-loop iterations.
Deﬁne (cid:107) Q K +1 , 1− Q (cid:107) ∞ := sup ( s,a ) ∈S×A (cid:12)(cid:12) Q K +1 , 1( s, a ) − Q ∗ ( s, a ) (cid:12)(cid:12) .
We ﬁrst consider the performance of Q-Rex with data derived from a linear MDP (Deﬁntion 1) or a ZIBEL MDP (Deﬁnition 2) and satisfying the Assumptions in Section 2.1.
Theorem 1 (ZIBEL /Linear MDP) .
Suppose we run Q-Rex using Option I with data from an MDP with IBE = 0 .
There exists constants C 1 , C 2 , C 3 , C 4 , C 5 > 0 such that whenever the parameter bounds given in Table 2 (row 1) are satisﬁed, then with probability at-least 1 − δ , we must have:
[Equation]
Given (cid:15) ∈ (0 , 1 (1 − γ ) ] , and the parameters as given in Table 2 (row 4)(up to constant factors), then with probability at-least 1 − δ : (cid:107) Q K +1 , 1( s, a ) − Q ∗ ( s, a ) (cid:107) ∞ < (cid:15) .
This has a sample complexity
[Equation]
We now consider the performance of Q-Rex and Q-RexDaRe in the case of tabular MDPs.
We refer to Table 1 for a comparison of our results to the state-of-art results provided in literature for
Q-learning based algorithms.
Remark 3. To the best of our knowledge, Theorem 1 presents the ﬁrst non-asymptotic convergence results for Q-learning based methods for ZIBEL MDPs under standard assumptions.
Notice that the sample complexity scales as 1 (cid:15) 2 + τ mix instead of τ mix (cid:15) 2 like in Zou et al. (2019b)
; Bhandari et al. This is because in the case of ZIBEL MDPs RER brings out the super-martingale structure present in the problem which forward pass does not.
Theorem 2 (Tabular MDP) .
Suppose we run Q-Rex using Option I with data derived from tabular
MDPs.
Whenever the algorithmic parameters are picked as given in Table 2 (row 2) for some universal constants C 1 , . .
, C 5 , we obtain with probability at-least 1 − δ :
[Equation]
Where L =
[Equation]
11 − γ .
Given (cid:15) ∈ (0 , 11 − γ ] , and the parameters are picked as given in Table 2 (row 5), then with probability at-least 1 − δ , we have: (cid:107) Q K +1 − Q ∗ (cid:107) ∞ < (cid:15) .
This gives us a sample complexity of
[Equation]
Even though the sample complexity provided in Theorem 2 matches the sample complexity of synchronous Q-learning even with asynchronous data, it is still sub-optimal with respect to the min-max lower bounds (i.e, it has a dependence of has a dependence of [Equation] instead of the optimal
[Equation] We resolve this gap for Q-RexDaRe in Theorem 3.
For tabular MDPs, the number states can be large but the support of P ( ·| s, a ) is bounded in most problems of practical interest.
Consider the following assumption (note that this holds for every tabular MDP with ¯ d = |S| .)
Assumption 5.
Tabular MDP is such that | supp ( P ( ·| s, a ))
| ≤ ¯ d ∈ N .
Theorem 3 (Tabular MDP with Data Reuse) .
For tabular MDPs, suppose additionally Assumptionholds
and we run Q-RexDaRe using Option I .
There exist universal constants C 1 , C 2 , C 3 , C 4 such that when the parameter values satisfy the bounds in Table 2 (row 3), with probability at-least 1 − δ : (cid:107) Q K +1 , 1−
[Equation]
∞ ≤ C
[Equation]
Suppose (cid:15) ∈ (0 , 11 − γ ] .
If we choose the parameters as per Table 2 (row 6), then with probability at-least 1 − δ we have: (cid:107) Q K +
( s, a ) − Q ∗ (cid:107) ∞ < (cid:15) .
The sample complexity in this case is
[Equation]
O VERVIEW OF THE A NALYSIS
We divide the analysis of Q-Rex and Q-RexDaRe into two parts: Analysis of w k, 1obtained at the end of outer-loop iteration k and the analysis of the algorithm within the outer-loop.
The algorithm reduces to SGD for linear regression with Markovian data within an outer-loop due to OTL.
That is, we try to ﬁnd w k +1 , 1such that (cid:104) w k +1 , [-
-]1, [Equation] ( s,
[Equation]
s
[Equation] [Equation] (cid:48) ) (cid:105) . Therefore, we write w k +1 , 1= T ( w k, 1 1 ) + (cid:15) k
( w k, 1 1 ) , where T is the γ contractive Bellman operator whose unique ﬁxed point is w ∗ and (cid:15) k is the noise to be controlled.
Following a similar setting in in (Jain et al., 2021b), we control (cid:15) k with the following steps:
(1) We introduce a ﬁctitious coupled process (see Section C)
(˜ s t , ˜ a t , ˜ r t ) where the data in different buffers are exactly independent (since the gaps of size u make the buffers approximately independent) and show that the algorithm run with the ﬁctitious data has the same output as the algorithm run with the actual data with high probability when u is large enough.
(2) We give a bias-variance decomposition (Lemma 5) for the error (cid:15) k where the exponentially decaying bias term helps forget the initial condition and the variance term arises due the inherent noise in the samples.
(3) We control the bias and variance terms separately in order to ensure that the noise (cid:15) k is small enough.
RER plays a key role in controlling the variance term by endowing it with a supermartingale structure, which is not possible with forward order traversal (see Theorem 4)
The procedure described above allows us to show that w k +1 , 1≈ T ( w k, 1 1 ) uniformly for k ≤ K , which directly gives us a convergence bound to the ﬁxed point of T i.e, w ∗ (Theorem 1).
In the tabular case, the approximate Bellman iteration connects to the analysis of synchronous Q-learning in (Li et al., 2021), which allows us to obtain a better convergence guarantee (Theorem 2).
To obtain convergence guarantees for Q-RexDaRe, we ﬁrst observe that if we re-use the data used in outerloop iteration 1 in all future outer-loop iterations k > 1 , (cid:15) k ( w k, 1 1 ) might not be small since w k, 1depends on (cid:15)
k ( · ) .
However, ( w k, 1 1 ) k approximates the deterministic path of the noiseless Bellman iterates: ¯ w 1 , 1:= w 1 , 1 1 and ¯ w k +1 , 1:= T ( ¯ w k, 1 1 ) .
Since (cid:107) (cid:15) k
( w k, 1 1 ) (cid:107) ∞ ≤ (cid:107) (cid:15) k ( w k, 1 1 ) − (cid:15) k
( ¯ w k, 1 1 ) (cid:107) ∞ + (cid:107) (cid:15)
k ( ¯ w k, 1 1 ) (cid:107) ∞ , we argue inductively that (cid:107) (cid:15) k
( w k, 1 1 ) − (cid:15) k
( ¯ w k, 1 1 ) (cid:107) ∞
≈ 0 since w k, 1≈ ¯ w k, 1and (cid:107) (cid:15)
k ( ¯ w k, 1 1 ) (cid:107) ∞
≈ 0 since ¯ w k, 1is a deterministic sequence and hence w k +1 , 1≈ ¯ w k +1 , 1.
A CKNOWLEDGMENTS
Most of this work was done when D.N. was a graduate student at MIT and was supported in part by NSF grant DMS-2022448.
Part of this work was done when D.N. was a visitor at the Simons
Institute for Theory of Computing, Berkeley. We would also like to thank Gaurav Mahajan for introducing us to low-inherent Bellman error setting, and providing intuition that our technique might be applicable in this more general setting (than linear MDP) as well.
A E XPERIMENTS
Even though OTL has a stabilizing effect on the Q-iteration, it reduces the rate of bias decay since the values are not updated for a long time.
Therefore, the success of our procedure depends on picking the right values for the parameters N, B and Option I vs. Option II .
However, under the right conditions the algorithms which include OTL+RER converge to a much smaller ﬁnal error as illustrated by the examples we provide in this section.
If a better sample complexity is desired, then Q-RexDaRe can be used as shown by Theorem 3.
Further research is needed to identify the practical conditions such as (function approximation coarseness, MDP reward structure etc.)
under which techniques like OTL+RER help.
100-State Random Walk
We ﬁrst consider the episodic MDP (Sutton & Barto, 2018, example 6.2) – but with 100 states instead of 1000 .
Here the agent can either move left or right on a straight line, receiving a reward of 0 at each step.
Reaching the right terminal point ends the episode with reward 1 , while the left terminal point ends the episode with reward 0 .
In each episode, the initial point is uniformly random and the ofﬂine policy chooses right and left directions uniformly at random.
We use state aggregation to obtain a linear function representation (Sutton & Barto, 2018) with total 10 aggregate states.
Along with 2 actions, this leads to a 20 dimensional embedding.
We compare vanilla Q-learning, OTL+ER+Q-learning and EpiQ-Rex ( Option II , N = 1 ).
The same step size ( 0 . 01 ) was chosen for all the algorithms.
OTL+ER+Q has the same structure as
EpiQ-Rex and was run with the same parameters as EpiQ-Rex .
The main difference between the two algorithms is that OTL+ER+Q processes the experiences collected in each episode in a random order processing instead of reverse order.
Refering to Figure 4, we note that the bias decay for EpiQRex and OTL+ER+Q are slower than vanilla Q-learning due to the online target structure.
However
EpiQ-Rex converges to a better solution than the other algorithms.
Mountain Car We run an online control type experiment with the Mountain car problem (Sutton & Barto, 2018, Example 10.1).
The task here is to control the car and help it reach the correct peak (which ends the episode) as soon as possible (i.e, terminate the episode with fewest steps).
The agent receives a reward of − 1 unless the correct peak is reached.
Here we run EpiQ-Rex with Option I and N = 1 , OTL+ER+Q-learning and vanilla Q-learning.
The k -th episode is generated with the policy at the end of k − 1 -th episode for each of the three algorithms.
We use a n = 4 tile coding to represent the Q values for a given action, each of which has 4 × 4 squares.
The step-size was picked as 0 .
1 /n
and the result was averaged over 500 runs of the experiment.
We refer to Figurefor the outcomes.
For the last 300 episodes, the mean episode length for Vanilla Q-learning was 145 , EpiQ-Rex was 136 and OTL+ER+Q-learning was 143 (rounded to the nearest integer).
Grid-World We consider the grid-world problem which is a tabular MDP (Sutton & Barto, 2018,
Example 3.5), which is a continuing task.
Here, an agent can walk north, south, east or west in a 5 × 5 grid.
Trying to fall off the grid accrues a reward of − 1 , while reaching certain special states accrues a reward of 10 or 5 .
In our example, we also add a unif [ − 0 .
5 , 0 . 5] noise to all rewards to make the problem harder.
We run the grid world experiment with the discount factor γ = 0 .
9 and step size 0 .
for Vanilla Q-learning, Q-Rex ( Option II , B = 3000 , N = 1 ) and OTL+ER+Qlearning which is run with the same parameters as Q-Rex but with random samples from the buffer.
We run the experiment 30 times and plot the error in the Q-values vs. time in Figure 3.
Baird’s Counter-Example Consider the famous Baird’s counter-example shown in Figure 6.
The features corresponding to each state is shown and the reward for any transition is 0 .
Thus, w ∗ =is the optimal solution.
Since Assumption 4 made in this work is violated for this example, we consider the analog of the problem where at each step, a state is sampled uniformly at random and the corresponding transition, along with the 0 reward is used to learn the vector w .
In the experiment, we set the problem and algorithmic parameters to be γ = 0 .
99 and η = 0 .
01 / √ 5 (the factor of √normalizes the features to satisfy Assumption 1).
We set B = 50 , u = 0 , and N = 5 since there is no need for keeping a gap between the buffers in this experiment.
Figure 7 shows the non-convergent behavior of vanilla Q-learning while OTL-based Q-learning converges.
Note that since the sampling of state, action and reward is done in a uniformly random fashion, it is not relevant to use reverse experience replay.
It is easy to see that with data reuse, we only need few samples to ensure all states are covered; the rate of convergence would be same as that of OTL+Q-learning.
Linear Dynamical System with Linear Rewards We also compare the algorithms on a linear dynamical systems problem.
While the problem described next is strictly not a linear MDP, the value functions for certain policies can be written as a linearly in terms of the initial condition; this is made precise next.
Consider a linear dynamical system with initial state being X 0 ∈ R d .
The state evolves as X t +1 =
AX t + η t , where A ∈ R d × d .
The reward obtained for such transition is given by (cid:104) X t , θ (cid:105) for a ﬁxed θ ∈ R d .
The maximum singular value of A is chosen less thanto ensure that the system is stable.
The inﬁnite horizon γ -discounted expected reward is given by (cid:104) X 0 , (cid:80) ∞ i =
( γA T ) i θ (cid:105) .
Thus, the expected reward can be written as (cid:104) X 0 , w ∗ (cid:105) , where
[Equation]
Since there are no actions in this case, the Q learning algorithms reduce to value function approximation (i.e, TD(0) type algorithms).
We take the embedding φ ( X t ) =
X t , the identity mapping.
We considered d = 5 , γ = 0 .
99 , η = 0 .
01 , and a randomly generated normal matrix A and θ .
We
Option II for Q-Rex along with B = 75 , u = 25 , and N = 5 for the experiments.
For OTL+ER+Qlearning, we keep the same parameters, but with random order sampling, while ER+Q-learning does not include OTL.
The results shown in Figure 5 are averaged over 100 independent runs of the experiment.
Note that the errors considered are using iterate averaging.
As seen from Figure 5, Q-Rex outperforms vanilla Q-learning and OTL+ER-based Q-learning as one would expect based on the theory presented in this work.
However, it is interesting to note that
ER+Q-learning performs slightly better than Q-Rex.
One possible reason could be due to the fact that in Q-Rex, the target gets updated at a slower rate at the cost of reducing bias.
However, setting a smaller value of NB might resolve the issue.
B D EFINITIONS AND N OTATIONS B.
[Equation] EX
[Equation] R E The psuedocode for Q-RexDaRe is given in Algorithm 2.
Note that we reuse the sample data in every outer-loop iteration instead of drawing fresh samples.
[Table]
(cid:105) φ k,jB +1 − i 11: w k,j +1= w k,jB + 12: w k +1 , 1= w k,N +113:
Return w T +
B. E PI Q-R EX
The psuedocode for EpiQ-Rex is given in Algorithm 3.
Note that the buffers here are individual episodes and can vary in size due to inherent randomness.
We do not require a gap here since separate episodes are assumed to be independent.
MDP deﬁnition
Here we construct the MDP with a (possibly) random reward r .
We consider non-episodic, i.e. inﬁnite horizon, time homogenous Markov Decision Processes (MDPs) and we denote an MDP by MDP( S , A , γ, P, r ) where S denotes the state space,
A denotes the action space, γ represents the discount factor, P ( s (cid:48) | s, a ) represents the probability of transition to state s (cid:48) from the state s on action a .
We assume for purely technical reasons that S and A are compact subsets of R n for some n ∈ N ).
r is a reward process (not to be confused with Markov Reward Processes i.e, MRP) indexed by S ×
A × S , such that r ( s, a, s (cid:48) )
∈ [0 , 1] almost surely.
We will skip the measure theoretic details of the deﬁnitions and assume that the sequence of i.i.d.
reward processes ( r t ) t ∈ N can be jointly deﬁned over a Polish probability space.
The function R : S ×
A → [0 , 1] represents the deterministic reward R ( s, a ) obtained on taking action a at state s and is deﬁned by
R ( s, a ) = E (cid:2)
E s (cid:48) ∼ P ( ·| s,a ) r ( s, a, s (cid:48) )
(cid:3) .
Here the expectation is with respect to both the randomness in the reward process and in state s (cid:48) .
Now, given a trajectory, ( s t , a t ) Tt =1 , which is independent of i.i.d sequence of rewards processes ( r t )
Tt =1 .
We observe ( s t , a t , r t ( s t , a t , s t +1 ))
Tt =1 , and we will henceforth denote r t ( s t , a t , s t +1 ) by just r t .
1: Input: learning rates η , number of episodes T , discount factor γ , Epsiodes E t
= { ( s t 1 , a t 1 , r t 1 ) , . . .
, ( s yB t , a tB t , r tB t ) } , Number of buffer per outer-loop iteration N.
2: Number of outer loop iterations: K ← T/N 3: w 1 , 1 1 = 0 initialization 4: for k = 1 , . .
, K do 5: for j = 1 , . . .
, N do 6: t ← ( k − 1) ∗ N + j 7:
Collect experience and form buffer Buf = { X t 1 , . . .
, X tB t } , where,
X ti =
( s ti , a ti , r ti ) 8:
Deﬁne for all i ∈ [1 , B t − 1] , φ k,ji
[Table]
9: for i = 1 , . . .
, B do 10: w
[Table]
jB
[Table]
Option I: w k,j +1= w k,jB t 12:
Option II: w k,j +1= 1 B t − 1 (cid:80) B t i =2 w k,j i 13: w k +1 , 1= w k,N +114: Return w K,
Notation Due to three loop nature of our algorithm it will be convenient to deﬁne some simplifying notation.
To this end, consider the outer loop with index k ∈
[ K ] and buffer number j ∈ [ N ] inside this outer loop.
Further given an i ∈ [ S ] deﬁne a time index as t k,ji
[Equation]
We now denote the i -th state-action-reward tuple inside this buffer by
[Equation]
Similarly, R k,ji = R ( s k,ji , a k,ji ) .
For conciseness, we deﬁne for all i, j, k , φ k,ji := φ ( s k,ji , a k,ji ) .
Since we are processing the data in the reverse order within the buffer, the following notation will be useful for analysis:
[Equation]
C C OUPLED P ROCESS
It can be seen that the buffers are approximately i.i.d.
whenever we take u = O ( τ mix log Tδ ) whenever Assumption 3 is satisﬁed.
For the sake of clarity of analysis, we will consider exactly independent buffers.
That is, we assume the algorithms are run with a ﬁctitious trajectory (˜ s t , ˜ a t , ˜ r t ) , where we assume that the ﬁctitious trajectory is generated such that the ﬁrst state of every buffer is sampled from the stationary distribution µ .
We show that we can couple this ﬁctitious process (i.e, deﬁne it on a common probability space as the original process ( s t , a t , r t ) )
such that
[Equation]
Notice that the equality does not hold within the gaps between the buffer which are of size u but inside the buffers of size B only.
That is, the sequence of iterates obtained by running the algorithm with the original data ( s t , a t , r t ) is the same as the sequence of iterates obtained by running the algorithm with the ﬁctitious coupled data (˜ s t , ˜ a t , ˜ r t ) with high probability.
We state this result formally in Lemma 16 and prove it in Section M.1.
Henceforth, we will assume that we run the algorithm with data (˜ s t , ˜ a t , ˜ r t ) and refer to Lemma 16 to carry over the results to the original data set with high probability.
We analogously deﬁne ˜
φ k,ji .
We will denote the iterates of the algorithm run with the coupled trajectory as ˜ w k,ji instead of w k,ji and will focus on it entirely.
We now provide some deﬁnitions based on the above process.
These deﬁnitions will be used repeatedly in our analysis.
D B ASIC S TRUCTURAL L EMMAS
We ﬁrst note some basic structural lemmas regarding ZIBEL MDPs under the assumptions in Section 2.1.
We refer to Section M for the proofs.
Lemma 1.
For tabular MDPs satisfying the assumptions in Section 2.1, for both Q-Rex and QRexDaRe, we have that for every k, j, i we have that (cid:107) ˜ w k,ji (cid:107) φ ≤ 11 − γ
The lemma above says, in particular, that the Q-value estimate given by our algorithm never exceeds 11 − γ due to 0 initialization.
The proof is a straightforward induction argument, which we omit.
We will henceforth use Lemma 1 without explicitly mentioning it.
Lemma 2.
Suppose Q w ( s, a ) :=
(cid:104) w, φ ( s, a ) (cid:105) and let Q ∗ ( s, a )
= (cid:104)
w ∗ , φ ( s, a ) (cid:105) be the optimal Q function.
Then:
[Equation]
Moreover, we must have: (cid:107) x (cid:107) ≥ (cid:107) x (cid:107) φ ≥ (cid:107) x
(cid:107)√ κ for any x ∈ R d .
Lemma 3.
For any w 0 ∈ R d , there exists a unique w 1 ∈ R d such that
[Equation]
We will denote this mapping w 0 → w 1 by w 1 = T ( w 0 ) .
Lemma 4.
T : R d → R d is γ contractive in the norm (cid:107) · (cid:107) φ .
The unique ﬁxed point of T is w ∗ .
Moreover, we have: (cid:107) w ∗ (cid:107) φ ≤ 11 − γ and (cid:107) w ∗ (cid:107) ≤ √ κ 1 − γ .
In view of Lemma 4, we can begin to look at the following noiseless Q -iteration.
Let ¯ w 1 = w 1 , 1 1 =and ¯ w k +
= T ( ¯ w k ) .
This converges geometrically to w ∗ with contraction coefﬁcient γ under the norm (cid:107) · (cid:107) φ .
In our case, however, we only have sample access to the operator T .
Therefore, our
Q-iteration at the end of k -th outer loop can be written as ˜ w k +1 , 1= T ( ˜ w k, 1 1 ) + ˜ (cid:15) k where ˜ (cid:15) k is the error introduced via sampling which needs to be controlled.
E B IAS V ARIANCE D ECOMPOSITION
We begin our analysis by providing a bias-variance decomposition of the error with respect to the noiseless Q -iteration at every loop.
We will need the following deﬁnitions which we use repeatedly through our analysis.
Given step size η , outer loop index k and buffer index j , we deﬁne the following contraction matrices for a, b ∈ [ B ] .
[Equation]
Whenever a > b , we will deﬁne ˜
H k,ja,b := I .
In the tabular setting, we deﬁne for any ( s, a ) ∈ S × A by ˜ N k ( s, a ) to be the number of samples of ( s, a ) seen in the outer loop k (excluding the gaps),i.e.
˜ N k ( s, a ) =
N (cid:88) j =
B (cid:88) i = 1 ((˜ s k,ji , ˜ a k
[Equation]
Further we denote by ˜ N k,j i ( s, a ) , the number of samples of ( s, a ) seen in the outer loop k in the buffers post the buffer j as well as the number of samples of ( s, a ) in buffer j before iteration i .
Formally,
[Equation]
We deﬁne the error term for any w :
[Equation]
Finally we deﬁne the following shorthands for any i, j, k :
[Equation]
Given the above deﬁnition, the following the lemma provides the Bias-Variance decomposition which is the core of our analysis.
Lemma 5 (Bias-Variance Decomposition) .
For every k , we have that,
[Equation]
Here and later on in the paper, we use the reverse order in the product to highlight the convention that higher indices l in ˜ H k,l 1 ,B appear towards the left side of the product and further deﬁne (cid:81)
N +1 l = N ˜
H k,l 1 ,B =
I .
We call the ﬁrst term in Equation (7) as the bias term and it decays geometrically with N and the second term is called the variance, which has zero mean.
We will bound these terms separately.
Since tabular setting allows for improved analysis of error, we will provide special cases for the tabular setting with reﬁned bounds.
We refer to Section M.5 for the proof of Lemma 5.
F B OUNDING THE B IAS T ERM F. T ABULAR C ASE
In the tabular case, we have the following expression for bias.
We omit the proof since it follows from a simple calculation.
Lemma 6.
In the tabular setting, we have:
[Equation]
For a particular outer loop k , we show that ˜ N k ( s, a ) is Ω( µ min NB ) with high probability whenever
NB is large enough.
For this we use (Paulin, 2015, Theorem 3.4) similar to the proof of (Li et al., 2020b, Lemma 8).
We give the proof in Section M.6.
Lemma 7.
There exists a constant C such that whenever B ≥ τ mix and NB ≥ C τ mix µ min log( |S||A| δ ) , with probability at least 1 − δ , we have that for every ( s, a ) ∈ S × A : ˜ N k
[Equation] s, a ) ≥ 12 µ [Equation] s, a ) NB ≥ 12 µ min NB
F. ZIBEL MDP C ASE
Lemma 8.
Suppose g ∈ R d is ﬁxed and ηB < 14 .
Then, the following hold:
[Equation]
With probability at-least 1 − δ , we have:
[Equation]
We refer to Section L
[Equation]
1 for the proof
[Equation]
G B OUNDING THE V ARIANCE T ERM G. T ABULAR C ASE
In the tabular case, it is clear that:
[Equation]
Now
[Equation]
we note that ˜ N k,ji ( s, a ) depends on data in buffers l > j and when i 1 <
i inside buffer j .
Following the discussion in (Li et al., 2021)
[Equation]
we deﬁne the vector Var P ( V k ) , Var P ( V ∗ ) ∈ R S×A such that
[Equation]
More generally, we deﬁne
[Equation]
Similarly, we deﬁne ˜
L k,j ( w ) by replacing w k, 1with any ﬁxed, arbitrary w .
It is easy to see that:
[Equation] [Equation]
Lemma 9.
In the tabular setting, suppose w is a ﬁxed vector such that (cid:104) w, φ ( s, a ) (cid:105) ∈ [Equation] − γ ] for every ( s, a ) ∈ S × A .
Fix ( s, a ) ∈ S × A .
Then there exists a universal constant C such that with probability atleast 1 − δ , we have that:
Then,
[Equation]
G. ZIBEL MDP C ASE
We will use an appropriate exponential super-martingale to bound the error term in the ZIBEL MDP case just like in the proof of (Jain et al., 2021b, Lemma 27).
The following thereom summarizes the result and we refer to Section L.3 for its proof.
Theorem 4.
Suppose x, w ∈ R d are ﬁxed.
Then, there exists a universal constant C such that with probability at least 1 − δ , we have:
[Equation]
By a direct application of (Vershynin,
[Equation]
Theorem 8.1.6)
[Equation]
we derive the following corollary.
We remind the reader that C Φ is the covering number deﬁned in Deﬁnition
[Equation]
Corollary 1.
Suppose x, w ∈ R d are ﬁxed.
Then, there exists a universal constant C such that with probability at least 1 − δ , we have:
[Equation]
log
[Equation]
[Equation]
In order to apply the theorem above, we will need to control (cid:107) w k, 1 1 (cid:107) φ uniformly for all k .
The following lemma presents such a bound
and we refer to Section L.4 for the proof.
Lemma 10.
Suppose ˜ w k, 1are the iterates of Q-Rex with coupled data from a ZIBEL MDP.
There exist universal constants C, C 1 , C 2 such that whenever NB > C 1 κη log (cid:16) Kκ δ (1 − γ ) (cid:17) , η <
C(1 − γ ) 2 C 2 φ +log( K/δ ) and ηB < 14 , with probability at-least 1 − δ , the following hold:
For every 1 ≤ k ≤ K , (cid:107) ˜ w k, 1 1 (cid:107) φ ≤ 41 − γ 2.
For every 1 ≤ k ≤ K , (cid:107) ˜ (cid:15) k (cid:107) φ ≤ (cid:113)
Kκ δ (1 − γ ) 2 exp( − ηNBκ )
+ C √ η (1 − γ ) (cid:104) C
Φ + (cid:113) log( 2 Kδ )
(cid:105)
H P ROOF OF T HEOREM
Proof.
Consider the Q learning iteration: [Equation]
Using Lemma 4, we conclude: ˜ w k +1 , {+
+}
[Equation]
=
[Equation]
[Equation]
) + (cid:15) k and thence:
[Equation]
Unrolling the recursion above, we conclude:
[Equation]
Now, we invoke item 2 of Lemma 10 along with the constraints on N, B, K, η and Lemma 2 to conclude the result.
I P ROOF OF T HEOREM
In this section we analyze the output of Q-Rex in the tabular setting and obtain convergence guarantees.
To connect with the standard theory for tabular MDP Q-learning in (Li et al., 2021), let us use the standard Q -function notation where we assume for all k , ˜ Q k, 1∈ R S×A and we have that ˜ Q k, 1 1 ( s, a ) = (cid:104)
˜
w k, 1 1 , φ ( s, a ) (cid:105) .
Since φ ( s, a ) are the standard basis vectors, we must have ˜ Q k, 1= ˜ w k, 1 1 .
In the tabular setting, we see by using Lemmas 5, 6, 7, and 9 that for any δ > 0 , there exists a universal constant C , whenever NB [Equation] ) , with probability at-least 1 − δ , for every k ∈ [ K ] and every ( s, a ) ∈ S × A :
[Equation]
(14) where ˜ (cid:15) k ∈ R
S×A is such that for all ( s, a ) ,
[Equation]
Now that we have set-up the notation, we will roughly follow the analysis methods used in (Li et al., 2021).
Since we start our algorithm with ˜ Q 1 , 1 1 = 0 , and r t ∈ [0 , 1] almost surely, we can easily show that ˜ Q k, 1 1 ( s, a ) ∈ [0 , 1 1 − γ ] for every k, s, a .
Therefore, we upper bound (cid:13)(cid:13)(cid:13)(cid:13)
˜ Q k, 1 1 −T (cid:104)
˜
Q k, 1 1 (cid:105)(cid:13)(cid:13)(cid:13)(cid:13) ∞ ≤
[Equation]
− γ in Equation (15) to conclude that with probability at-least 1 − δ , for every k ∈ [ K ] and every ( s, a ) ∈ S × A :
[Equation]
Where α η :=
C
[Equation]
log (cid:0) K |S||A| δ
[Equation]
Now we deﬁne ∆ k = ˜ Q k, 1− Q ∗ and π k : S
→ A to be the deterministic policy given by Q k, 1i.e, π k ( s ) :
= arg sup a ∈A ˜ Q k, 1 1 ( s, a ) and π ∗ to be optimal policy given by π ∗ ( s ) :
= arg sup a ∈A Q ∗ ( s, a ) .
We use the convention that we pick a single maximizing action using some rule whenever there are multiple.
Similarly, we let P π k , to be the Markov transition kernel over
S × A given by P π k (( s, a ) , ( s (cid:48) , a (cid:48) ))
=
P ( s (cid:48) | s, a ) 1 ( a (cid:48)
=
π k ( s (cid:48) )) .
Similarly, we deﬁne P π ∗ with respect to the policy π ∗ .
It is easy to show that:
[Equation]
Similarly,
[Equation]
Furthermore given any Q ∈ R S×A , letting π Q being the greedy policy with respect to the function Q we have that for any policy π , it can be easily seen from the deﬁnitions that following element-wise inequality follows:
[Equation]
We now use Equation (14) along with the equations above to conclude:
[Equation]
Here, the inequality is assumed to be point-wise.
By properties of Markov transition kernels, we can write:
[Equation]
Here, we use the convention that (cid:81)
K
+1 l = K P π l = I .
We bound the lower bound and the upper bound given in Equation (18) separately in order to bound (cid:107) ∆ K +1 (cid:107) ∞ .
We ﬁrst consider the lower bound.
Using (Azar et al., 2013, Lemma 7), we have:
[Equation]
We also note from (Li et al., 2021, Equation 64) and basic calculations that:
[Equation]
[Equation] Using Equations (15) (16), we conclude that there exist universal constants C, C 1 such that with probability at least 1 − δ
(interpreting the inequalities as element-wise):
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
In the above chain
[Equation]
the ﬁrst inequality follows from Equations (16) (18), the second inequality from
Equation (20) and the fourth inequality from Equation (19).
For the upper bound consider the following set of equations interpreting them element-wise which hold for a universal constant C and with probability at least 1 − δ .
[Equation]
(cid:116) (cid:32) k
+1 (cid:89) l
= K
P π l (cid:33)
Var P ( V k ) ≤ α η + γ K 1 − γ + C (cid:113)
ηγ 2 log (cid:0) K |S||A| δ (cid:1) (cid:118)(cid:117)(cid:117)(cid:116) K (cid:88) k
= γ K − k (cid:118)(cid:117)(cid:117)(cid:116) K (cid:88) k
= γ K − k (cid:32)
k +1 (cid:89) l
= K
P π l (cid:33)
Var P ( V k ) ≤ α η + γ K 1 − γ
+ C (cid:115) ηγ1 − γ log (cid:0) K |S||A| δ (cid:1) (cid:118)(cid:117)(cid:117)(cid:116)
K (cid:88) k
= γ K − k (cid:32)
k +1 (cid:89) l
= K
P π l (cid:33)
Var P ( V k ) ≤ α η + γ K 1 − γ
+ C (cid:115) ηγ1 − γ log (cid:0) K |S||A| δ (cid:1) (cid:118)(cid:117)(cid:117)(cid:116)
K (cid:88) k
= K/ 2+ γ K − k
(cid:32)
k +1 (cid:89) l
= K
P π l (cid:33)
Var P ( V k ) + γ K/(1 − γ ) 3 .
In the above chain, the ﬁrst inequality follows from Equation (18), the second inequality follows from Equation (16), the third inequality follows from Jensen’s inequality and noting that P π is a
Markov operator, the fourth inequality via Cauchy-Schwartz and the last inequality by noting that
It can now be veriﬁed that (Li et al., 2021, Lemma 5) applies in our setting to conclude that:
[Equation]
Using the equation above, along with Equations (22) and (21), we conclude there exists a universal constant C such that with probability at least 1 − δ , we have:
[Equation]
We note that this works with every K replaced with l for any l ≤ K , importantly under the same event (with probability at least 1 − δ ) described above for which Equation (23) holds.
Deﬁne L = K 2 s for some s =
(cid:100) log 2 (1 + C log( 11 − γ ))
(cid:101) .
Under the conditions of the Theorem, i.e,
K > C
[Equation]
2 , we have γ L/ 2 1 − γ < 1 .
Therefore we conclude from the discussion above that for every K ≥ l ≥ L , we must have:
[Equation]
To analyze this recursion, we have the following lemma which establishes hyper-contractivity, whose proof we defer to Section M.8.
Lemma 11.
Suppose α, β ≥ 0 .
Consider the function f : R + → R + given by f ( u )
=
α + β √ u .
Then, f has the unique ﬁxed point: u ∗ :=
(cid:18) β + √ β 2 +4 α (cid:19).
For t ∈ N , denoting f ( t ) to be the t fold composition of f with itself, we have for any u ∈ R + :
[Equation]
Now consider for 0 ≤ a < s ,
[Equation]
In lemma 11, deﬁne f with α
= α η + γ L 1 − γ + C
[Equation]
log
[Equation]
K |S||A| δ (cid:1) (1 − γ [Equation] and β = C (cid:114) [Equation]
log
[Equation]
K |S||A| δ [Equation] (1 − γ ). The ﬁxed point u ∗ is such that:
[Equation]
Clearly, by Equation (24), we have: u a ≤ f ( u a − 1 ) and (cid:107) ∆ K +1 (cid:107) ≤ f ( u s − 1 ) .
By monotonicity of f ( · ) and the fact that u 0 ≤ 1 1 − γ (Lemma 1):
[Equation]
Therefore,
[Equation]
Applying Lemma 11, we conclude:
[Equation] [Equation]
Note that under the constraints on the parameter η and K as stated in the Theorem, we must have u ∗ ≤ 1 (1 − γ ) 3 .
By our choice of s , we must have: | 1 (1 − γ ) − u ∗ |
12 s ≤ C (cid:48) .
Using this in Equationand the fact that β ( 2 − 1 2 s − 1 ) ≤ β + β 2 , we conclude that with probability at-least 1 − δ , we must have:
[Equation]
This proves the ﬁrst part of the theorem.
For the second part, we directly substitute the values provided to verify that we indeed obtain (cid:15) error.
J P ROOF OF T HEOREM
We will now show uniform convergence type result under Assumption 5.
For ( s, a ) ∈ S × A and s (cid:48) ∈ supp ( P ( ·| s, a )) .
We deﬁne the random variables for all k :
[Equation]
Lemma 12.
Suppose Assumption 5 holds.
Then, with probability at-least 1 − δ , we must have for any ﬁxed ( s, a ) :
[Equation]
We refer to Section L.5 for the proof.
We now proceed with the proof of Theorem 3.
Recall the noiseless iteration ¯ w k deﬁned in the discussion following the statement of Lemma 4.
We deﬁne ¯ D k := w k, 0− ¯ w k .
Observe that we cannot apply Lemma 9 as in the proof of Theorem 2 where we used w = w k, 0in order to bound (cid:107) (cid:15) k (cid:107) ∞ .
This is because of data re-use which causes w k, 0to depend on the ‘variance’ term.
However, note that ¯ w k is a deterministic sequence and we can apply Lemma 9 and then use the fact that ¯ w k ≈ w k, 0to show a similar concentration inequality.
To this end, we prove the following lemma:
Lemma 13.
In the tabular setting, we have almost surely:
[Equation]
The proof follows from elementary arguments via.
the mean value theorem and triangle inequality.
We refer to Section M.7 for the proof.
We are now ready to give the proof of Theorem 3.
Proof of Theorem 3.
We proceed with a similar setup as the proof of Theorem 2.
Notice that due to data reuse, the noise (cid:15) k in outer loop k is given by (cid:15) 1 ( ˜ w k, 1 1 )
[Equation]
We also deﬁne the noiseless Q iteration such that ¯ Q 1 , 1 1 = ˜ Q 1 , 1 1 and ¯ Q k +1 , 1= T ( ¯ Q k, 1 1 ) .
Let ¯∆ k :=
˜ Q k, 1 1 − ¯ Q k, 1 1 .
Now, by Lemma 5, we can write down
[Equation]
We bound each of the terms above separately.
By union bound, the following statements all hold simulataneously with probability at-least 1 − δ .
By Theorem 7, whenever NB > C τ mix µ min log( |S||A δ ) , we must have ˜ N (1) ( s, a ) ≥ µ min 2 NB .
A simple observation using recursion shows that ˜ Q k, 1 1 ( s, a ) , ¯ Q k, 1 1 ( s, a ) ∈ [0 , 11 − γ ] .
Using Lemmas 12 and 13 we conclude that we must have:
[Equation]
Now observe that ¯ Q k, 1 1 is a deterministic sequence.
Therefore, using Lemma 9 uniformly for every k ≤ K and ( s, a ) ∈ S × A :
[Equation]
Where [Equation] Var P ( ¯ V k ) :=
Var P ( V )( s, a ; ¯ Q k, 1 1 ) .
We now combine all the above with Equation (28) to show that with probability at least (1 − δ ) , we must have uniformly for every k ≤ K and ( s, a ) ∈ S
× A :
[Equation]
We will now present a crude bound on (cid:107) ¯∆ k (cid:107) ∞ to reduce the analysis of Q-RexDaRe to the analysis of Q-Rex.
Claim 1.
Whenever η ≤ C
[Equation]
, with probability at-least 1 − δ , we must have for every k ≤ K uniformly:
[Equation]
Applying this directly to Equation (31) , we conclude that with probability at least (1 − δ ) , we must have uniformly for every k ≤ K and ( s, a ) ∈ S
× A :
[Equation]
Proof of Claim 1.
First note that T is γ contractive under the sup norm.
Therefore,
[Equation]
In Equation (31), note that Var P ( ¯ V k )( s, a ) ≤ 1 (1 − γ ) 2 .
Therefore, under the conditions of this Claim, we can take C 1 small enough so that uniformly for every ( s, a ) and k ≤ K with probability at-least (1 − δ ) , Equation (31) becomes:
[Equation]
Combining the display above and the fact that ˜ (cid:15) k
= ˜ (cid:15) 1 ( ˜ Q k, 1 1 ) in Equation (33), we conclude that with probability at-least 1 − δ , for every k ≤ K uniformly:
[Equation]
Unrolling the recursion above and using the fact that (cid:107) ¯∆ 1 (cid:107) ∞
= 0 , we conclude the statement of the claim.
We also note that the deterministic iterations ¯ Q k, 1converges exponentially in sup norm to Q ∗ due to γ contractivity of T .
That is:
[Equation]
We are now ready to connect up with the proof of Theorem 2 with minor modiﬁcations.
We follow the same analysis as the proof of Theorem 2 but with Var P ( ˜ V k ) replaced with Var P ( ¯ V k ) .
Similar to the proof of Theorem 2, we can control Var P ( ¯ V k ) with respect to Var P ( V ∗ ) and (cid:107) ¯ Q k, 1 1 − Q ∗ (cid:107) ∞ along with Equation (35).
We also replace α η with
[Equation]
Therefore
[Equation]
we conclude a version of Equation (23):
[Equation]
Where ∆ k :=
Q k, 1 1 − Q ∗ .
Using the bounds on the parameters given in the statement of the Theorem, we conclude the result.
K M INIMAX L OWERBOUNDS FOR T ABULAR MDP S
Suppose Θ denotes the class of all tuples ( M , π ) where M is a tabular MDP and π is an exploratory policy such that under the policy π , the MDP achieves a stationary distribution µ with minimum probability at-least µ min (see section 2.1).
The rewards are almost surely in the set [0,1].
We receive the stationary sequence ( s t , a t , r t )
Tt =1 from MDP M under policy π , which we denote as ( s t , a t , r t )
Tt =1 ∼ ( M , π ) .
Let F be the class of all estimators f which estimate Q ∗ for the MDP M with f (( s t , a t , r t )
Tt =1 ) .
We write the minimax risk as:
[Equation]
Theorem 5.
There exists a constant C such that, for every µ min ∈ (0 , 1 / 4) , γ ∈ (0 , 1 / 2) and
T ≥ C 1 µ min (1 − γ ) , we must have:
[Equation]
Therefore, we need T ≥
[Equation]
2 µ min (1 − γ ) 3 in order to achieve (cid:15) error for the Q values.
Proof.
We will use Le-Cam’s two point method to prove the result.
Consider a class of MDPs denoted by MDP( q, p ) ( p, q ∈ [0 , 1] ) with state space S
= { 0 , 1 } , only one possible action, reward function R : S
→ R , R ( s ) = s , and discount factor γ ∈ [0 , 1) .
The transition probability for the
MDP is given by P (0 | 0)
= 1 − q, P (1 | 0)
= q, P (0 | 1) =
p and P (1 | 1) = 1 − p .
Under these conditions, the stationary distribution is given by µ (0)
=
pp + q and µ (1) =
qp + q and the value function can be written as
[Equation]
where h ( q ) :
= γq 1 − γ (1 − q ) .
Since there is only one action in each state, the value function coincides with the Q function.
For showing the lower bound, consider two MDPs with parameters ( p 1 , q ) and ( p 2 , q ) with q < min { p 1 , p 2 } and p 1 , p 2 < 0 .
The stationary trajectories of length T under the two MDPs, denoted by the random variables
X ( i ) 1: T for i = 1 , 2 , satisﬁes
[Equation]
using KL ( Ber ( a ) ||
Ber ( b )) ≤ 2( a − b ) 2 b ln2 whenever b ≤ 1 / 2 .
Observe that µ (2)min = µ (2) (1) =
q p 2 + q and µ (1)min = µ (1) (1) =
q p 1 + q .
{+Combining this with the KL divergence bound above, we conclude that KL ( X (2)1: T || X (1)1: T ) < 
+}
if we set Now let p 1 = 1 − γ 2 , p 2 − p 1 = c (cid:113) p 1 Tµ (1)min for small enough constant c and T (cid:38) p ( p + q ) q .
(notice that the equation gives p 2 < 0 .
5 when T satisﬁes the condition above and p 1 < 0 . 25 ).
By Pinsker’s inequality, we must have TV ( X (2)1: T , X (1)1: T ) ≤ 12 .
Elementary calculations show that | V (1) (1) − V (2) (1) | (cid:38) | p 1 − p 2 | (1 − γ ) 2 =
[Equation]
[Equation]
2 (cid:113) p 1 Tµ (1)min since q < p 1 = 1 − γ 2 .
Therefore:
[Equation]
Consider the semi-metric ρ ( x, y )
= (cid:12)(cid:12)(cid:12) 1 1 −
[Equation]
(1 − x ) − γxh ( q ) − 1 1 − γ (1 − y ) − γyh ( q ) (cid:12)(cid:12)(cid:12) for x, y ∈ (0 , 1) .
Let
P i be the distribution of X ( i ) 1: T , and let the distribution be parameterised by θ ( P i ) =
V ( i ) (1) for i = 1 , 2 .
Let ˆ θ be any estimator based on the observations.
By (Wainwright, 2019c, Chapter 15),
[Equation]
= ⇒ for less than (cid:15) error, T has to be at-least
[Equation]
q =
[Equation]
1 (1 − γ ) µ min , we can ensure that µ
[Equation]
1)min = 2 µ min and µ (2)min ≥ µ min , which ensures that MDP( p 1 , q ) and MDP( p 2 , q ) are both elements of Θ .
Therefore, the two point risk in the LHS of Equation (37) lower bounds L (Θ , T ) , which allows us to conclude the result by substituting µ (1)min = µ min .
L P ROOFS OF C ONCENTRATION I NEQUALITIES L. P ROOF OF L EMMA
First, we leverage the techniques established in (Jain et al., 2021b, Lemma 28) to show Lemma 14.
The proof follows by a simple re-writing of the proof of the aforementioned lemma which uses a linear approximation (in η ) to ˜ H k,j 1 ,B .
We omit the proof for the sake of clarity.
Lemma 14.
Suppose ηB < 13 .
Then, the following PSD inequalities hold almost surely: I
[Equation]
Proof of Lemma 8.
We begin by proving the ﬁrst part.
Using Assumption 1 and the fact that the decoupled trajectory (˜ s, ˜ a ) t is assumed to be mixed at the start of every buffer
[Equation]
we begin by ﬁrst noting from Lemma 14 that:
[Equation]
Now, observe that:
[Equation]
[Equation]
Now note that ˜ H k,N 1 ,B is independent of ˜ H k,j 1 ,B for j ≤ N − 1 .
Therefore, taking conditional expectation conditioned on H k,j 1 ,B for j ≤ N − 1 in Equation (40) and using Equation (39), we conclude:
[Equation]
Applying the equation above inductively, we conclude the result.
We now prove Part 2.
We apply Markov’s inequality to Part 1 along with Lemma 2 to show that with probability at least 1 − δ :
[Equation]
L. P ROOF OF L EMMA
Proof.
We intend to apply Freedman’s inequality (Freedman, 1975) like in (Li et al., 2021, Theorem 4), but in an asynchronous fashion and with Markovian data.
Here, reverse experience replay endows our problem with the right ﬁltration structure.
Using Equation (9), we can write
[Equation]
Where X k,ji :
[Equation]
s
[Equation]
˜
[Equation]
[Equation] ˜ [Equation] k
[Equation]
i ) =
( s, a ) (cid:1) .
This allows us to deﬁne the sequence of sigma algebras F k,ji = σ ((˜ s k,lm , ˜ a k,lm ) :
( l > j and m ∈
[ B ]) or ( l = j and m ≤ i )) - that is, it is the sigma algebra of all states and rewards which appeared before and including (˜ s k,ji , ˜ a k,ji ) inside the buffer j and all the states in buffers l > j .
Notice that ˜ N k,ji ( s, a ) is measurable with respect to the sigma algebra F k,ji .
Using the fact that the buffers are independent, we conclude that:
[Equation]
It is also clear from our assumptions that | X j,ki | ≤ 21 − γ almost surely.
Consider the almost sure inequality for the sum of conditional variances:
[Equation]
[Equation]
We now apply (Li et al., 2021, Equation (144)
[Equation]
Theorem 4) with R = 11 − γ and σ 2 =
[Equation]
(cid:17) to conclude the result.
L. P ROOF OF T HEOREM
Proof.
For the sake of convenience [Equation] in this proof we will take σ 2 := 4(1 + (cid:107) w (cid:107) 2 φ ) .
Suppose λ ∈ R .
For 1 ≤ m ≤ N consider:
[Equation]
k,l 1 ,B (cid:33) (cid:62) x (cid:29) + λ (cid:28) x,
N (cid:88) j
= N − m + (cid:32) j +1 (cid:89) l = N ˜
H k,l 1 ,B (cid:33) ˜
L k,j ( w ) (cid:29)
X 0 :
[Equation] (cid:107) x (cid:107) [Equation] Here we use the convention that (cid:81) j +1 l = N ˜ H k,l 1 ,B
= I whenever j + 1 > N .
We claim that the sequence exp( X m ) forms a super martingale under an appropriate ﬁltration.
In this proof only, consider the sigma algebra F m to be the sigma algebra of all the state action reward tuples in buffers N, . . .
, N − m + 1 and let F 0 be the trivial sigma algebra.
Notice that exp( X m ) is F m measurable.
Lemma 15.
Fix N ≥ m ≥ 1 .
Suppose Y ∈ R
d is a F m − 1 measurable random vector.
Then,
[Equation]
In particular, taking Y = (cid:16)(cid:81) N − m +2 l = N ˜ H k,l 1 ,B (cid:17) (cid:62) x , we conclude that exp( X m ) is a super martingale with respect to the ﬁltration F m
Proof of Lemma 15.
In the proof of this lemma, we will drop the superscripts k, m for the sake of convenience and due to conditioning on F m − 1 , we will treat Y as a constant.
Now we deﬁne the natural ﬁltration on the buffer under consideration ( G i )
Bi
=1 where G i is the sigma algebra of the all state-action tuples from ( s 1 , a 1 ) , . .
, ( s i , a i ) and rewards ( r h ) 1 ≤ h<i .
Note that by the deﬁnition of ˜ L ( w ) , we write:
˜ L ( w ) =
(cid:80) Bi
=1 η ˜ (cid:15) i ( w ) ˜
H 1 ,i − 1 ˜ φ i .
With this in mind, for h ∈ [ B ] deﬁne ˜
L h ( w ) =
(cid:80) hi =1 η ˜ (cid:15) i ( w )
˜ H 1 ,i − 1 ˜ φ i .
Now, ˜ L ( w ) =
η ˜ (cid:15) B ( w ) ˜ H 1 ,B − 1 ˜ φ B + ˜ L B − 1 ( w )
Now, notice that the random variables (cid:104) Y, ˜ L B − 1 (cid:105) , (cid:104) Y, ˜ H 1 ,B − 1 ˜ φ B (cid:105) and (cid:104) Y,
˜ H k,m 1 ,B (cid:16) ˜
H k,m 1 ,B (cid:17) (cid:62) Y (cid:105) are
G B measureable.
Furthermore, we must have: E (cid:2) ˜
(cid:15) B ( w ) (cid:12)(cid:12) G B (cid:3) = 0 and | ˜ (cid:15) B ( w ) | ≤ 2(1 + (cid:107) w [Equation] φ ) ≤ √ 2 σ .
Therefore, applying conditional Hoeffding’s lemma, we have:
[Equation]
In the third step we have used the conditional version of Hoeffding’s lemma.
Now, consider Z :=
( ˜ H 1 ,B − 1 ) (cid:62) Y .
Clearly,
[Equation]
Here we have used the fact that η < 1 and (cid:107) ˜ φ B (cid:107) < 1 .
Using the bounds above in Equation (43)
[Equation]
we conclude:
[Equation]
Using Equation (44) recursively [Equation] we conclude the ﬁrst statement of the lemma.
The last part of the lemma follows easily from the deﬁnition of X m .
By Lemma [Equation] we conclude that exp( X m ) is a super martingale with respect to the ﬁltration F m .
Therefore, we must have:
[Equation]
It is also clear that X N ≥ λ
[Equation]
Applying Chernoff bound with we conclude that the concentration inequality in Equation (13).
We can then directly apply (Vershynin, 2018, Theorem 8.1.6) to Equation (13) in order to obtain uniform concentration bounds.
L. P ROOF OF L EMMA
Proof of Lemma 10.
By deﬁnition of ˜ (cid:15) k , we have: ˜ w k +1 , {+
 [Equation]+} + ˜ [Equation]
k . Therefore, for any ( s, a ) ∈ S × A , we must have:
[Equation]
Using the fact that R ( s, a ) ∈ [0 , 1] , we conclude:
[Equation]
By independence of outer-loops for the coupled data, we note that ˜ w k, 1is independent of the data in buffer k .
Therefore we can apply Theorem 4 (and resp.
Lemma 8) conditionally with w = ˜ w k, [-
 (and-]1(and resp.
g =
˜ w k, 1− w ∗ ), the bias variance decomposition given in Lemma 5 and the bound on (cid:107) w ∗ (cid:107) φ in Lemma 4 to conclude that with probability at-least 1 − δ , for every k ≤ K : [Equation]
[Equation]
We now choose constants C 1 and C 2 in the statement of the Lemma such that Equation 46 implies:
[Equation]
Using the equation above in Equation (45), we conclude:
[Equation]
Unrolling the recursion above and noting w 1 , 1 1 = 0 , we conclude that with probability at-least 1 − δ , we must have (cid:107) ˜ w k, 1 1 (cid:107) φ ≤ 41 − γ for every k ≤ K .
The bound in item 2 follows by using item 1,
Equation (46) and the fact that ˜ w k, 1is independent of the data in buffer k due to our coupling.
L. P ROOF OF L EMMA
Proof.
For the sake of convenience, we will take ¯ d = | supp ( P ( ·| s, a )) | and index supp ( P ( ·| s, a )) by [ ¯ d ] .
Consider Y ∈ {− 1 , 1 } ¯ d .
We consider the class of random variables indexed by elements of {− 1 , 1 } ¯ d :
[Equation]
The proof proceeds in a similar way to the proof of Lemma 9 via the Freedman inequality.
To bring out the similarities we deﬁne similar notation.
Consider the sequence of sigma algebras F k,ji for i ∈
[ B ] and j ∈ [ K ] as deﬁned in the proof of Lemma 9.
We now deﬁne
X k,ji ( Y ) :
= (1 − η ) ˜ N k,ji ( s,a ) (cid:88)
[Equation]
= a )
[Equation]
We note that E (cid:104)
X k,ji ( Y )
|F k,ji (cid:105) = 0 and | X k,ji ( Y ) | ≤ 2 almost surely and a simple calculation reveals that: E (cid:104)
| X k,j i ( Y ) | 2 |F k,j i (cid:105) ≤ (1 − η ) 2 ˜ N k,ji 1 (˜ s k,j i = s, ˜ a k,j i = a )
It is also clear that: ∆( Y ; s, a ) =
η (cid:80) Kj =1 (cid:80)
Bi
=1 X k,ji ( Y ) .
Apply Freedman’s concentration inequality, we conclude for any ﬁxed Y ∈ {− 1 , 1 } ¯ d and ( s, a ) ∈ S × A ,
P ( | ∆( Y ; s, a ) | > C (cid:112) η log(2 /δ ) + η log(2 /δ )) ≤ δ
Applying a union bound over all Y ∈ − 1 , 1 d , we conclude:
[Equation]
We complete the proof by noting that
[Equation]
M T ECHNICAL L EMMAS M. C OUPLING L EMMA
We ﬁrst introduce some useful notation: Let D k,j (resp.
˜
D k,j ) be the tuple of random variables ( s k,j i , a k,ji , r k,j i ) B +1 i =1 (resp.
(˜ s k,ji , ˜ a k,j i , ˜ r k,ji ) B +1 i =1 ).
Lemma 16.
Suppose
[Equation] [Equation]
Then, we can deﬁne the sequences ( s t , a t , r t ) and (˜ s t , ˜ a t , ˜ r t ) on a common probability space such that:
The tuples D k,j and ˜ D
k,j have the same distribution for every ( k, j ) , 2.
The sequence ˜ D k,j for k ≤ N, j ≤ K is i.i.d.
Equation (4) holds
Proof.
The proof of this lemma for the tabular case is a rewriting of the the proofs of Lemmas 1,2, in (Bresler et al., 2020).
For the general state space case, we apply appropriate modiﬁcations as pioneered in (Goldstein, 1979).
M. P ROOF OF L EMMA
Proof.
The ﬁrst part follows from the deﬁnitions.
For the second part, note that: (cid:107) x (cid:107) ≥ (cid:107) x (cid:107) φ follows from Cauchy-Schwarz inequality and Assumption 1.
For the reverse inequality we use
Assumption 4 to show that:
[Equation]
M. P ROOF OF L EMMA
Proof.
Existence is guaranteed by Deﬁnition (2) and uniqueness follows from the assumption that span (Φ) =
R d .
M. P ROOF OF L EMMA
Proof.
Suppose w 0 , ˜ w 0 ∈ R
d are arbitrary.
Let w
=
T ( w 0 ) and ˜ w 1 = T ( ˜ w 0 ) .
For any φ ( s, a ) , we have:
[Equation]
By Assumption 2, for every ﬁxed s (cid:48) there exist a (cid:48) max
[Equation]
s
[Equation]
s
[Equation]
) such that
[Equation]
and
[Equation]
Therefore for any s (cid:48) it holds that,
[Equation]
−
[Equation] (cid:107) [Equation] Combining this with Equation (48), we conclude the γ -contractivity of T .
By contraction mapping theorem, T has a unique ﬁxed point we conclude that it is w ∗ by considering Q ( s, a ) :
= (cid:104) w ∗ , φ ( s, a ) (cid:105) and showing that it satisﬁes the bellman optimality condition in Equation (2).
For the norm inequality, we note that since R ( s, a ) ∈ [0 , 1] by assumption and w ∗ =
T ( w ∗ ) , we have: |(cid:104) φ ( s, a ) , w ∗ (cid:105)| ≤ 1 + γ (cid:107) w ∗ (cid:107) φ .
Therefore, (cid:107) w ∗ (cid:107) ≤ 11 − γ .
The second norm equality follows from
Lemma 2.
M. P ROOF OF L EMMA
Proof.
We write the iteration in the outer-loop k as:
[Equation]
Using the fact that (cid:104) ˜ w k +1 , ∗ , ˜ φ k,j − i (cid:105) =
˜ R k,j − i + {+γ E s (cid:48) ∼ P ( ·| ˜ s k,j − i , ˜ a k,j − i ) sup a (cid:
+}
) ∈
A (cid:104) φ ( s (cid:48) , a (cid:48) ) , ˜ w k, 1 1 (cid:105) , and recalling the notation in (5),(6), we write:
[Equation]
, ∗ = (cid:104) I − η ˜ φ k,j − i [˜ φ k,j − i ] (cid:62) (cid:105) (cid:16) ˜ w k,ji − w k +1 , ∗ (cid:17) + η ˜ φ k,j − i (cid:15) k,j − i .
Therefore, ˜ w k,j +1− ˜ w k +1 , ∗ =
˜
H k,j 1 ,B ( ˜ w k,j 1 − ˜ w k
+1 , ∗ )
+ ˜ L k,j
Unfurling further, and using the fact that ˜ w k,N +1= ˜ w k +1 , 1, we conclude the statement of the lemma.
M. P ROOF OF L EMMA
We let ˜ N k,j ( s, a ) :
= (cid:80) B i =1 1 ((˜ s k,j i , ˜ a k,j i ) =
( s, a )) .
We want to show that the quantity (cid:80) Nj =1 ˜ N k,j ( s, a ) concentrates around its expectation.
Note that E ˜ N k,j ( s, a ) =
Bµ ( s, a ) and that ˜ N k,j ( s, a ) are i.i.d.
for j ∈ [ N ] .
From the proof of (Paulin, 2015, Theorem 3.4) and the fact that γ ps ≥ 1 2 τ mix , we conclude:
[Equation]
Following the Chernoff bound in the proof of (Paulin, 2015, Theorem 3.4), we conclude:
[Equation]
Taking t = 12 NBµ ( s, a ) , whenever B ≥ τ mix (by assumption), we must have:
[Equation]
for some constant C .
Further, via a union bound over all state action pairs ( s, a ) , we conclude the statement of the lemma.
M. P ROOF OF L EMMA
We ﬁrst prove a simple consequence of multivariate calculus.
Lemma 17.
Let f : R n → R be deﬁned by f ( x ) =
sup i x i .
Then, for every x, y ∈ R n , there exists ζ ∈ R d such that (cid:107) ζ (cid:107)
= 1 , ζ i ≥
and:
[Equation]
Proof.
We consider the log-sum-exp function.
Given L ∈
R + , deﬁne f L ( x ) :=
L log ( (cid:80) ni =1 exp( Lx i )) .
An elementary calculation shows that:
[Equation]
Therefore, for any ﬁxed x ,
[Equation]
Now, ∇ f L ( x ) =
( p 1 , . . .
, p n ) where p i = exp( Lx i ) (cid:80) nj =1 exp( Lx j ) .
Clearly, (cid:104)∇ f L ( x ) , e i (cid:105) ≥ 0 and (cid:107)∇ f L ( x ) (cid:107) 1 = 1 .
By the mean value theorem there exists β L such that:
[Equation]
Now, the simplex in R n (denoted by ∆ n ) is compact.
Therefore, there exists a sub-sequence
L k → ∞ such that lim k →∞ ∇ f L k ( β L k ) = ζ ∈ ∆ n .
Taking limit along the sub-sequence L k in Equation (51), we conclude the result.
Proof of Lemma 13.
In the tabular setting, we have the following expression using Equation (41):
[Equation]
Where
[Equation]
Now, by an application of Lemma 17, we show that for some η ( ·| s (cid:48) , w, v ) ∈ ∆( A ) , depending only on ( s (cid:48) , w, v ) , we have:
[Equation]
Now
[Equation]
using the deﬁnition of ˆ P k ( ·| s, a ) and ¯ P ( ·| s, a ) in the discussion preceding Lemma
[Equation]
we can simplify Equation (52) to:
[Equation]
[Equation]
We conclude the statement of the lemma by an application of the Holder inequality
[Equation]
M. P ROOF OF L EMMA
Proof.
We solve for f ( u ∗ )
=
u ∗ to obtain the relation: u ∗ =
α + β √ u ∗ , which after discarding the negative solution for √ u ∗ yields the unique solution: u ∗ = (cid:18) β
+
[Equation]
To prove the second part, consider for arbitrary x, y ∈ R + , the following hyper-contractivity:
[Equation]
The second step follows from the fact that |√ a − √ b | ≤ (cid:112) | a − b | .
Since u ∗ = f ( u ∗ ) , we apply the inequality above:
[Equation]
We then conclude the result by induction.