A Non-asymptotic Analysis of Non-parametric Temporal-Difference Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Temporal-difference learning is a popular algorithm for policy evaluation.
In thispaper, we study the convergence of the regularized non-parametric TD(0) algorithm,in both the independent and Markovian observation settings.
In particular, when
TD is performed in a universal reproducing kernel Hilbert space (RKHS), we proveconvergence of the averaged iterates to the optimal value function, even when itdoes not belong to the RKHS.
We provide explicit convergence rates that dependon a source condition relating the regularity of the optimal value function to the
RKHS.
We illustrate this convergence numerically on a simple continuous-state
Markov reward process.
Introduction
One of the main ingredients of reinforcement learning (RL) is the ability to estimate the long-termeffect on future rewards of employing a given policy.
This building block, known as policy evaluation,already contains crucial features of more complex RL algorithms, such as SARSA or Q-learning [59].
Temporal-difference learning (TD), proposed by [57], is among the simplest algorithms for policyevaluation.
The estimation of the performance of the policy is made through a value function.
It isupdated online , after each new observation of a couple composed of a state transition and a reward.
Although the formulation of TD is quite natural, its theoretical analysis has proved more challenging,as it combines two difﬁculties.
The ﬁrst one is that TD bootstraps , in the sense that it uses itsprevious – possibly inaccurate – predictions to correct its next predictions, because it does nothave access to a ﬁxed ground truth.
The second difﬁculty is that the observations are producedalong a trajectory following a ﬁxed policy ( on-policy ), hence they are correlated, which calls formore involved stochastic approximation tools compared to independent identically distributed ( i.i.d. )
samples.
Moreover, using off-policy samples, produced by a different policy than the one beingevaluated, can make the algorithm diverge [15].
Off-policy sampling is out of our scope in this paper
A third element which is not inherent to TD further complicates the
While TD was originally proposed in a tabular setting, its large-scale applicability has been greatly extended by its combination with parametric function approximation [16].
This enables the use of any linear or non-linear function approximation method to model the value function, including neural networks.
However, one can exhibit unstable diverging behaviors of TD even with simple non-linear approximation schemes [61].
This combination of difﬁculties has been coined the “deadly triad” by [58]
We argue that convergence can be obtained even with non-linear function approximation by making use of the non-parametric formalism of reproducing kernel Hilbert spaces (RKHS), involving linear approximation in inﬁnite-dimension.
Studying this case could bring us closer to understanding other universal approximators in practice neural networks
1.1 Contributions
We study the policy evaluation algorithm TD(0) in the non-parametric case, ﬁrst when the observationsare sampled i.i.d.
from the invariant distribution of the Markov chain resulting from the evaluatedpolicy, and then when they are collected from a trajectory of the Markov chain with geometric mixing.
In that sense we follow a similar outline as the analysis of [10] which is dedicated to the linear case.
The non-parametric formulation of TD closes the gap between the original tabular formulation
andthe parametric formulation which involves semi-gradients.
It allows the use of classical tools andtheory from kernel methods [19].
In particular, we highlight the central role of inﬁnite-dimensionalcovariance operators [5, 2] which already appear in the analysis of other non-parametric algorithms,like least-squares regression.
We study a regularized variant of TD, a widely used way of dealingwith misspeciﬁcation in regression.
Importantly, when the regularized TD approximation is run on aninﬁnite-dimensional RKHS which is dense in the space of square-integrable functions, then there isno approximation error and the algorithm converges to the true value function.
More precisely, weprovide a proof of convergence in expectation of TD without approximation error, even when the truevalue function does not belong to the RKHS, under a weaker source condition.
Furthermore, we givenon-asymptotic convergence rates related to this source condition, which measures the regularity ofthe true value function relative to the RKHS, e.g. , its smoothness if the RKHS is a Sobolev space [46].
Note that using a universal kernel [43] to obtain convergence of TD to the true value function is alsointeresting from a theoretical point of view.
Indeed it exempts us from a possibly tedious study ofthe approximation (or projection) error on a given basis, and simply removes an error term which ingeneral scales linearly with the horizon of the Markov reward process [44, 65].
In the rest of this section, we review the related literature.
In Sec. 2, we present the algorithm, alongwith generic results and notations.
In Sec. 3, we analyze a simpliﬁed version of the algorithm, namelypopulation TD in continuous time.
This allows to catch the main features of the analysis, whilepostponing the technicalities related to stochastic approximation.
Sec. 4 is dedicated to the analysis ofnon-parametric TD with i.i.d.
observations, while Sec. 5 consists in a similar analysis for correlatedobservations sampled from a geometrically mixing Markov chain.
Finally, in Sec. 6, we presentsimple numerical simulations illustrating the convergence results and the role the main parameters.
1.2 Related literature
Temporal-difference learning.
The TD algorithm was introduced in its tabular version by [57],with a ﬁrst convergence result for linearly independent features, later extended to dependent featuresby [24].
Further stochastic approximation results were proposed by [36] for the tabular case, andby [53] for the linear approximation case.
[61] provided a thorough asymptotic analysis of TDwith linear function approximation, while failure cases were already known [4].
A non-asymptoticanalysis was later proposed by [40] in the i.i.d.
sampling case with constant step size, concurrentlyto another approach extending to Markov sampling by [10].
Other problem-dependent bounds forlinear TD were derived around the same period [23, 55], along with an analysis of variance-reduced
TD [39, 64].
All of the analyses mentioned above focus either on the tabular or on the linearparametric TD algorithm.
A recent work by [42] deals with the batch counterpart of non-parametric
TD, namely the least-squares TD algorithm (LSTD), but they rather focus on the analysis of thestatistical estimation error.
Importantly, LSTD only requires ofﬂine computations and is not relatedto stochastic approximation.
Most closely related to our work is the non-parametric regularized TDsetting studied by [38].
However, their analysis is limited to the case where the optimal value functionbelongs to the RKHS.
This is not sufﬁcient to get rid of the approximation error term.
Also, we willshow later that regularization is not necessary in this case.
Furthermore, their analysis is restricted tothe i.i.d.
setting, for which we will require fewer regularity assumptions.
Kernel methods in RL.
To tackle large-dimensional problems, kernel methods have been combinedwith various RL algorithms, including approximate dynamic programming [48, 11, 6, 34], policyevaluation [22], policy iteration [32], LSTD [42], the linear programming formulation of RL [26],upper conﬁdence bound [29], or ﬁtted Q-iteration [30].
Such kernel methods often come along withpractical ways to reduce the computational complexity that grows with the number of observedtransitions and rewards [7, 38].
Stochastic approximation.
The analysis of TD requires tools from stochastic approximation [8],among which the ODE method [13].
Such tools are primarily designed for ﬁnite-dimensionalproblems.
Stochastic gradient descent (SGD) [14] is a speciﬁc instance of stochastic approximationthat has received extensive attention for supervised learning.
In particular, the role of regularizationof SGD for least-squares regression has been studied [17, 21], as well as the effect of of samplingdata from a Markov chain [45].
Finally, we use a formalism which is close to the analyses [28, 49, 9]of non-parametric SGD for least squares regression.
Problem formulation and generic results2.1 The non-parametric TD(0) algorithm
We consider a Markov reward process (MRP), i.e. , a Markov chain with a reward associated toeach state.
This is what results from keeping the policy ﬁxed in a Markov decision process (MDP)for policy evaluation.
We consider MRPs in discrete-time, but not necessarily with a countablestate space X .
Speciﬁcally, we use the formalism of Markov chains on a measurable state spacewhich uniﬁes discrete- and continuous-state Markov chains.
Formally, let X ⇢ R d a measurable setassociated with the  -algebra A of Lebesgue measurable sets.
Let ( x n ) n  1 a time-homogeneous Markov chain with Markov kernel  .
A Markov kernel [51, 37] is a mapping  : X ⇥ A !
[0 , 1] thathas the following two properties: (1) for every x 2 X ,  ( x, · ) is a probability measure on A , and (2)for every A 2 A ,  ( · , A ) is A -measurable.
If X is a countable set,  is represented by a transitionmatrix Q such that Q i,j :=
P ( j | i ) =
 ( i, { j } ) , for any i, j 2 X .
We deﬁne a reward function r : X ! R uniformly bounded by R < 1 , and a discount factor  2 [0 , 1) .
The aim of policy evaluation is to compute the value function of the MRP:
[Equation]
where the ( x n ) n  1 are drawn from the Markov chain.
A probability distribution p : A !
Ris a stationary distribution for  if for all A 2 A , p ( A ) =
R X  ( x, A ) p ( dx ) .
The existence anduniqueness of a stationary distribution p , along with the convergence of the Markov chain to p intotal variation, is ensured by ergodicity conditions.
A sufﬁcient condition is that the Markov chainis Harris ergodic, i.e. , it has a regeneration set, and is aperiodic and positively recurrent (see [1]and [31] for an exposition of Harris chains).
For discrete-state Markov chains, ergodicity conditionscan be expressed somewhat more simply, and any aperiodic and positive recurrent Markov chainhas a unique invariant distribution.
Throughout this paper, we assume that p is the unique invariantdistribution of the Markov chain, and that it has full support on X .
Only in Sec. 5, we will in additionassume that the Markov chain is geometrically mixing.
We deﬁne L 2 ( p ) , the set of squared integrable functions f : X ! R with respect to p , with thenorm k f k 2 L 2 ( p ) =
R X f ( x ) 2 p ( dx ) <
+ 1 .
We also consider a reproducing kernel Hilbert space H of
A -measurable functions, associated to a positive-deﬁnite kernel K : X ⇥ X ! R .
For all x 2 X , weuse the notation  ( x ) :
= K ( x, · ) for the mapping of x in H , and h · , · i H for the inner product in H(we sometimes drop the index).
We assume that M H :=
sup x 2 X K ( x, x ) is ﬁnite, which impliesthat H ⇢ L 2 ( p ) .
More precisely, the H -norm controls the L 2 ( p ) -norm: any sequence convergingin H thus converges in L 2 ( p ) .
Indeed, if f 2 H :
[Equation]
We also assume that r 2 L 2 ( p ) .
The non-parametric TD(0) algorithm in the RKHS H is deﬁnedas follows [48, 38].
Draw a sequence ( x n ) n [-≥ 
-] 0 according to the Markov chain with initial distribution p , and collect the corresponding rewards ( r ( x n )) n  0 .
Deﬁne a sequence of non-negative stepsizes ( ⇢ n ) n  1 .
We build recursively a sequence of approximate value functions ( V n ) n  0 in L 2 ( p ) .
Throughout the paper, we take V
= 0 for simplicity, but note that all the results can be adapted tothe case V 0 2 H .
For n  1 :
[Equation]
where x 0 n := x n +1 .
The term in brackets is called a temporal-difference.
Equivalently, in the RKHS:
[Equation]
This update has a running time complexity of O ( n 2 ) , which can be improved to O ( n ) , e.g. using
Nyström approximation or random features [35].
More details on the implementation are givenin App.
B.2.
This non-parametric formulation is a natural extension of the tabular TD algorithm.
Indeed, if X is a countable set and K ( x, y ) = 1 x = y is a Dirac kernel – a valid positive-deﬁnitekernel – then we exactly recover tabular TD: the update rule (3) becomes, after observing a transition( i, i 0 , r i ) :
= ( x n , x 0 n , r ( x n )) :
[Equation]
This also covers the semi-gradient formulation of TD for linear function approximation [59].
Suppose H has ﬁnite dimension d , then V n can be identiﬁed to ⇠ n 2 R d , and we are searching for anapproximation of the form V n ( x ) = ⇠ > n  ( x ) .
Then (4) becomes:
[Equation]
Since V 0 2 H , all the iterates V n are in the RKHS, in particular V n 2 span {  ( x k ) } 1  k  n .
Conse- 141 quently, if the sequence ( V n ) converges in the topology induced by the L 2 ( p ) -norm, it converges in
H , the closure of H with respect to the L 2 ( p ) -norm.
In particular, for a dense RKHS and because phas full support on X , H =
L 2 ( p ) , but in general it only holds that H ⇢ L 2 ( p ) .
To understand the behavior of the algorithm, we will ﬁrst consider the population version (also calledmean-path in [10]) of the algorithm: set V
= 0 and for n  1 :
[Equation]
where the expectation is taken with respect to q ( dx, dx 0 ) :
= p ( dx )  ( x, dx 0 ) .
Since V n  1 2 H , thereproducing property holds: V n  1 ( x ) =
h V n  1 ,  ( x ) i H .
Hence the update is afﬁne and reads: V n = 148 V n  1 + ⇢ n ( AV n  1 + b ) , with A :=
E q [   ( x ) ⌦  ( x 0 )   ( x ) ⌦  ( x )] and b :=
E p [ r ( x )  ( x )] ,where ⌦ denotes the outer product in H deﬁned by g ⌦ h : f 7! h f, h i H g .
2.2 Covariance operators
Assume that the expectations ⌃ :=
E p [  ( x ) ⌦  ( x )] and ⌃ 1 :=
E q [  ( x ) ⌦  ( x 0 )] are well-deﬁned.
⌃ and ⌃ 1 are the uncentered auto-covariance operators of order 0 and 1 of the Markov process( x n ) n  1 , under the invariant distribution p .
They are operators from H to H , such that, for allf, g 2 H , using the reproducing property:
[Equation]
0 ) i H ] =
h f, E p
[ h g,  ( x 0 )
i H  ( x )]
i H = h f, ⌃ 1 g i H . In particular, for all y 2 X and f 2 H , ( ⌃ f )( y ) = h  ( y ) , ⌃ f i H =
E p
[ f ( x ) K ( x, y )] and similarly,( ⌃ 1 f )( y ) =
E q [ f ( x 0 ) K ( x, y )] .
Following [28], ⌃ and ⌃ 1 can therefore be extended to operators ⌃ eand ⌃ e 1 from L 2 ( p ) to L 2 ( p ) deﬁned by:
[Equation]
dx ) , such that 8 y 2 X , ( ⌃ e f )( y ) =
E p [ f ( x ) K ( x, y )]
[Equation]
, dx
[Equation]
, such that 8 y 2 X , ( ⌃ e 1 f )( y ) = E q [ f ( x 0 ) K ( x, y )] .
[Equation]
These two operators are the building blocks of the TD iteration (7).
In particular, A =  ⌃ 1  ⌃ andb =
⌃ e r , the latter being valid for r 2 L 2 ( p ) .
With a slight abuse of notation, we denote simply as ⌃ ,⌃ 1 the extended operators.
Furthermore [28], Im ( ⌃ ) ⇢ H and ⌃ 1 / 2 is an isometry from L 2 ( p ) to H :
[Equation]
The fact that p is a stationary distribution for  implies a particular constraint linking ⌃ and ⌃ 1 :
Lemma 1.
There exists a unique bounded linear operator ˜ ⌃ 1 : H ! H such that ⌃ 1 = ⌃ 1 / 2 ˜ ⌃ 1 ⌃ 1 / 2on H , and k ˜ ⌃ 1 k op  1 ( k · k op is the H -operator norm).
This results from an application of [5, Thm. 1], valid on H and extended by continuity to H .
Seealso [33] for an exposition of cross-covariance operators speciﬁcally in an RKHS.
In ﬁnite dimension,this is retrieved with generic results on positive semi-deﬁnite (PSD) matrices.
Speciﬁcally, if H ⇢ R m ,the uncentered covariance matrix of the random variable (  ( x ) ,  ( x 0 )) , when ( x, x 0 ) ⇠ q is:
✓ ⌃ ⌃
[Equation] Using a classical condition on block matrices [12, Prop. 1.3.2], this matrix is PSD if and only if thereexists a matrix ˜ ⌃ 1 such[-that (cid:107) ˜Σ 1 (cid:107) op ≤ 1 and Σ 1 = Σ 1 / 2 ˜Σ 1 Σ 1 / 
-]
that k ˜ ⌃ 1 k op  1 and ⌃ 1 = ⌃ 1 / 2 ˜ ⌃ 1 ⌃ 1 / 2 ( k · k op is also the spectral norm inthis case).
This corresponds to the fact that the Schur complement of a PSD block matrix is also PSD.
Assumptions on ⌃ and V ⇤ .
We assume that x 7! K ( x, x ) is uniformly bounded by M H .
Therefore, the eigenvalues of ⌃ are upper-bounded.
However, unlike [61] and [10], we do not assume themto be lower-bounded, i.e. , ⌃ ⌫ 0 is not invertible in general.
We will formulate our convergenceresults for two sets of assumptions.
The ﬁrst one recovers known results from [10] for linear functionapproximation.
The second one assumes that V ⇤ veriﬁes a source condition [27, Chap. 1]:
(A1) V ⇤ 2 H , H is ﬁnite-dimensional and ⌃ has full-rank;
(A2) V ⇤ 2 ⌃ ✓ / 2 ( H ) for some ✓ 2 (  1 , 1] (and consequently, k ⌃  ✓ / 2 V ⇤ k H < + 1 ), and
H =
L 2 ( p ) ( i e. , K is a universal kernel).
In (A1) , H is ﬁnite-dimensional because ⌃ cannot be simultaneously compact ( x 7! K ( x, x )being uniformly bounded) and invertible in inﬁnite-dimension [18].
Recalling the isometry property (10), the case ✓ =  1 always holds in (A2) because V ⇤ 2 L 2 ( p ) (which we prove in thenext subsection).
The case ✓ = 0 is equivalent to V ⇤ 2 H .
For ✓ > 0 , it must be interpreted as:
[Equation]
2 V } , with k ⌃  ✓ / 2 V ⇤ k H = + 1 if V ⇤ / 2 ⌃ ✓ / 2 ( H ) .
Using a universal approximation removes the need for a projection operator on H , as typically usedfor ﬁnite-dimensional function approximation, and hence there will be no projection error [61].
2.3 Non-expansiveness of the Bellman operator
It is known that the value function V ⇤ of the MRP is a ﬁxed point of the Bellman operator T .
Wedeﬁne two operators P and T : L 2 ( p ) !
L 2 ( p ) by, for V 2 L 2 ( p ) , PV ( x ) =
E x 0 ⇠  ( x, · ) V ( x 0 ) and
TV ( x ) =
r ( x ) +  PV ( x ) .
Both operators can be expressed in terms of ⌃ and ⌃ 1 .
For V 2 L 2 ( p ) :
[Equation]
Lemma 2.
For any V 2 L 2 ( p ) : k PV k L 2 ( p )  k V k L 2 ( p ) .
This is a direct reformulation of [61, Lemma 1], the proof of which is given in App.
A.1.
As stressedby [61], this strongly relies on the fact that p is a stationary distribution of the Markov chain.
Itimplies that T is a  -contraction mapping on L 2 ( p ) and has as unique ﬁxed point V ⇤ .
One cancheck that if ⌃ is non-singular, Lemma 2 is exactly equivalent to k ⌃  1 / 2 ⌃ 1 ⌃  1 / 2 k op  1 , that is,
Lemma 1.
Moreover, using Lemma 2, we obtain k V ⇤ k L 2 ( p )  k r k L 2 ( p ) / (1   ) and V {+⇤ 
+}
L 2 ( p ) .
Analysis of a continuous-time version of the population TD algorithm
Before considering regularized TD with stochastic samples, we look at simpliﬁed versions of thealgorithm that momentarily remove the difﬁculties related to stochastic approximation.
Speciﬁcally,we consider the population version of TD to capture a “mean” behavior, and a continuous-timealgorithm to avoid choosing step sizes.
Instead, we focus on the role of the regularization parameter.
3.1 Existence of a ﬁxed-point for regularized TD
For   0 , let us consider the regularized population recursion:
[Equation]
If the TD iterations converge, its limit will be a solution of the regularized ﬁxed point equation:
[Equation]
Proposition 1.
If  > 0 ,  ⌃ 1  ⌃   I is non-singular on H and the ﬁxed point equation (13) admitsa unique solution V ⇤  in L 2 ( p ) , deﬁned by V ⇤
= (  ⌃ 1  ⌃   I )  1 ⌃ r .
Furthermore, V ⇤  2 H and:
[Equation]
The proof is in App.
A.2.
Hence, for  > 0 , the H -norm of V ⇤  is always bounded, unlike k V ⇤ k H .
3.2 Convergence of the regularized ﬁxed point to the optimal value function
Recalling that V ⇤ 2 L 2 ( p ) , it satisﬁes the relation TV ⇤ =
V ⇤ , implying that ⌃ TV ⇤ =
⌃ V ⇤ , i.e. ,⌃ r + (  ⌃ 1  ⌃ ) V ⇤ = 0 .
This unregularized ﬁxed point equation possibly has other solutions, butif K is a universal kernel, as assumed by (A2) , then ⌃ is injective [56] and V ⇤ is the unique solution.
Let us recall that (A2) does not imply that V ⇤ has a bounded H -norm.
However, we can control the
L 2 ( p ) -norm of V ⇤   V ⇤ when  is small using the source condition (A2) .
Proposition 2.
Assume that  > 0 and assumption (A2) .
Then:
[Equation]
The proof in App.
A.2 is inspired by similar results [17, 21] in the context of ridge regression(recovered for  = 0 ).
Note that only k V ⇤   V ⇤ k L 2 ( p ) is controlled, not k V ⇤   V ⇤ k H .
Consequently,we obtain the convergence of V ⇤  to V ⇤ in L 2 ( p ) -norm when  ! 0 : the higher ✓ is, the faster the rateof convergence.
For universal Mercer kernels [20], if we drop the source condition (A2) , using onlythe fact that V ⇤ 2 L 2 ( p ) – corresponding to ✓ =  1 in (A2) – we can still prove that V ⇤  convergesto V ⇤ in L 2 ( p ) -norm when  ! 0 , but without an explicit rate (see App.
A.2, Cor. 1).
3.3 Convergence of continuous-time population TD
Following the ordinary differential equation (ODE) method [13], we study the continuous-timecounterpart of the population iteration (12).
At least formally, this consists in deﬁning e
V t = V n ( t ) for tand n ( t ) satisfying t =[-(cid:80) n ( t ) i =
-]
P n ( t ) i =1 ⇢ i , and letting ⇢ i tend to 0 for any i  1 , where V n ( t ) is deﬁned byrecursion using (12).
With a slight abuse of notation, we use the notation V t instead of e V t .
We thenobtain the following ODE in H :
V 0 = 0 and for t  0 :
[Equation]
[Equation]
We can exhibit a Lyapunov function for this dynamical system, see [54].
This implies that V tconverges to V ⇤  when t tends to inﬁnity, where V ⇤  is deﬁned in Prop. 1.
More precisely, for 2 {  1 , 0 } , we deﬁne W  , the Lyapunov function, by W  ( t ) :
= k ⌃   / 2 ( V t  V ⇤  ) k 2 H (pleasenote that  ’s role in W  is an index, not a power).
W 0 ( t ) strictly decreases with t as follows:
Lemma 3 (Descent Lemma) .
For  > 0 , for all t  0 , the following holds:
[Equation]
The proof mainly relies on the contraction property of the Bellman operator (see App. A.2).
We canthen deduce the convergence of the ODE (16) to V ⇤  .
Proposition 3.
Under assumption (A1) , the solution V t of the ODE (16) with  = 0 is such that:
[Equation]
where V T is the Polyak-Ruppert average [50] of V t , deﬁned by V T := 1 T [-(cid:82) T 
-]R
T 0 V t dt .
Under assumption (A2) , the solution V t of the ODE (16) with  > 0 is such that:
For T
[Equation]
Under (A1) , we recover the same O (1 /T ) convergence rate as [10].
We focus on (A2) , where weget a fast convergence to V ⇤  in H -norm (stronger than L 2 ( p ) ).
However, we are rather interested inconvergence to V ⇤ .
Prop. 2 quantiﬁes how far V ⇤  is from V ⇤ .
Indeed, the error decomposes as:
[Equation]
Combining Propositions 1, 2, 3 shows a trade-off on  : k V T  V[-∗ (cid:107) 
-]
⇤ k 2 L 2 ( p
[-O (cid:0) e − 2 λT /λ 2 + λ θ +
-][Equation] =
[Equation]
Taking
= (3 + ✓ ) log T/ (2 T ) balances the terms up to logarithmic factors: k V T  V ⇤ k 2 L 2 ( p ) =
˜ O  T  1  ✓  (where ˜ O ( g ( n )) :
= O ( g ( n ) log( n ) ` ) , for some ` 2 R ).
In particular, for ✓ = 0 ,i.e. , V ⇤ 2 H , we recover a convergence rate ˜ O (1 /T ) :
up to logarithmic factors, it is the same as theunregularized case with averaging, assuming (A1) . In this case, regularization brings no beneﬁts.
Stochastic TD with i.i.d. sampling
We now consider stochastic TD iterations (4), where the couples ( x n , x 0 n ) n  1 are sampled i.i.d.
fromthe distribution q ( dx, dx 0 ) =
p ( dx )  ( x, dx 0 ) .
Such i.i.d.
samples can be obtained by running the
Markov chain until it has mixed so that x n ⇠ p , collecting a couple ( x n , x 0 n ) , and restarting.
With
A n :=   ( x n ) ⌦  ( x 0 n )   ( x n ) ⌦  ( x n ) and b n := r ( x n )  ( x n )
, we study the recursion: [Equation]
[Equation]
In particular, E q [ A n ] =
A , E p [ b n ] = b , and A n and b n are independent of the past ( V k ) k<n .
For  2 { 0 , 1 } , let W  n := k ⌃   / 2 ( V n  V ⇤  ) k 2 H .
Adapting the proof of Lemma 3, we exhibit asimilar decreasing behavior of W 0 n in expectation, hence showing that E [ k V n  V ⇤  k 2 H ] {+! 
+} forwell-chosen step sizes ⇢ n .
Finally,  is chosen to balance E [ k V n  V ⇤  k 2 L 2 ( p ) ] and k V ⇤   V ⇤ k 2 L 2 ( p ) .
We deﬁne V (e) n and V (t) n as the exponentially-weighted and the tail-averaged n -th iterates respectively:
[Equation]
Theorem 1.
Let n  9 .
Under assumption (A2) with  1 < ✓  1 , there exist a positive real 256 number  ✓ independent of n such that, for  0   ✓ ,
(a) Using  =  0 n  13+ ✓ and a constant step size ⇢ = log n  n , then:
[Equation]
(b) Using  =  0 n  12+ ✓ and a constant step size ⇢ = log n  n , then:
[Equation]
(c) Using  =  0 n  12+ ✓ and a constant step size ⇢ = 2log n  n for the ﬁrst b n/ 2 c  1 iterates and then a decreasing step size ⇢ k = 1  k , then:
[Equation]
A similar exponentially-weighted averaging scheme as in (b) has been used to study constant stepsize SGD in [25].
When  = 0 , the rates can be compared to existing results for SGD.
For example,for ✓ 2 [0 , 1] , [60] proves almost sure convergence for regularized least-mean-squares withoutaveraging at rate O ( n  1+ ✓ 2+ ✓ ) .
The dependence in ✓ is similar to what we obtain.
Moreover, underassumption (A1) , we recover the same O (1 / p n ) convergence rate as [10] (see Prop. 4 stated in
App. A.3).
Finally, our bounds have a polynomial dependence in the horizon 1 / (1   ) of the MRP.
Stochastic TD with Markovian sampling
We now consider the truly online TD algorithm, where the samples are produced by a Markovchain.
In particular, there is now a correlation between the current samples ( x n , x 0 n ) and the previousiterate V n  1 .
To control it, we assume that the Markov chain mixes at uniform geometric rate:
[Equation]
where d TV denotes the total variation distance.
This is always veriﬁed for irreducible, aperiodic ﬁnite
Markov chains [41].
We give an example of continuous-state Markov chain with geometric mixing in
Sec. 6.
Furthermore, following [10], in our analysis we need to control the magnitude of the iteratesalmost surely.
To do so, we add a projection step at each TD iteration:
[Equation]
where ⇧ B is the projection on the H ball of radius B > 0 .
If k V ⇤  k H  B , the convergence ofthe method is preserved.
In the following theorem, we consider two regimes with different rates ofconvergence.
In the ﬁrst one, we assume like [10] that we are given an oracle B upper-boundingk V ⇤  k H , with B independent of  .
In the second one, we use the bound of Prop. 1, but this will affectthe convergence rate since in this case B
=
O
(1 /  ) .
Theorem 2.
Assuming (A2) and that the samples are produced by a Markov chain with uniformgeometric mixing (A3) , the projected TD iterations (24) are such that:(i) Using  = n  12+ ✓ , a constant step size ⇢ = log n 2  n , and using a projection radius Bindependent of  provided by an oracle and such that k V ⇤
k H  B , then:
[Equation]
(ii) Using  = n  14+ ✓ , ⇢ = log n 2  n , and the projection radius B = p M H k r k L 2 ( p
[Equation]
/  , then:
[Equation]
[Equation]
with V (e) n =
P
[Equation]
When an oracle is given for B (i.e., setting (i)), we recover the same rate as i.i.d.
sampling, up toa multiplicative factor log( n ) / log(1 /µ ) which represents the mixing time of the Markov chain.
Ifno oracle is provided (i.e., setting (ii)), the convergence will be slower because the bound B is oforder 1 /  .
Note that the slight changes in the deﬁnitions of ⇢ ,  , V ( e ) , and the absence of constrainton  , as compared to Thm. 1, are implied by the boundedness of the iterates.
Following a similarstudy for SGD [45], we might compare these rates to those of a naive algorithm which we call“ ⌧ -Skip-TD”, for some ⌧  1 , where only one every ⌧ samples from the Markov chain is used tomake TD updates: [Equation]
[Equation]
For ⌧ large enough, of the order of the mixing time of the Markov chain, the new sample ( x n ⌧ , x 0 n ⌧ )is almost independent from the past ones ( x k ⌧ , x 0 k ⌧ ) k<n .
Of course, since we need to generate ⌧times more samples to make a step, we must look at the distance of V n/ ⌧ to the optimum.
Suchconvergence rates for ⌧ -Skip-TD are derived in App.
A.4, Cor. 2.
In setting (i) , they are similar to
Theorem 2 up to a log( n ) factor.
This suggests that making updates at each sample of the Markovchain is not more efﬁcient than ⌧ -Skip-TD for large ⌧ , at least in our theoretical analysis.
In practice,using all samples seems slightly better, especially for a slowly mixing Markov chain (see App.B.3).
In setting (ii) , we obtain a rate for Skip-TD whose leading term does not depend on log(1 /µ ) – whichonly appears in higher order terms – suggesting that the rate and parameters of Thm. 2, setting (ii)might be suboptimal.
Experiment on artiﬁcial data
Building a value function.
We build a toy model for which the main parameters can be computedin closed form.
We consider the dynamics on the circle X
=
[0 , 1] deﬁned by: with probability " ,x n +1 ⇠ U ([0 , 1]) , and with probability 1  " , x n +1 = x n .
Because the Markov kernel is symmetric,the invariant distribution is p =
U ([0 , 1]) .
In particular, the mixing parameter can be boundedexplicitly with m = 1 and µ = 1  " (see App. B.1).
Also, simple computations show that V ⇤ isan afﬁne transform of r : V ⇤ ( x ) =
ar ( x ) + b , with a = (1   (1  " ))  1 and b =  a R 1 0 r ( u ) du .
Hence we can build a V ⇤ with a given regularity by choosing an appropriate reward with the sameregularity.
We consider two rewards: r abs ( x ) :
= 2 | x  1 / 2 | and r cos ( x ) :
= (1 + cos(2 ⇡ x )) / 2 .
Kernels on the torus.
We consider the RKHS of splines on the circle [62] of regularity s 2 N ⇤ ,denoted by H s per .
It is a Sobolev space equipped with the following norm:
[Equation]
Its corresponding reproducing kernel K s is a translation-invariant kernel deﬁned by:
[Equation]
where { x } := x  b x c and B j is the j -th Bernoulli polynomial [47].
Let us recall that the Fourierseries expansion on the torus of a 1-periodic function f 2 L 2 ( p ) is: f ( x ) = P ! 2 Z e 2 i !⇡ x ˆ f ! ,with ˆ f [-ω := (cid:82) 
-]! :
= R 1 0 f ( x ) e  2 i ! ⇡ x dx , for ! 2 Z .
The kernel K s has an embedding in the space of
Fourier coefﬁcients  ( x ) =
( p c ! e 2 i !⇡ x ) > m 2 Z with c ! :
= | ! |  2 s if ! 6 = 0 and c 0 : = 1 .
Using
Parseval’s theorem in Eqn.
(28), one can compute the norm of f from its Fourier coefﬁcientsk
f k 2 H s per = P !
2 Z | ˆ f ! | 2 /c ! .
The operators ⌃ and ⌃ 1 can be represented as countably inﬁnitedimensional matrices ⌃ =
diag ( c ) and ⌃ 1 = (1  " ) ⌃ + " p c ( p c ) > .
Hence the source conditionstates that | ˆ f 0 | 2 +
[-(cid:54) =
-][Equation]
| 2 < 1 .
In particular, it holds if f 2 H s 0 per , for any s 0  s (1+ ✓ ) .
In our example, we consider two Sobolev spaces H 1 per and H 2 per , and our two example functions have
Fourier coefﬁcients (ˆ r abs ) !
= 1  (  1) !
⇡ 2 !for !
6 = 0 , and (ˆ r cos ) ! = 0 for | ! | > 1 .
The largest ✓ 2 [0 , 1]such that the source condition holds are indicated in the ﬁrst row of Tab. 1.
Results.
We run TD on functions r abs and r cos , with kernels K 1 and K 2 .
We use parameters  and ⇢and exponential averaging as prescribed in Thm.
1 (b) .
Each experiment is repeated 10 times andwe record the mean ± one standard deviation.
The implementation is based on a ﬁnite dimensionalrepresentation of the iterates ( V k ) k  n in R n (see further details in App.
B.2).
This implies computingthe kernel matrix in O ( n 2 ) operations.
To accelerate this computation when the eigenvalues decreasefast, we approximate it with the incomplete Cholesky decomposition [3].
In Tab. 1, we set " = 0 .
8 , = 0 .
5 and report the observed convergence rates v.s.
the ones expected by Thm. 2, which are fairlyconsistent.
In Fig.
1, we show the respective effects of varying " and  .
Larger values of " or  makethe problem more difﬁcult and slow down convergence, presumably in the constants without affectingthe rates, as predicted by Thm. 2.
Additional experiments are provided in App.
B.3.
[Table]
[Figure]
Conclusion
We have provided convergence rates for the regularized non-parametric TD algorithm in the i.i.d.
and
Markovian sampling settings.
The rates depend on a source condition that quantiﬁes the relativeregularity of the optimal value function to the RKHS.
They are compatible with our empiricalobservations on a one-dimensional MRP, but we have not proved optimality of such rates.
Interestingdirections include the extension to the TD(  ) algorithm, which we believe can be achieved withsimilar tools, as well as more challenging extensions to control counterparts of TD (Q-learning,
SARSA,...) for which the policy is optimized.