PER-ETD: A P OLYNOMIALLY E FFICIENT E MPHATIC T EMPORAL D IFFERENCE L EARNING M ETHOD
Emphatic temporal difference (ETD) learning (Sutton et al., 2016) is a successful method to conduct the off-policy value function evaluation with function approximation.
Although ETD has been shown to converge asymptotically to a desirable value function, it is well-known that ETD often encounters a large variance so that its sample complexity can increase exponentially fast with the number of iterations.
In this work, we propose a new ETD method, called PER-ETD (i.e.,
PE riodically R estarted-ETD), which restarts and updates the follow-on trace only for a ﬁnite period for each iteration of the evaluation parameter.
Further, PERETD features a design of the logarithmical increase of the restart period with the number of iterations, which guarantees the best trade-off between the variance and bias and keeps both vanishing sublinearly.
We show that PER-ETD converges to the same desirable ﬁxed point as ETD, but improves the exponential sample complexity of ETD to be polynomials.
Our experiments validate the superior performance of PER-ETD and its advantage over ETD.
I NTRODUCTION
As a major value function evaluation method, temporal difference (TD) learning (Sutton, 1988;
Dayan, 1992) has been widely used in various planning problems in reinforcement learning.
Although TD learning performs successfully in the on-policy settings, where an agent can interact with environments under the target policy, it can perform poorly or even diverge under the off-policy settings when the agent only has access to data sampled by a behavior policy (Baird, 1995; Tsitsiklis &
Van Roy, 1997; Mahmood et al., 2015).
To address such an issue, the gradient temporal-difference (GTD) (Sutton et al., 2008) and least-squares temporal difference (LSTD) (Yu, 2010) algorithms have been proposed, which have been shown to converge in the off-policy settings.
However, since
GTD and LSTD consider an objective function based on the behavior policy, which adjusts only the distribution mismatch of the action and does not adjust the distribution mismatch of the state, their converging points can be largely biased from the true value function due to the distribution mismatch between the target and behavior policies, even when the express power of the function approximation class is arbitrarily large (Kolter, 2011).
In order to provide a more accurate evaluation, Sutton et al.
(2016) proposed the emphatic temporal difference (ETD) algorithm, which introduces the follow-on trace to address the distribution mismatch issue and thus adjusts both state and action distribution mismatch.
The stability of ETD was then shown in Sutton et al.
; Mahmood et al.
(2015), and the asymptotic convergence guarantee for ETD was established in Yu (2015), it has also achieved great success in many tasks (Ghiassian et al., 2016; Ni, 2021).
However, although ETD can address the distribution mismatch issue to yield a more accurate evaluation, it often suffers from very large variance error due to the follow-on trace estimation over a long or inﬁnite time horizon (Hallak et al., 2016).
Consequently, the convergence of ETD can be unstable.
It can be shown that the variance of ETD can grow exponentially fast as the number of iterations grow so that ETD requires exponentially large number of samples to converge.
Hallak et al.
(2016) proposed an ETD method to keep the follow-on trace bounded but at the cost of a possibly large bias error.
This thus poses the following intriguing question:
Can we design a new ETD method, which overcomes its large variance without introducing a large bias error, and improves its exponential sample complexity to be polynomial at the same time?
In this work, we provide an afﬁrmative answer.
1. M AIN C ONTRIBUTIONS
We propose a novel ETD approach, called PER-ETD (i.e., PE riodically R estarted-ETD), in which for each update of the value function parameter we restart the follow-on trace iteration and update it only for b times (where we call b as the period length ).
Such a periodic restart effectively reduces the variance of the follow-on trace.
More importantly, with the design of the period length b to increase logarithmically with the number of iterations, PER-ETD attains the polynomial rather than exponential sample complexity required by ETD.
We provide the theoretical guarantee of the sample efﬁciency of PER-ETD via the ﬁnite-time analysis.
We show that PER-ETD (both PER-ETD(0) and PER-ETD( λ )) converges to the same ﬁxed points of ETD(0) and ETD( λ ), respectively, but with only polynomial sample complexity (whereas
ETD takes exponential sample complexity).
Our analysis features the following key insights.
(a)
The period length b plays the role of trading off between the variance (of the follow-on trace) and bias error (with respect to the ﬁxed point of ETD), and its optimal choice of logarithmical increase with the number of iterations achieves the best tradeoff and keeps both errors vanishing sublinearly.
(b) Our analysis captures how the mismatch between the behavior and target policies affects the convergence rate of PER-ETD.
Interestingly, the mismatch level determines a phase-transition phenomenon of PER-ETD: as long as the mismatch is below a certain threshold, then PER-ETD achieves the same convergence rate as the on-policy TD algorithm; and if the mismatch is above the threshold, the converge rate of PER-ETD gradually decays as the level of mismatch increases.
Experimentally, we demonstrate that PER-ETD converges in the case that neither TD nor ETD converges.
Further, our experiments provide the following two interesting observations.
(a) There does exist a choice of the period length for PER-ETD, which attains the best tradeoff between the variance and bias errors.
Below such a choice, the bias error is large so that evaluation is not accurate, and above it the variance error is large so that the convergence is unstable.
(b) Under a small period length b , it is not always the case that PER-ETD( λ ) with λ = 1 attains the smallest error with respect to the ground truth value function.
The best λ depends on the geometry of the locations of ﬁxed points of PER-ETD( λ ) for 0 ≤ λ ≤ 1 , which is determined by chosen features.
1. R ELATED W ORKS
TD learning and GTD: The asymptotic convergence of TD learning was established by Sutton (1988);
Jaakkola et al.
; Dayan & Sejnowski (1994); Tsitsiklis & Van Roy (1997), and its nonasymptotic convergence rate was further characterized recently in Dalal et al.
(2018a)
; Bhandari et al.
(2018); Kotsalis et al.
; Chen et al.
; Kaledin et al.
; Hu & Syed (2019); Srikant & Ying (2019).
The gradient temporal-difference (GTD) was proposed in Sutton et al.
for off-policy evaluation
and was shown to converge asymptotically.
Then, Dalal et al.
(2018b)
; Gupta et al.
; Wang et al.
Xu et al.
; Xu & Liang (2021) provided the ﬁnite-time analysis of GTD and its variants.
Emphatic Temporal Difference (ETD) Learning: The ETD approach was originally proposed in the seminal work Sutton et al.
(2016), which introduced the follow-on trace to overcome the distribution mismatch between the behavior and target policies.
Yu (2015) provided the asymptotic convergence guarantee for ETD.
Hallak et al.
(2016) showed that the variance of the follow-on trace may be unbounded.
They further proposed an ETD method with a variable decay rate to keep the follow-on trace bounded but at the cost of a possibly large bias error.
Our approach is different and keeps both the variance and bias vanishing sublinearly with the number of iterations.
Imani et al.
(2018) developed a new policy gradient theorem, where the emphatic weight is used to correct the distribution shift.
Zhang et al.
(2020b) provided a new variant of ETD, where the emphatic weights are estimated through function approximation.
Van Hasselt et al.
; Jiang et al.
(2021) studied
ETD with deep neural function class.
Comparison to concurrent work: During our preparation of this paper, a concurrent work (Zhang & Whiteson, 2021) was posted on arXiv, and proposed a truncated ETD (which we refer to as
T-ETD for short here), which truncates the update of the follow-on trace to reduce the variance of ETD.
While T-ETD and our PER-ETD share a similar design idea, there are several critical differences between our work from Zhang & Whiteson (2021).
(a) Our PER-ETD features a design of the logarithmical increase of the restart period with the number of iterations, which guarantees the convergence to the original ﬁxed point of ETD, with both the variance and bias errors vanishing sublinearly.
However, T-ETD is guaranteed to converge only to a truncation-length-dependent ﬁxed point, where the convergence is obtained by treating the truncation length as a constant.
A careful review of the convergence proof indicates that the variance term scales exponentially fast with the truncation length, and hence the polynomial efﬁciency is not guaranteed as the truncation length becomes large.
(b) Our convergence rate for PER-ETD does not depend on the cardinality of the state space and has only polynomial dependence on the mismatch parameter of the behavior and target policies.
However, the convergence rate in Zhang & Whiteson (2021) scales with the cardinality of the state space, and increases exponentially fast with the mismatch parameter of behavior and target policies.
(c) This paper further studies PER-ETD( λ ) and the impact of λ on the converge rate and variance and bias errors, whereas Zhang & Whiteson (2021) considers further the application of
T-ETD to the control problem.
B ACKGROUND AND P RELIMINARIES 2. M ARKOV D ECISION P ROCESS
We consider the inﬁnite-horizon Markov decision process (MDP) deﬁned by the ﬁve tuple ( S , A , r, P , γ ) .
Here, S and A denote the state and action spaces respectively, which are both assumed to be ﬁnite sets, r : S ×A → R denotes the reward function, P : S ×A → ∆( S ) denotes the transition kernel, where ∆( S ) denotes the probability simplex over the state space S , and γ ∈ (0 , 1) is the discount factor.
A policy π : S → ∆( A ) of an agent maps from the state space to the probability simplex over the action space A , i.e., π ( a | s ) represents the probability of taking the action a under the state s .
At any time t , given that the system is at the state s t , the agent takes an action a t with the probability π ( a t
| s t ) , and receives a reward r ( s t , a t ) .
The system then takes a transition to the next state s t +1 at time t
+ 1 with the probability P ( s t +1 | s t , a t ) .
For a given policy π , we deﬁne the value function corresponding to an initial state s
= s ∈ S as V π ( s ) =
E [ (cid:80) ∞ t =0 γ t r ( s t , a t ) | s 0 = s, π ] .
Then the value function over the state space can be expressed as a vector V π
= ( V π (1) , V π (2) , . .
, V π ( |S| )) (cid:62) ∈ R |S| .
Here, V π is a deterministic function of the policy π .
We use capitalized characters to be consistent with the literature.
When the state space is large, we approximate the value function V π via a linear function class as
V θ ( s ) =
φ (cid:62) ( s ) θ , where φ ( s ) ∈ R
d denotes the feature vector, and θ ∈ R d denotes the parameter vector to be learned.
We further let Φ =
[ φ (1) , φ (2) , . .
. , φ ( |S| )] (cid:62) denote the feature matrix, and then V θ =
Φ θ .
We assume that the feature matrix Φ has linearly independent columns and each feature vector has bounded (cid:96) 2 -norm, i.e., (cid:107) φ ( s ) (cid:107) 2 ≤ B φ for all s ∈ S .
2. T EMPORAL D IFFERENCE (TD) L EARNING FOR O N - POLICY E VALUATION
In order to evaluate the value function for a given target policy π (i.e., ﬁnd the linear function approximation parameter θ ), the temporal difference (TD) learning can be employed based on a sampling trajectory, which takes the following update rule at each time t :
[Equation]
where η t is the stepsize at time t .
The main idea here is to follow the Bellman operation update to approach its ﬁxed point, and the above sampled version update can be viewed as the so-called semigradient descent update.
If the trajectory is sampled by the target policy π , then the above TD algorithm can be shown to converge to the ﬁxed point solution, where the convergence is guaranteed by the negative deﬁniteness of the so-called key matrix A : = lim t →∞ [Equation]
(cid:2) [Equation] γφ [Equation]
2. E MPHATIC TD (ETD) L EARNING FOR O FF - POLICY E VALUATION
Consider the off-policy setting, where the goal is still to evaluate the value function for a given target policy π , but the agent has access only to trajectories sampled under a behavior policy µ .
Namely, at each time t , the probability of taking an action a t given s t is µ ( a t | s t ) .
Let d µ denote the stationary distribution of the Markov chain induced by the behavior policy µ , i.e., d µ satisﬁes d (cid:62) µ = d (cid:62) µ P π .
We assume that d µ ( s ) > 0 for all states.
The mismatch between the target and behavior policies can be addressed by incorporating the importance sampling factor ρ ( s, a ) :
= π ( a | s ) µ ( a | s ) into eq.
(1) to adjust the TD learning update direction.
However, with such modiﬁcation, the key matrix A may not be negative deﬁnite so that the algorithm is no longer guaranteed to converge.
In order to address this divergence issue, the emphatic temporal difference (ETD) algorithm has been proposed by Sutton et al.
(2016), which takes the following update
[Equation]
[Equation]
In eq.
(2), in addition to the importance sampling factor ρ , a follow-on trace coefﬁcient F t is introduced as a calibration factor, which is updated as [Equation]
[Equation]
with initialization F 0 = 1 .
With such a follow-on trace factor, the key matrix becomes negative deﬁnite, and ETD has been shown to converge asymptotically in Yu (2015) to the ﬁxed point
[Equation]
4) where F = diag( f (1) , f (2) , . . .
, f ( |S| )) and f ( i ) =
d µ ( i ) lim
t →∞ E [ F t | s t
= i ] .
Similarly, the ETD( λ ) algorithm can be further derived, which has the following update
[Equation]
where e t is updated as e t = γλρ ( s t − 1 , a t − 1 )
e t − 1 + M t φ ( s t ) and M t = λ
+(1 − λ ) F t , where M 0 =and e 0 = φ ( s 0 ) .
It has been shown that with a diminishing stepsize (Yu, 2015), ETD( λ ) converges to the ﬁxed point given by θ ∗ λ = (cid:0) Φ (cid:62)
M ( I − γλP π ) − 1 ( I − γP π )Φ θ (cid:1) − 1 Φ (cid:62) M
( I − γλP π ) − 1 r π , where M = diag( m (1) , m (2) , . . .
, m ( |S| )) and m ( i ) =
d µ ( i ) lim
t →∞ E [ M t | s t
= i ] .
2. N OTATIONS
For the simplicity of expression, we adopt the following shorthand notations.
For a ﬁxed integer b , let s τt :=
s t ( b +1)+ τ , a τt := a t ( b +1)+ τ , ρ τt = π ( a τt | s τt ) µ ( a τt | s τt ) and
φ τt = φ ( s τt ) .
We also deﬁne the ﬁltration F t = σ (cid:0) s 0 , a 0 , s 1 , a 1 , . . .
, s t ( b +1)+ b , a t ( b +1)+ b , s t ( b +1)+ b +1 (cid:1) .
Further, let r π ∈ R |S| , where r π ( s ) =
(cid:80) a ∈A r ( s, a ) π ( a | s ) .
Let P π ∈ R |S|×|S| , where P π ( s (cid:48) | s ) =
(cid:80) a ∈A π ( a | s ) P ( s (cid:48) | s, a ) .
For a matrix
M ∈ R N × N , M ( s, · ) denotes its s -th row and M ( · ,s ) denotes its s -th column.
We deﬁne B φ := max s (cid:107) φ ( s ) (cid:107) 2 as the upper bound on the feature vectors, and deﬁne ρ max := max s,a π ( a | s ) µ ( a | s ) as the maximum of the distribution mismatch over all state-action pairs.
P ROPOSED PER-ETD A LGORITHMS
Drawbacks of ETD:
In the original design of ETD (Sutton et al., 2016) described in Section 2.3, the follow-on trace coefﬁcient F t is updated throughout the execution of the algorithm.
As a result, its variance can increase exponentially with the number of iterations, which causes the algorithm to be unstable and diverge, as observed in Hallak et al.
(also see our experiment in Section 5).
In order to overcome the divergence issue of ETD, we propose to PE riodically R estart the follow-on trace update for ETD, which we call as the PER-ETD algorithm (see Algorithm 1).
At iteration t ,
PER-ETD reinitiates the follow-on trace F and update it for b iterations to obtain an estimate F bt , where we call b as the period length.
The emphatic update operator at t is then given by
[Equation]
and PER-ETD updates the value function parameter θ t as θ t +1 = Π Θ (cid:16) θ t − η t (cid:98) T t ( θ t ) (cid:17) , where the projection onto an bounded closed convex set Θ helps to stabilize the algorithm.
It can be shown that lim b →∞ E [ (cid:98) T t ( θ ) |F t − 1 ]
=
T ( θ ) where T ( θ ) :
= (cid:0) Φ (cid:62) F ( I − γP π )Φ (cid:1)
θ − Φ (cid:62)
Fr π .
The ﬁxed point of the operator T ( θ ) is θ ∗ deﬁned in eq.
(4), which is exactly the ﬁxed point of original ETD.
Deﬁnition 1 (Optimal point and (cid:15) -accurate convergence) .
We call the unique ﬁxed point θ ∗ of T ( θ ) as the optimal point (which is the same as the ﬁxed point of ETD).
The algorithm attains an (cid:15) - accurate optimal point if its output θ T satisﬁes (cid:107) θ T − θ ∗ (cid:107) 22 ≤ (cid:15) .
The goal of PER-ETD is to ﬁnd the original optimal point θ ∗ of ETD, which is independent from the period length b .
Our analysis will provide a guidance to choose the period length b in order for
PER-ETD to keep both the variance and bias errors below the target (cid:15) -accuracy with polynomial sample efﬁciency.
τt = ρ τ − 1 t
[Equation] [Equation] t +1 =
Π Θ (cid:0) θ
[Equation]
η t ρ bt (cid:0)
[Equation]
We then extend PER-ETD( 0 ) to PER-ETD( λ ) (see Algorithm 2), which incorporates the eligible trace.
Speciﬁcally, at each iteration t , PER-ETD( λ ) reinitiates the follow-on trace F t and updates it together with M t and the eligible trace e t for b iterations to obtain an estimate e bt .
Then the emphatic update operator at t is given by
[Equation]
and the value function parameter θ t is updated as θ t +1 = Π Θ (cid:16) θ t − η t (cid:98) T λt ( θ t ) (cid:17) .
It can be shown that [Equation] T λ ( θ ) , where T λ ( θ ) = Φ (cid:62) M ( I − γλP π ) − 1 ( I − γP π )Φ θ − Φ (cid:62)
M
( I − γλP π ) − 1 r π , which takes a unique ﬁxed point θ ∗ λ as the original ETD( λ ).
The optimal point and the (cid:15) -accurate
convergence can be deﬁned in the same fashion as in Deﬁnition 1.
It has been shown in Hallak et al.
(2016) that θ ∗ λ is exactly the orthogonal projection of V π to the function space when λ = 1 , and thus is the optimal approximation to the value function.
F INITE -T IME A NALYSIS OF PER-ETD A LGORITHMS 4. T ECHNICAL A SSUMPTIONS
We take the following standard assumptions for analyzing the TD-type algorithms in the literature (Jiang et al., 2021; Zhang & Whiteson, 2021; Yu, 2015).
Assumption 1 (Coverage of behavior policy) .
For all s ∈ S and a ∈ A , the behavior policy µ satisﬁes µ ( a | s ) > 0 as long as π ( a | s ) > 0 .
Assumption 2.
The Markov chain induced by the behavior policy µ is irreducible and recurrent.
The following lemma on the geometric ergodicity has been established.
Lemma 1 (Geometric ergodicity) .
(Levin & Peres, 2017, Thm. 4.9) Suppose Assumption 2 holds.
Then the Markov chain induced by the behavior policy µ has a unique stationary distribution d µ over the state space S .
Moreover, the Markov chain is uniformly geometric ergodic, i.e., there exist constants C M ≥ 0 and 0 < χ < 1 such that for every initial state s 0 ∈ S , the state distribution d µ,t ( s )
=
P ( s t = s | s 0 ) after t transitions satisﬁes (cid:107) d µ,t − d µ (cid:107) 1 ≤ C M χ t .
4. F INITE - TIME A NALYSIS OF PER-ETD( 0 )
In PER-ETD(0), the update of the value function parameter is fully determined by the empirical emphatic operator (cid:98) T t ( θ ) deﬁned in eq.
Thus, we ﬁrst characterize the bias and variance errors of (cid:98) T t ( θ ) , which serve the central role in establishing the convergence rate for PER-ETD(0).
Proposition 1 (Bias bound) .
Suppose Assumptions 1 and 2 hold.
Then we have
[Equation]
where (cid:15) approx
= (cid:107) Φ θ ∗ − V π (cid:107) ∞ is the approximation error of the ﬁxed point, ξ = max { γ, χ } < 1 ,
B φ = max s (cid:107) φ ( s ) (cid:107) 2 , and C b > 0 is a constant whose exact form can be found in the proof.
Proposition 1 characterizes the conditional expectation of the bias error of the empirical emphatic operator (cid:98) T t ( θ ) .
Since ξ = max { γ, χ } < 1 , such a bias error decays exponentially fast as b increases.
Proposition 2 (Variance bound) .
Suppose Assumptions 1 and 2 hold.
Then we have
[Equation]
≤ σ 2 , where σ 2 =
[Equation]
where O ( · ) is with respect to the scaling of b , and ρ max = max s,a π ( a | s ) µ ( a | s ) .
Proposition 2 captures the variance bound of the empirical emphatic operator.
It can be seen that if the distribution mismatch is large (i.e., γ 2 ρ max > 1 ), the variance bound grows exponentially large as b increases, which is consistent with the ﬁnding in Hallak et al.
However, as we show below, as long as b is controlled to grow only logarithmically with the number of iterations, such a variance error will decay sublinearly with the number of iterations.
At the same time, the bias error can also be controlled to decay sublinearly, so that the overall convergence of PER-ETD can be guaranteed with polynomial sample complexity efﬁciency.
Theorem 1.
Suppose Assumptions 1 and 2 hold.
Consider PER-ETD(0) speciﬁed in Algorithm 1.
Let the stepsize η t =
O (cid:0) 1 t (cid:1) and suppose the period length b and the projection set
Θ are properly chosen (see Appendix D.3 for the precise conditions).
Then the output θ T of PER-ETD(0) falls into the following two cases.
(a) If γ 2 ρ max ≤ 1 , then E (cid:2) (cid:107) θ T − θ ∗ (cid:107) 22 (cid:3) ≤ ˜ O (cid:0) 1 T (cid:1) .
(b) If γ 2 ρ max > 1 , then E (cid:2) (cid:107) θ T − θ ∗ (cid:107) 22 (cid:3) ≤ O (cid:0) 1 T a (cid:1) , where a = 1 / (log 1 /ξ ( γ 2 ρ max ) + 1) < 1 .
Thus, PER-ETD( 0 ) attains an (cid:15) -accurate solution with ˜ O (cid:0) 1 (cid:15) (cid:1) samples if γ 2 ρ max ≤ 1 , and with ˜ O (cid:0) 1 (cid:15) 1 /a (cid:1) samples if γ 2 ρ max > 1 .
Theorem 1 captures how the convergence rate depends on the mismatch between the behavior and target policies via the parameter
ρ max (where ρ max ≥ 1 ).
(a) If γ 2 ρ max ≤ 1 , i.e., the mismatch is less than a threshold, then PER-ETD( 0 ) converges at the rate of ˜ O (cid:0) 1 T (cid:1) , which is the same as that of on-policy TD learning (Bhandari et al., 2018).
This result indicates that even under a mild mismatch 1 < ρ max ≤ 1 /γ 2 , PER-ETD achieves the same convergence rate as on-policy TD learning.
(b)
If γ 2 ρ max ≥ 1 , i.e., the mismatch is above the threshold, then PER-ETD( 0 ) converges at a slower rate of ˜ O (cid:0) 1 T a (cid:1) because a < 1 .
Further, as the mismatch parameter ρ max gets larger, the converge becomes slower, because a becomes smaller.
Bias and variance tradeoff: Theorem 1 also indicates that although PER-ETD(0) updates the follow-on trace only over a ﬁnite period length b , it still converges to the optimal ﬁxed point θ ∗ .
This beneﬁts from the proper choice of the period length, which achieves the best bias and variance tradeoff as we explain as follows.
The proof of Theorem 1 shows that the output θ T of PER-ETD(0) satisﬁes the following convergence rate:
[Equation]
If γ 2 ρ max ≤ 1 , then σ 2 in the variance term in eq.
(8) satisﬁes σ 2 ≤ O ( b ) as given in eq.
(7), which increases at most linearly fast with b .
Then we set b
= O (cid:16) log T log(1 /ξ )
(cid:17) so that both the variance and the bias terms in eq.
(8) achieve the same order of O (cid:0) 1 T (cid:1) , which dominates the overall convergence.
If γ 2 ρ max > 1 , then σ 2 in the variance term in eq.
(8) satisﬁes σ 2 =
O (cid:0) ( γ 2 ρ max ) b (cid:1) as given in eq.
(7), Now, we need to set b as b
= O (cid:16)
[Equation]
, where the increase with log T has a smaller coefﬁcient than the previous case, so that both the variance and the bias terms in eq.
(8) achieve the same order of O (cid:0) 1 T a (cid:1) .
Such a choice of b balances the exponentially increasing variance and exponentially decaying bias to achieve the same rate.
4. F INITE - TIME A NALYSIS OF PER-ETD( λ )
In PER-ETD( λ ), the update of the value function parameter is determined by the empirical emphatic operator (cid:98) T λt ( θ ) deﬁned in eq.
Thus, we ﬁrst obtain the bias and variance errors of (cid:98) T λt ( θ ) , which facilitate the analysis of the convergence rate for PER-ETD( λ ).
Proposition 3.
Suppose Assumptions 1 and 2 hold.
Then we have
[Equation]
where (cid:15) approx
= (cid:107) Φ θ ∗ λ − V π (cid:107) ∞ is the approximation error of the ﬁxed point, ξ = max { χ, γ } < 1 ,
B φ = max s (cid:107) φ ( s ) (cid:107) 2 , and C b,λ is a constant given a ﬁxed λ whose exact form can be found in proof.
The above proposition shows that the bias error of the empirical emphatic operator (cid:98) T λt ( θ ) in PERETD( λ ) decays exponentially fast as b increases, because ξ = max { γ, χ } < 1 .
Proposition 4.
Suppose Assumptions 1 and 2 hold.
Then we have E (cid:104) (cid:13)(cid:13)(cid:13)(cid:98) T λt ( θ t ) (cid:13)(cid:13)(cid:13) 2 2 (cid:12)(cid:12)(cid:12) F t − 1 (cid:105) ≤ σ 2 λ , where σ 2 λ
= O (cid:0) ρ bmax (cid:1) .
Compared with Proposition 2 of PER-ETD( 0 ), Proposition 4 indicates that ETD( λ ) has a larger variance, which always increases exponentially with b when ρ max > 1 .
This is due to the fact that the eligible trace e bt carries the historical information and is less stable than φ bt .
Theorem 2.
Suppose Assumptions 1 and 2 hold.
Consider PER-ETD( λ ) speciﬁed in Algorithm 2.
Let the stepsize η t =
O (cid:0) 1 t (cid:1) and suppose the period length b and the projection set
Θ are properly chosen (see Appendix E.3 for the precise conditions).
Then the output θ T of PER-ETD( λ ) satisﬁes E (cid:2) (cid:107) θ T − θ ∗ λ (cid:107) 22 (cid:3) ≤ O (cid:0) 1 T aλ (cid:1) , where a λ = 1 log 1 /ξ ( ρ max )+1 .
PER-ETD( λ ) attains an (cid:15) -accurate solution with ˜ O (cid:16) 1 (cid:15)
1 /aλ (cid:17) samples.
Theorem 2 indicates that PER-ETD( λ ) converges to the optimal ﬁxed point θ ∗ λ determined by the inﬁnite-length update of the follow-on trace.
Furthermore, PER-ETD( λ ) converges at the rate of ˜
O (cid:0) 1 T aλ (cid:1) which is slower than PER-ETD( 0 ) (as a λ < a ) due to the larger variance of PER-ETD( λ ).
Bias and variance tradeoff: We next explain how the period length b achieves the best tradeoff between the bias and variance errors and thus yields polynomial sample efﬁciency.
The proof of
Theorem 2 shows that the output θ T of PER-ETD( λ ) satisﬁes the following convergence rate:
[Equation]
In eq.
(9), σ 2 λ in the variance term takes the form σ 2 λ
= O (cid:0) ρ bmax (cid:1) as given in Proposition 4.
We need to set b
= O (cid:16) [Equation] + [Equation]
[Equation]
) (cid:17) so that both the variance and the bias terms in eq.
(9) achieve the same order of O (cid:0) 1 T aλ (cid:1) .
Thus, such a choice of b balances the exponentially increasing variance and exponentially decaying bias to achieve the same rate.
Impact of the eligible trace (via the parameter λ ) on error bound: It has been shown that with the aid of eligible trace, both TD and ETD achieve smaller error bounds (Sutton & Barto, 2018;
Hallak et al., 2016).
However, this is not always the case for PER-ETD.
Since PER-ETD applies a ﬁnite period length b , the ﬁxed point of PER-ETD( 1 ) is generally not the same as the projection of the ground truth to the function approximation space.
Thus, as λ changes from 0 to 1, depending on the geometrical locations of the ﬁxed points of PER-ETD( λ ) for all λ (determined by chosen features) with respect to the ground truth projection, any value 0 ≤ λ ≤ 1 may achieve the smallest bias error.
We illustrate this further by experiments in Section 5.2.
E XPERIMENTS 5. P ERFORMANCE OF PER-ETD(0)
We consider the BAIRD counter-example.
The details of the MDP setting and behavior and target policies could be found in Appendix A.1.
We adopt a constant learning rate for both PERETD( 0 ) and PER-ETD( λ ) and all experiments take an average over 20 random initialization.
We set the stepsize η = 2 − 9 for all algorithms for fair comparison.
For PER-ETD(0), we adopt onedimensional features Φ 1 = (0 .
35 , 0 . 37) (cid:62) .
The ground truth value function V π
=
(10 , 10 , 10 , 10 , 10 , 10 , 10) (cid:62) and does not lie inside the linear function class.
[Figure]
In Figure 1(a), we compare the performance of of TD, vanilla ETD( 0 ) and PER-ETD( 0 ) with b = 2 , 4 , 8 in terms of the distance between the ground truth and the learned value functions.
It can be observed that our proposed PER-ETD(0) converges close to the ground truth at a properly chosen period length such as b = 4 and b = 8 , whereas TD diverges due to no treatment on off-policy data historically, and ETD ( 0 ) also diverges due to the very large variance.
In Figure 1(b), we plot how the bias and the variance of PER-ETD(0) change as the period length b changes.
Clearly, small b (e.g., b = 4 ) yields a small variance but a large bias.
Then as b increases from 4 to 6 , bias is substantially reduced.
As b continues to increase from 8 to 20 , there is a signiﬁcant increase in variance.
This demonstrates a clear tradeoff between the bias and variance as we capture in our theory.
5. P ERFORMANCE OF PER-ETD( λ )
We next focus on PER-ETD( λ ) under the same experiment setting as in Section 5.1 and study how λ affects the performance.
We conduct our experiments under three features Φ 1 , Φ 2 , and Φ 3 speciﬁed
[Figure] [Figure]
in Appendix A.2. Figure 2 shows how the bias error with respect to the ground truth changes as λ increases under the three chosen features.
As shown in Figure 2 (a), (b), and (c), λ = 0 , 1, and some value between 0 and 1 respectively achieve the smallest error under the corresponding feature.
This is in contrast to the general understanding that λ
= 1 typically achieves the smallest error.
In fact, each case can be explained by the plot in Figure 3 under the same feature.
Each plot in
Figure 3 illustrates how the ﬁxed points of PER-ETD( λ ) are located with respect to the ground truth projection (as V π projection) for b = 4 .
Since the period length b is ﬁnite, the ﬁxed point of PERETD( 1 ) is not located at the same point as the ground truth projection.
The geometric locations of the ﬁxed points of PER-ETD( λ ) for 0 ≤ λ ≤ 1 are determined by chosen features.
The bias error corresponds to the distance between the ﬁxed point of PER-ETD( λ ) and the V π projection.
Then under each feature, the value of λ that attains the smallest error with respect to the V π projection can be readily seen from the plot in Figure 3.
For example, under the feature Φ 3 , Figure 3 (c) suggests that neither λ = 0 nor λ = 1 , but some λ between 0 and 1 achieves the smallest error.
This explains the result in Figure 2 (c) that λ
= 0 .
4 achieves the smallest error among other curves.
As a summary, our experiment suggests that the best λ , under which PER-ETD( λ ) attains the smallest error, depends on the geometry of the problem determined by chosen features.
In practice, if
PER-ETD( λ ) is used as a critic in policy optimization problems, λ may be tuned via the ﬁnal reward achieved by the algorithm.
C ONCLUSION
In this paper, we proposed a novel PER-ETD algorithm, which uses a periodic restart technique to control the variance of follow-on trace update.
Our analysis shows that by selecting the period length properly, both bias and variance of PER-ETD vanishes sublinearly with the number of iterations, leading to the polynomial sample efﬁciency to the desired unique ﬁxed point of ETD, whereas
ETD requires exponential sample complexity.
Our experiments veriﬁed the advantage of PER-ETD against both TD and ETD.
Moreover, our experiments of PER-ETD( λ ) illustrated that under the ﬁnite period length in practice, the best λ that achieves the smallest bias error is feature dependent.
We anticipate that PER-ETD can be applied to various off-policy optimal control algorithms such as actor-critic algorithms and multi-agent reinforcement learning algorithms.
S UPPLEMENTARY M ATERIALS A S PECIFICATION OF E XPERIMENTS IN S ECTIONA. E XPERIMENT SETTINGS
The BAIRD counter-example is illustrated in Figure 4, which has 7 states and 2 actions.
If the ﬁrst action (illustrated as dashed lines) is taken, then the environment transitions from the current state to states 1 to 6 following the uniform distribution and returns a reward 0 ; and if the second action (illustrated as solid lines) is taken, the environment transitions from the current state to state 7 with probability 1 and returns a reward 1 .
We choose the target policy as π (0 | s ) = 0 .
1 and π (1 | s )
= 0 .for
all states; and choose the behavior policy as µ (0 | s )
= 6 / 7 and µ (1 | s )
= 1 / 7 for all states.
Moreover, we specify the discount factor γ = 0 .
A. F EATURES FOR E XPERIMENTS IN S ECTION 5.
In the experiment in Section 5.2, we choose the following features: Φ 1 =(0 .
35 , 0 . 37) (cid:62) ; Φ 2 =((0 .
3425 , 0 . 0171) (cid:62) , (0 .
(cid:62) , (0 . 1354 , 0 .
76) (cid:62) , (0 .
7973) (cid:62) , (0 . 8674 , 0 . 8774) (cid:62) , (0 . 5166 , 0 . 9493)
(cid:62) , (0 . 3094 , 0 . 8535) (cid:62) ) (cid:62) ; Φ 3 =((0 . 5162 , 0 . 9013) (cid:62) , (0 . 5128 , 0 . 5999) (cid:62) , (0 . 289 , 0 . 4649) (cid:62) , (0 . 3399 , 0 . 5334)
(cid:62) , (0 . 315 , 0 . 2278) (cid:62) , (0 . 667 , 0 . 461) (cid:62) , (0 . 3706 , 0 . 1457) (cid:62) )
(cid:62) .
A. C OMPUTATION OF THE FIXED POINT OF PER-ETD( λ )
In this section, we provide the steps to compute the ﬁxed point of PER-ETD( λ ) in Figure 3.
We ﬁrst deﬁne the matrix A and c as follows
[Equation]
It can be shown that, the ﬁxed point of PER-ETD( λ ) algorithm is θ ∗ =
A − 1 c .
We next show how to derive the formulation of the matrix A and vector c .
As we will show later in eqs.
(51), (53) and (55), we have
[Equation]
(11) where ¯ β b := lim t →∞ E [ β b ] and
[Equation]
where D µ,τ = diag( d µ,τ ) , d µ,τ ( s ) =
P ( s τt = s |F t − 1 ) ,
F b = diag( f b ) , and f b is determined iteratively by eq.
(25) as follows
[Equation]
Taking expectation on both sides of eqs.
(12) and (13) with respect to F t − 1 and letting t → ∞ yield
[Equation]
with
[Equation]
with
[Equation]
where ¯ f b := lim t →∞ E [ f b ] and ¯ F b =
diag( ¯ f b ) .
The explicit formulation of ¯ β b can be derived by applying eqs.
(14) and (15) iteratively.
We can then obtain A and c by substituting the obtained formulation of ¯ β b into eq.
(10) and eq.
(11), respectively.
A. R EPLOTTED F IGURES 1 AND 2 WITH V ARIANCE B ARS
In this subsection, we replotted Figures 1 and 2 with variance bars (rather than error bands) in
Figures 5 and 6, respectively.
[Figure] [Figure]
B M ORE E XPERIMENTS
In this section, we conduct further experiments to answer the following two intriguing questions:
• If the distribution mismatch parameter ρ max changes, how will different approaches perform and compare with each other?
• Focusing on our algorithm PER-ETD, how do the choices of behavior policy and target policy affect its convergence?
[Figure] [Figure]
[Figure]
We focus on the MDP environment in Appendix A.1.
In our experiments, the performance of every algorithm is averaged over 20 random initializations and the error band in our plots captures the actual variation of the performance during these experimental runs (which can be viewed as the variance of the algorithms).
In Figure 7, we consider two settings with the distribution mismatch parameter ρ max = 1 . 17 and 5 .
60 , respectively, and compare the performance of three off-policy algorithms: TD, vanilla ETD and our PER-ETD.
More speciﬁcally, we choose the behavior policy as µ (1 | s )
= 6 7 and µ (2 | s )
= 1 for all states, and choose two target polices, whose probabilities to take the second action are 0 .and 0 .
8 , respectively, on all states.
(Then their probabilities to take the ﬁrst action are determined automatically through π (1 | s ) + π (2 | s ) = 1 ).
Therefore, the maximum distribution mismatch ρ max for the these two target policies are 1 . 17 and 5 .
60 , respectively.
Figure 7 (a) shows that under only slightly mismatch (i.e., ρ max = 1 . 17 ), TD suffers from a large convergence error (i.e., the error with respect to the ground truth value function at the point of convergence) and converges slowly.
Vanilla
ETD converges with the fastest rate and achieves a smaller convergence error than TD, but suffers from a relatively large variance.
Our PER-ETD achieves a better tradeoff between the convergence rate and the variance (faster rate than TD, almost the same convergence error as ETD but with smaller variance).
Figure 7 (b) shows that under a large distribution mismatch (i.e., ρ max = 5 . 6 ),
TD does not converge, and vanilla ETD experiences a substantially large variance.
However, our
PER-ETD still convergences fast as long as the period length b is chosen properly, e.g., b = 4 , 6 , 8 .
Further note that PER-ETD has a smaller convergence error as the period length b increases, but the variance gets larger; which are consistent with our theorem.
In Figure 8, we focus on our PER-ETD, and study how different target policies affect the performance.
We choose the same behavior policy as the above experiment, i.e., µ (1 | s ) =
67 and µ (2 | s )
= 17 for all states.
We choose 5 target polices, whose probabilities to take the second action are 0 .
167 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 for all states, respectively.
These different target policies affect the performance via their resulting distribution mismatch ρ max = 1 . 17 , 1 . 40 , 2 . 80 , 4 . 20 , 5 . 60 , respectively.
For both b = 4 and b = 6 , Figure 8 indicates that larger mismatch causes slower convergence rate, larger convergence error and larger variance, which agrees with our theorem.
In Figure 9, we also focus on our PER-ETD, and study how different behavior policies affect the performance.
We pick the target policy to be π (1 | s ) = 0 .
1 and π (2 | s )
= 0 .
9 for all states, anddifferent behavior policies with µ (2 | s ) =
7 and 0 .
8 for all state, respectively.
These different behavior policies affect the performance via their different resulting distribution mismatch ρ max = 1 . 12 , 1 . 29 , 1 . 5 , 2 . 25 , 4 . 5 , respectively.
Figure 9 clearly demonstrates that larger distribution mismatch results in slower convergence rate, larger convergence error and larger variance, which is in the same nature as changing the target policy shown in Figure 8 and is consistent with our theorem.
C S UPPORTING L EMMAS
The following lemma is well-known.
We include it for the convenience of our proof.
Lemma 2.
Consider a transition matrix P ∈ R N × N , where (cid:80) j P
( i,j )
=
for all i and 0 ≤
P ( i,j ) ≤ 1 for all j .
We have for any n ∈ N , (cid:107) P n (cid:107) ∞ = 1 , and (cid:107) ( P (cid:62) ) n (cid:107) 1 = 1 .
Lemma 3.
Consider 0 < p, q < 1 , with p (cid:54) = q .
We have (cid:80) n − 1 k =
p k q n − k ≤ 1 | p − q | ξ n , where ξ = max { p, q } .
Proof of Lemma 3.
If p > q , we have
[Equation]
Without loss of generality, we assume p < q and ξ = q .
We have
[Equation]
Lemma 4.
Consider a matrix P ∈ R N × N where (cid:80) j P
( i,j ) ≤ C for all i and 0 ≤ P ( i,j ) ≤ 1 for all j , and a positive vector x where x ∈ R N and x i ≥
for all i .
We have
[Equation]
Proof of Lemma 4.
We have
[Equation]
Lemma 5.
Consider a diagonal matrix D ∈ R |S|×|S| , where D = diag( d (1) , d (2) , . . .
, d ( |S| )) .
a transition matrix P ∈ R |S|×|S| where (cid:80) j P
i,j = 1 for all i = 1 , 2 , . .
, |S| and P i,j ≥ 0 for all j , and an arbitrary vector x ∈ R |S| .
We have
[Equation]
Proof of Lemma 5.
We have
[Equation]
and
[Equation]
which implies
[Equation]
Taking (cid:96) 2 norm on both sides of the above equality yields
[Equation]
Lemma 6. Consider a diagonal matrix D ∈ R |S|×|S| where D = diag( d (1) , d (2) , . . .
, d ( |S| )) , a transition matrix P ∈ R |S|×|S| where (cid:80) j P
i,j = 1 for all i = 1 , 2 , . .
, |S| and P i,j ≥ 0 for all j , a matrix Q ∈ R |S|×|S| that satisﬁes 0 ≤ Q i,j ≤ CP i,j , where C > 1 is a constant, and an arbitrary vector x ∈ R |S| .
We have
[Equation]
and,
[Equation]
for any m and n ∈ N ≥ 0 .
Proof of Lemma 6.
For any given τ ≥ m ≥ 0 and n ≥ 0 ,
[Equation]
[Equation]
where ( i ) follows from the H ¨o lder’s inequality.
Furthermore, for the term of ( P n ΦΦ (cid:62) ) ( · ,j ) , we have,
[Equation]
where ( i ) follows from Lemma 2.
Moreover, for the i th entry of (ΦΦ (cid:62) ) ( · ,j ) , we have
[Equation]
The above uniform bounds over all i imply that (cid:107) (ΦΦ (cid:62) ) ( · ,j ) (cid:107) ∞ ≤ B 2 φ .
Hence,
[Equation]
Substituting the above inequality back into eq.
(16), we obtain
[Equation]
(17) where ( i ) follows by the condition of Q , Q ( i,j ) ≤ CP ( i,j ) for all i, j .
Finally, we have
[Equation]
where ( i ) follows from eq.
Following steps similar to those in eqs.
(16) to (18), we can obtain
[Equation]
Lemma 7.
The operators T ( θ ) and T λ ( θ ) satisfy the generalized monotone variational inequality.
There exist µ 0 , µ λ > 0 , s.t., (cid:104)T ( θ ) , θ − θ ∗ (cid:105) ≥ µ 0 (cid:107) θ − θ ∗ (cid:107) 22 ,and (cid:10) T λ ( θ ) , θ − θ ∗ (cid:11) ≥ µ λ (cid:107) θ − θ ∗ (cid:107) 22 .
Proof of Lemma 7.
We have
[Equation]
where ( i ) follows from the deﬁnition of the T and θ ∗ .
Recall that F ( I − γP π ) is positive deﬁnite(Sutton et al., 2016; Mahmood et al., 2015) and Φ has linearly independent columns.
For any x ∈ R d with x (cid:54) = 0 , we have Φ x (cid:54) = 0 and
[Equation]
The above inequality shows that Φ (cid:62) F ( I − γP π )Φ is positive deﬁnite and thus, there exists µ 0 >such that µ 0 =
λ min (cid:0) Φ (cid:62) F ( I − γP π )Φ (cid:1) .
Following steps similar to those in eq.
(19) and applying the positive deﬁniteness of M ( I − γλP π ) − 1 ( I − γP π ) (Sutton et al., 2016; Mahmood et al., 2015) yield
[Equation]
Lemma 8.
The operators T ( θ ) and T λ ( θ ) satisfy the Lipschitz condition.
There exist L 0 , L λ > 0 , such that, (cid:107)T ( θ 1 ) − T ( θ 2 ) (cid:107) 2 ≤ L 0 (cid:107) θ 1 − θ 2 (cid:107) 2 , and (cid:13)(cid:13)
T λ ( θ 1 ) − T λ ( θ 2 ) (cid:13)(cid:13) 2 ≤ L λ (cid:107) θ 1 − θ 2 (cid:107) 2 .
Proof of lemma 8.
We have
[Equation]
Let L 0 :=
(cid:13)(cid:13)
Φ (cid:62) F ( I − γP π )Φ (cid:13)(cid:13) 2 , eq.
(20) completes the proof of the ﬁrst inequality in the Lemma.
Let L λ := (cid:107) Φ (cid:62)
M
( I − γλP π ) − 1 ( I − γP π )Φ (cid:107) 2 , the steps similar to those in eq.
(20) ﬁnalizes the proof of the second inequality in the Lemma.
Lemma 9 (Three point lemma) .
Suppose Θ is a closed and bounded subset of R d , and θ ∗ is the solution of the following maximization problem, max θ ∈ Θ η (cid:104) G, θ (cid:105) + 12 (cid:107) θ − θ 0 (cid:107) 22 , where G ∈ R d is a vector.
Then, we have, for any θ ∈ Θ ,
[Equation]
Proof.
The proof can be found in Lan (2020).
D P ROOFS OF P ROPOSITIONS AND T HEOREM FOR PER-ETD(0) D. P ROOF OF P ROPOSITION
First, by the deﬁnition of (cid:98) T t ( θ t ) , we have
[Equation]
( i )
[Equation]
a
[Equation]
ii
[Equation]
s
[Equation]
a ∈A [Equation]
[Equation]
where ( i ) follows from the law of total probability, ( ii ) follows from rewriting
P (cid:0) s bt = s, a bt = a, s b +1 t =
s (cid:48) (cid:12)(cid:12) F t − 1 (cid:1) , and ( iii ) follows from the facts that F bt only depends on ( s 0 , a 0 , s 1 , a 1 , . . .
, s t ( b +1)+ b − 1 , a t ( b +1)+ b − 1 ) and the chain is Markov.
Recall the deﬁnition of T ( θ t ) , we have
[Equation]
Equations (21) and (22) together imply the following,
[Equation] [Equation]
where in ( i )
We deﬁne f b ( s ) : = P (cid:0) s bt = s (cid:12)(cid:12) F t − 1 (cid:1) E (cid:2) F bt (cid:12)(cid:12) F t − 1 , s bt = s (cid:3) .
Taking (cid:96) 2 norm on both sides of the above equality yields
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
We next proceed to bound (cid:107) f − f b (cid:107) 1 .
Consider f b ( s ) , we have f b (
[Equation]
) =
[Equation]
( ii ) =
P
[Equation]
· 
[Equation]
˜ [Equation] where ( i ) follows from the law of total probability and ( ii ) follows from the Bayes rule and the facts that F b − 1 t only depends on the chain elements ( s 0 t , a 0 t , s 1 t , . .
, s b − 1 t , a b − 1 t ) and the Markov property.
Deﬁne d µ,b ( s ) =
P (cid:0) s bt =
s (cid:12) (cid:12) F t − 1 (cid:1) , the above equality can be rewritten as
[Equation]
[Equation]
Since eq.
(24) holds for all s ∈ S , we have
[Equation]
[Equation]
Note that for f we have the following holds (Sutton et al., 2016; Zhang et al., 2020a)
[Equation] [Equation]
Equations (25) and (26) imply
[Equation]
Applying the above equality recursively yields
[Equation]
[Table]
(27) where ( i ) follows from Lemma 2, ( ii ) follows from Lemma 1 , and ( iii ) follows from Lemma 3 and deﬁning ξ :
= max { χ, γ } .
To bound the term (cid:107) ( I − γP π )Φ θ t − r π (cid:107) ∞ , we proceed as following
[Equation]
where ( i ) follows from the fact
V π =
( I − γP π ) − 1 r π and ( ii ) follows from the facts that (cid:107) I − γP
[Equation]
=
[Equation]
approx :
=
[Equation]
Φ [Equation] ∗ − V π (cid:107) ∞ , and
[Equation]
Substituting eqs.
(27) and (28) into eq.
(23) yields
[Equation]
t ( θ t ) (cid:12)(cid:12)(cid:12) F t − 1 (cid:105)(cid:13)(cid:13)(cid:13) 2 ≤ (cid:0) B 2 φ (cid:107) θ t − θ ∗ (cid:107) 2 + B φ (cid:15) approx (cid:1) (1 + γ ) (cid:16) C M ξ b | χ − γ | + γ b (1 + (cid:107) f (cid:107) 1 ) (cid:17) ≤ C b ( B φ (cid:107) θ t − θ ∗ (cid:107) 2 + (cid:15) approx )
ξ b , where ξ = max { χ, γ } and C b =
B φ (1 + γ ) (cid:16) C M | χ − γ | + (1 + (cid:107) f (cid:107) 1 ) (cid:17) .
D. P ROOF OF P ROPOSITION
According to the deﬁnition of (cid:98) T t ( θ t ) , we have E
[Equation] (cid:21) = E (cid:104)(cid:0) [Equation]
bt F bt (cid:1) [Equation]
(cid:0) θ (cid:62) t φ bt − r bt − γθ (cid:62) t φ b +1 t (cid:1) [Equation] (cid:107) φ bt (cid:107) {+
+} (cid:12)(cid:12)(cid:12) F t − 1 (cid:105) ( i ) = (cid:88) s ∈S (cid:88) a ∈A (cid:88) s (cid:48) ∈S
P (cid:0) s bt = s, a bt = a, s b
[Equation]
1 t = s (cid:48) (cid:12) (cid:12) F t − [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
where ( i ) follows from the law of total probability and ( ii ) follows from the fact that F bt is independent from previous states and actions given s bt .
Note that (cid:107) φ ( s ) (cid:107) 2 ≤ B φ for all s ∈ S , r ( s, a ) ≤ r max for all ( s, a ) ∈ S × A , and (cid:107) θ t (cid:107) 2 ≤ B θ for all t due to projection.
We have
[Equation]
We also have (cid:88)
[Equation]
Substituting the above two inequalities into eq.
(29) yields
[Equation]
Deﬁne
[Equation]
We have the following equations hold for r b ( s ) :
[Equation]
[Equation]
For the second term in the RHS of eq.
(32), we have d µ,b ( s ) E
[Equation]
ρ b − 1 t
[Equation]
where ( i ) follows from the law of total probability, ( ii ) follows from the Bayes rule, ( iii ) follows from Markov property and ( iv ) follow from the deﬁnition of f b which is given above eq.
[Equation]
For the third term on the RHS of eq.
[Equation]
we have
[Equation]
where ( i ) follows from the law of total probability, ( ii ) follows from the Bayes’ rule, and in ( iii ) we deﬁne P µ,π ∈ R |S|×|S| where ( P µ,π ) s, ˜ s
= (cid:80) ˜ a ∈A π 2 (˜ a | ˜ s ) µ (˜ a | ˜ s ) P ( s | ˜ s, ˜ a ) for each ( s, ˜ s ) ∈ S × S .
Substituting eqs.
(33) and (34) into eq.
(32) yields
[Equation]
We also have the following inequality holds
[Equation]
where ( i ) follows from 1 (cid:62) P (cid:62)
π =
( P π 1 ) (cid:62) = 1 (cid:62) , and ( ii ) follows from the facts that r b − 1 (cid:23) 0 and
[Equation]
Recursively applying eq.
(35) yields
[Equation]
where ( i ) follows from the fact that 1 (cid:62) r 0 = 1 .
Recall that f b = d µ,b + γP (cid:62) π f b − 1 and f = d µ + γP (cid:62) π f .
We have
[Equation]
where ( i ) follows from the facts that d µ,τ and d µ are both probability distributions and 1 (cid:62) d µ,τ = 1 (cid:62) d µ = 1 and 1 (cid:62) P (cid:62) π = 1 (cid:62) and ( ii ) follows from recursively applying ( i ) .
Thus, we have
[Equation]
Substituting eq.
(37) into eq.
(36) yields
[Equation]
Under different conditions of ρ max
[Equation]
the term 1 (cid:62) r b is upper bounded differently as following:
[Equation]
Substituting the above inequalities into eq.
[Equation]
we can upper-bound the term
[Equation]
under different conditions accordingly:
[Equation]
where we specify C
[Equation] [Equation]
= C
[Equation]
where we specify C σ,
[Equation]
(cid:19) :
= C σ, 3 . where we specify C σ, 3 = ρ max (cid:16) 4(1 + γ 2 ) B
φ B 2 θ + 2 r 2 max (cid:17) B 2 φ (cid:16) 3+2 γ (cid:107) f (cid:107) 1 1 − γ 2 ρ max + 1 (cid:17) .
To summarize, the variance term (cid:13)(cid:13)(cid:13)(cid:98) T t ( θ t ) (cid:13)(cid:13)(cid:13) 2 2 can be bounded as following
[Equation]
where
[Equation]
D. P ROOF OF T HEOREM
Theorem 3 (Formal Statement of Theorem 1) .
Suppose Assumptions 1 and 2 hold.
Consider
PER-ETD(0) speciﬁed in Algorithm 1.
Let the stepsize η t = 2 µ
( t + t 0 ) , where t 0 = 8 L 20 µ 20 , µis deﬁned in Lemma 7, and L 0 is deﬁned in Lemma 8 in Appendix C. Let the projection set Θ = (cid:8) θ ∈ R
d : (cid:107) θ (cid:107) 2 ≤ B θ (cid:9) , where B θ = [Equation] 2 r max [Equation]
1 − γ ) µ(which implies θ ∗ ∈ Θ ).
Then the convergence guarantee falls into the following two cases depending on the value of ρ max . [Equation]
a )
[Equation]
γ
[Equation]
ρ
[Equation] ≤ 1 , let b [Equation] max
[Equation]
, where C b is a constant deﬁned in the proof of Proposition 1 in Appendix D.1, B φ := max s ∈S (cid:107) φ ( s ) (cid:107) 2 , and ξ := max { γ, χ } .
Then the output θ T satisﬁes
[Equation]
b )
[Equation]
γ [Equation]
ρ max > 1 , let
[Equation]
, where C b is a constant whose deﬁnition could be found in the proof of Proposition 1 in Appendix D.1, B φ := max
[Equation]
(cid:107) 2 , and ξ := max { γ, χ } .
Then the output θ T satisﬁes
[Equation]
where a = 1 log 1 /ξ ( γ 2 ρ max )+1 < 1 .
Thus, PER-ETD( 0 ) attains an (cid:15) -accurate solution with ˜ O (cid:0) 1 (cid:15) (cid:1) samples if γ 2 ρ max ≤ 1 , and with ˜ O (cid:0) 1 (cid:15) 1 /a (cid:1) samples if γ 2 ρ max > 1 .
Proof.
Note the θ update speciﬁed in Algorithm 1 is the closed form solution of the following maximization problem.
[Equation]
Applying Lemma 9 with θ ∗ =
θ t +1 , η
= η t , G = (cid:98) T t ( θ t ) , and θ
= θ t yields, for any θ ∈ Θ ,
[Equation] [Equation]
Proceed with the ﬁrst term in the above inequality as follows
[Equation]
where ( i ) follows from the Cauchy-Schwartz inequality and Lemma
[Equation]
Substituting the above inequality into eq.
(41) yields
[Equation]
[Table]
[Equation]
Applying Young’s inequality to η
[Equation]
yields
[Equation]
and applying Young’s inequality to η t L
[Equation]
yields
[Equation]
Substituting the above two inequalities into eq.
(42) yields
[Equation]
Taking expectation conditioned on F t − 1 on the both sides of the above inequality
[Equation]
we obtain
[Equation]
Letting θ = θ ∗ and applying Lemma 7 to eq.
(43) yields
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
44) where ( i ) follows from Propositions 1 and
[Table]
and the facts that ( x + y ) 2 ≤ 2 x 2 + 2 y 2 and [Table] − [Table]
− T
[Table]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
t
[Equation]
[Equation]
Taking expectation on both sides of the above inequality yields
[Equation]
Recall that we set t 0 =
L 20 µ 20 .
Let α t = ( t + t 0 + 1)( t + t 0 + 2) .
Multiplying 2 α t on both sides of the above inequality and telescoping from t = 0 , 1 , 2 , . .
, T − 1 yields
T − 1 (cid:88) t =
[Equation]
[Table]
[Equation]
Recall the setting of η t , we have
[Equation]
where ( i ) follows from the fact that 1 − 4 η t L 20 µ≥ 1 − 8 L 20 µ 2 0 t 0 ≥ 0 .
Multiplying α t on both sides of the above inequality yields
[Equation]
Under appropriate value of b , we have C b B φ ξ b ≤ µ 0 5 .
Which implies that
[Equation]
Multiplying α t +1 on both sides of the above inequality yields
[Equation]
Equations (46) and (47) together imply that
[Equation]
= ( t + t 0 + 2)( t + t 0 + 3) ( t + t 0 )( t + t 0 + 1) > 0 .
The above inequality shows that the (cid:107)
[Equation]
−
[Equation]
22 , t = 1 , . . .
, T − 1 , terms on both sides of eq.
(45) can be canceled, which indicates the following
[Equation]
Note that (cid:80) T − {+1 t =
 α t η+}
[Equation]
≤
[Equation]
=
[Equation]
≤
[Equation]
20 , 1 + 2 µ
[Equation]
η T − 1 − 2 η
[Equation]
− 1 L 20 ≥ 1 , and
[Equation]
Dividing ( T + t 0 )
( T + t 0 + 1)
[Equation]
1 + 2 µ 0 η
[Equation]
−
[Equation]
− 2 η 2 T − 1 L 20 (cid:1) on both sides of eq.
(48) yields
[Equation]
Based on different conditions of σ 2 , we pick different b and the convergence rate is as follows.
( a ).
γ 2 ρ max ≤ 1 ,
Proposition 2 show that
σ 2 ≤ O ( b )
We specify
[Equation]
Equation (49) yields,
[Equation]
( b ).
γ 2 ρ max > 1 ,
Proposition 2 show that σ 2 =
O (cid:0) ( γ 2 ρ max ) b
(cid:1) .
We specify
[Equation]
Equation (49) yields
[Equation]
E P ROOFS OF P ROPOSITIONS AND T HEOREM FOR PER-ETD( λ ) E. P ROOF OF P ROPOSITION
Deﬁne the matrix
A t :=
ρ bt e b t ( φ bt − γφ b +1 t ) and c t := r bt ρ b t e bt .
We have (cid:98) T λt ( θ t ) =
A t θ t − c t .
Recall that θ t is F t − 1 -measurable.
We have
[Equation]
t
[Equation]
To bound the bias error term (cid:13)(cid:13)(cid:13)
[Equation] F [Equation] [Equation]
) (cid:13)(cid:13)(cid:13) 2 , we ﬁrst take conditional expectations on A t and c t , respectively, as following
E [
[Equation]
51) where ( i ) follows from the law of total probability and ( ii ) follows from the Markov property and the fact that e bt only depends on ( s 0 t , a 0 t , s 1 t , . . .
, s bt ) .
Deﬁne β τ ( s ) =
P ( s τt = s |F t − 1 ) E (cid:2) e bt (cid:12)(cid:12) F t − 1 , s τt = s (cid:3) .
We have β b ( s ) =
P (cid:0) s bt =
s (cid:12)(cid:12) F t − 1 (cid:1) E (cid:2) e bt (cid:12)(cid:12) s bt = s, F t − 1 (cid:3) (
[Equation]
(cid:1) · [Equation]
[Equation]
γλρ b − 1 t e b
[Equation]
ρ b − 1 t γF
[Equation]
t )
[Equation]
[Equation]
s
[Equation]
1 t =
[Equation]
, a
[Equation]
t
[Equation]
( ii ) =
P (cid:0) s
[Equation]
s (cid:12)(cid:12) F t
[Equation]
1 (cid:1)
[Equation]
( s ) +
[Equation]
∈A
[Equation]
t
[Equation] s ) [Equation] [Equation]
γF
[Equation]
t
[Equation]
( iii ) [Equation] ( λd [Equation]
[Equation]
( s )
[Equation]
(1 − λ ) f b ( s )) · φ ( s ) + γλ ( P (cid:62) π β b − 1 ) s , (52) where ( i ) follows from the law of total probability, ( ii ) follows from the Bayes rule and the Markov property, and ( iii ) follows from the following deﬁnitions:
[Equation]
= s,
[Equation]
t
[Equation]
, f b = d µ,b + γP (cid:62) π f b − 1 , and β τ ( s ) =
P ( s τt = s |F t − 1 ) E (cid:2) e bt (cid:12)(cid:12) F t − 1 , s τt = s (cid:3) .
Deﬁne the matrix β τ ∈ R d ×|S| , where β τ = ( β τ (1) , β τ (2) , . .
, β τ ( |S| )) .
Then, eq.
(52) implies that
[Equation]
(53) where D µ,b :=
diag( d µ,b (1) , d µ,b (2) , . . .
, d µ,b ( |S| )) and F b = diag( f b ) .
Recursively applying the above equality yields
[Equation]
Taking expectation of c t conditioned on F t − 1 , we have
[Equation]
Substituting eqs.
(51) and (55) into eq.
(50) yields
[Equation]
Recall the deﬁnition of T λ ( θ ) .
We have
[Equation]
[Equation] We then proceed to bound the term Φ (cid:62) M [Equation] I
[Equation]
where ( i ) follows from the fact that ( I − γP π ) −
= (cid:80) ∞
τ =
γ τ P τπ , and ( ii ) follows from eq.
Substituting the above equality into eq.
(56) and taking (cid:96) 2 norm on the both sides yield
[Equation]
[Table]
[Equation]
where ( i ) follows from Lemma 5 and eq.
(28), ( ii ) follows from Lemma 1 and eq.
(27), in ( iii ) we deﬁne
[Equation]
λ
[Equation]
, and ( iv ) follows from Lemma
[Equation]
E. P ROOF OF P ROPOSITION
According to the deﬁnition of (cid:98) T λt
[Equation]
we have
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
57) where ( i ) follows from the law of total probability, ( ii ) follows from the Markov property and the fact that e bt only depends on ( s 0 t , a 0 t , . . .
, s bt ) , and ( iii ) follows from eq.
(30) and the fact (cid:80)
[Equation]
Deﬁne ∆ b ( s ) =
P (cid:0) s bt =
s |F t − 1 (cid:1) E (cid:2) ( e bt ) (cid:62) e bt (cid:12)(cid:12) s bt = s, F t − 1 (cid:3) .
We then proceed to bound the term ∆ b ( s ) .
We have ∆
[Equation]
[Equation]
˜ a ∈A
P
[Equation]
t [Equation] [Equation] t
[Equation] [Equation] ) (cid:62) e b − 1 t + (1 − λ ) 2 γ 2 ( ρ b − 1 t F b − 1 t ) 2 φ ( s ) (cid:62) φ ( s ) + φ ( s ) (cid:62) φ ( s ) +2 γ 2 λ (1 − λ )( ρ b − 1 t ) 2 F b − 1 t φ ( s ) (cid:62) e b − 1 t + 2 γλρ b − 1 t φ ( s ) (cid:62) e b − 1 t
+2(1 − λ ) γρ b − 1 t
F b − 1 t φ ( s ) (cid:62) φ ( s ) (cid:12)(cid:12) s b − 1 t =
˜ s, a b − 1 t = ˜
a, s bt = s, F t − 1 (cid:3) =
P (cid:0) s bt =
s |F t − 1 (cid:1) (cid:107) φ ( s ) (cid:107)+ φ (cid:62) ( s ) (cid:88)
˜ s ∈S
P π ( s | ˜ s ) P (cid:0) s b − 1 t =
˜ s (cid:12)(cid:12) F t − 1 (cid:1) E (cid:2) 2 γλe
b − 1 t + 2 γ
(1 − λ ) F b − 1 t φ ( s ) (cid:12)(cid:12) s b − 1 t =
˜ s, F t − 1 (cid:3) + λ 2 γ 2 (cid:88) ˜ s ∈S
P µ,π ( s | ˜ s ) P (cid:0) s b − 1 t =
˜ s (cid:12)(cid:12) F t − 1 (cid:1) E (cid:2) ( e b − 1 t ) (cid:62) e b − 1 t (cid:12)(cid:12) s b − 1 t =
˜ s, F t − 1 (cid:3) + 2 γ 2 λ (1 − λ ) (cid:88)
˜ s ∈S
P µ,π ( s | ˜ s ) P (cid:0) s b − 1 t = ˜ s (cid:12)(cid:12) F t − 1 (cid:1) E (cid:2) F b − 1 t φ ( s ) (cid:62)
e b − 1 t (cid:12) (cid:12) s b − 1 t = ˜ s, F t − 1 (cid:3) + (cid:107) φ ( s ) (cid:107) 22 γ 2 (1 − λ ) 2 (cid:88)
˜ s ∈S
P µ,π ( s | ˜ s ) P (cid:0) s b − 1 t = ˜ s (cid:12)(cid:12) F t − 1 (cid:1) E (cid:2) ( F b − 1 t ) 2 (cid:12)(cid:12) s b − 1 t = ˜ s, F t − 1 (cid:3) [Equation]
[Equation]
( s )
[Equation]
[Equation]
(cid:62) ( s )(
[Equation]
P π ) ( · ,s )
[Equation]
2 γ (1 − λ ) B
[Equation]
( P (cid:62)
[Equation]
f
[Equation]
1 ) s + λ
[Equation]
γ
[Equation]
( P (cid:62) µ,π ∆ b − 1 ) s
+ γ 2 (1 − λ )
2 B 2 φ ( P (cid:62) µ,π r b − 1 ) s + 2 γ 2 λ (1 − λ ) (cid:88)
˜ s ∈S
P µ,
[Equation]
( s | ˜ s ) P (cid:0) s b − 1 t = ˜ s (cid:12)(cid:12) F t
[Equation]
1 (cid:1) E (cid:2) F b − 1 t φ ( s ) (cid:62)
e b − 1 t (cid:12)(cid:12) s b − 1 t = ˜ s, F t
[Equation]
(cid:3)
[Equation]
) =
[Equation]
[Equation]
( s )
[Equation]
λ
[Equation]
γ
[Equation]
( P (cid:62) µ,π ∆
[Equation]
1 ) s
[Equation]
2 γ (1 − λ ) B 2 φ ( P (cid:62) π
[Equation] 58) where ( i ) follows from the law of total probability, ( ii ) follows from the Bayes rule, ( iii ) follows the update rule of e bt and in ( iv ) we deﬁne
[Equation]
Summing eq.
(58) over S yields
[Equation]
where ( i ) follows from Lemma 4 with P = P µ,π .
Recursively applying the above inequality
[Equation]
we have
[Equation]
Substituting eq.
(54) into trace (Φ β τ P π ) with b and τ replaced by τ and m respectively yields trace (Φ
[Equation]
(60) where ( i ) follows from eq.
[Equation]
( ii ) follows from the facts that trace( AB )
= trace( BA ) and δ
=
Φ (cid:62) D µ, 0 , ( iii ) follows from Lemma 6 with P = P π [Equation] and ( iv ) follows from eq.
Next, we proceed to bound the term trace (cid:0) Φ (cid:62) δ τ − 1 P µ,π (cid:1) .
δ b − 1 (˜ s ) ( i ) =
P (cid:0) s b − 1 t = ˜ s (cid:12)(cid:12) F t − 1 (cid:1) E (cid:2) F b − 1 t (cid:0) γλρ b − 2 t
e b − 2 t + ( λ + (1 − λ ) F b − 1 t ) φ
(˜ s ) (cid:1)(cid:12)(cid:12) s b − 1 t = ˜ s, F t − 1 (cid:3) ( ii )
= (1 − λ ) P (cid:0) s b − 1 t = ˜ s (cid:12)(cid:12) F t − 1 (cid:1) E (cid:2) ( F b − 1 t ) 2 (cid:12)(cid:12)
s b − 1 t = ˜ s, F t − 1 (cid:3) φ (˜ s ) + λ P (cid:0)
s b − 1 t = ˜ s (cid:12)(cid:12) F t − 1 (cid:1) E (cid:2) F b − 1 t (cid:12)(cid:12) s b − 1 t =
˜ s, F t − 1 (cid:3) φ (˜ s ) + γλ P (cid:0) s b − 1 t = ˜ s (cid:12) (cid:12) F t − 1 (cid:1) E (cid:2) ρ b − 2 t e b − 2 t (cid:12) (cid:12) s b − 1 t = ˜ s, F t − 1 (cid:3) φ (˜ s )
+ γ 2 λ P (cid:0) s b − 1 t =
˜ s (cid:12)(cid:12) F t − 1 (cid:1) E (cid:2) ( ρ b − 2 t ) 2 F b − 2 t e b − 2 t (cid:12)(cid:12)
[Equation]
iii
[Equation]
[Table]
[Equation]
where ( i ) follows from the update of e b − 1 t , ( ii ) follows from the update rule of F b − 1 t , and ( iii ) follows from the law of total probability.
The above equality implies that
[Equation]
Recursively applying the above equality yields
[Equation]
Note that the above inequality holds for any ﬁxed b > = 2 .
As a result, by changing of notation, for all τ ≥
[Equation]
we have
[Equation]
[Equation]
Substituting eq.
(61) into trace (Φ δ τ P µ,π ) , we have trace (
[Equation]
= trace
[Equation]
[Equation]
where ( i ) follows from the fact
that trace( AB ) = trace( BA ) and
trace( A + B ) =
trace( A ) + trace( B ) .
Applying Lemma 6 with Q = P
µ,π , C = ρ max , P = P π and D =
(1 − λ )diag( r m ) , we have trace (cid:0)
[Equation]
P π
[Equation]
Applying Lemma 6 with Q = P
µ,π , C = ρ max , P = P π and D = λ diag( f m ) , we have trace (cid:0) ( P π,µ )
τ +1 − m ΦΦ (cid:62) λ diag( f m ) (cid:1) ≤ λρ τ +1 − m max
[Equation]
[Equation]
where ( i ) follows from eq.
For the term trace (cid:0) ( [Equation] ,µ ) τ +1 − m Φ β m − 1 P π (cid:1) , we have trace (cid:0) ( P π,µ )
τ +1 − m Φ β m − 1 P π (cid:1)
[Equation]
where ( i ) follows from eq.
(54) and the facts that trace( A + B ) =
trace( A ) + trace( B ) and trace( AB ) =
trace( BA ) , ( ii ) follows from Lemma 6 with Q = P
µ,π , P = P π , and D = F m − 1 − l and D π,m − 1 − l respectively, and ( iii ) follow from the eq.
Recall δ
=
Φ (cid:62) D µ, 0 .
Applying Lemma 6 with Q = P µ,π and D = D µ, 0 yields
[Equation]
[Equation]
Substituting eqs.
(63) to (66) into eq.
(62), we have trace (Φ δ
[Equation]
[Equation]
Substituting eqs.
(60) and (67) into eq.
(59), we have
[Equation]
where we let C β :
=
B 2 φ 1 − γλ (1 + (1 − λ ) (cid:107) f (cid:107) 1 ) .
Under different conditions of ρ max , the term 1 (cid:62) ∆ b and E (cid:20)(cid:13)(cid:13)(cid:13)(cid:98) T λt ( θ t ) (cid:13)(cid:13)(cid:13) 2 2 (cid:12)(cid:12)(cid:12)(cid:12)
F t −(cid:21) can be upper bounded differently as following: ( a ).
γ 2 ρ max < 1 , substituting eq.
(40) into eq.
(68) yields
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
where the last two terms of the above inequality are of the order O (cid:0) ρ bmax (cid:1) .
Therefore, we have 1 (cid:62)
∆ b ≤ Cρ bmax for some C > 0 .
Substituting eq.
(69) into eq.
(57) yields
[Equation]
where C σ,λ, 1 > 0 is a constant and is determined by eq.
( b ).
γ 2 ρ max = 1 , substituting eq.
(39) into eq.
(68) yields
[Equation]
where the last term of the above inequality is of the order O (cid:0) ρ bmax (cid:1) .
Therefore, we have 1 (cid:62) ∆ b ≤
Cρ bmax for some C > 0 .
Substituting eq.
(70) into eq.
(57) yields
[Equation]
where C σ,λ, 2 > 0 is a constant and is determined by eq.
( c ).
γ 2 ρ max > 1 , substituting eq.
(38) into eq.
(68) yields 1 (cid:62)
[Equation]
[Table]
[Equation]
where the last term of the above inequality is of the order O (cid:0) ρ bmax (cid:1) .
Therefore, we have 1 (cid:62) ∆ b ≤
Cρ bmax for some C > 0 .
Substituting eq.
(71) into eq.
(57) yields
[Equation]
where C σ,λ, 3 > 0 is a constant and is determined by eq.
To summarize, the variance term E (cid:20)(cid:13)(cid:13)(cid:13)(cid:98) T t
[Equation]
{+θ t ) (cid:13)(cid:13)(cid:13) 
+}
[Equation]
F t −(cid:21) can be bounded by σ 2 λ
= O ( ρ bmax ) .
E. P ROOF OF T HEOREM
Theorem 4 (Formal Statement of Theorem 2) .
Suppose Assumptions 1 and 2 hold.
Consider PERETD( λ ) speciﬁed in Algorithm 2.
Let the stepsize η t = 2 µ λ ( t + t λ ) , t λ = 8 L 2 λ µ 2 λ , where µ λ is deﬁned in
Lemma 7 and L λ is deﬁned in Lemma 8 in Appendix C. Further let
[Equation]
where C b,λ is a constant deﬁned in the proof of Proposition 3 in Appendix E.1, B φ := max s ∈S (cid:107) φ ( s ) (cid:107) 2 , and ξ := max { γ, χ } .
Let the projection set Θ = (cid:8) θ ∈ R
d : (cid:107) θ (cid:107) 2 ≤ B θ (cid:9) , where
B θ = (cid:107) Φ (cid:62) (cid:107) 2 r max [Equation] [Equation] ) µ λ (which implies θ ∗ λ ∈ Θ ).
Then the output θ T of PER-ETD( λ ) satisﬁes
[Equation]
where a λ = 1 log 1 /ξ ( ρ max )+1 .
Further, PER-ETD( λ ) attains an (cid:15) -accurate solution with ˜ O (cid:16) 1 (cid:15)
/aλ (cid:17) samples.
Proof.
The proof follows the same steps as Theorem 1 by replacing the terms (cid:98) T t , T , t 0 , L 0 , µ 0 , C b , σ 2 and θ ∗ with (cid:98) T λt , T λ , t λ , L λ , µ λ , C b,λ , σ 2 λ and θ ∗ λ respectively.
Speciﬁcally, Propositions 3 andare applied to bound the bias and variance over the steps similarly to eq.
We then have the convergence as follows.
[Equation]
We further specify b
[Equation]
Then Equation (72) yields
[Equation]