G REEDY -GQ WITH V ARIANCE R EDUCTION : F INITETIME A NALYSIS AND I MPROVED C OMPLEXITY
Anonymous authors
Paper under double-blind review
A BSTRACT
Greedy-GQ is a value-based reinforcement learning (RL) algorithm for optimal control.
Recently, the ﬁnite-time analysis of Greedy-GQ has been developed under linear function approximation and Markovian sampling, and the algorithm is shown to achieve an (cid:15)
-stationary point with a sample complexity in the order of
O ( (cid:15) − 3 ) .
Such a high sample complexity is due to the large variance induced by the Markovian samples
In this paper
, we propose a variance reduced Greedy
GQ
VR
Greedy
GQ algorithm for off-policy optimal control.
In particular, the algorithm applies the SVRG based variance reduction scheme to reduce the stochas tic variance of the two time
-scale updates.
We study the ﬁnite-time convergence of
VR
Greedy
-GQ under linear function approximation and Markovian sampling and show that the algorithm achieves a much smaller bias and variance error than the original
Greedy-GQ.
In particular
, we prove that VR-Greedy-
GQ achieves an improved sample complexity that is in the order of O (
(cid:15) − 2 ) .
We further compare the performance of VR-Greedy-GQ with that of Greedy
GQ in various
RL exper- iments to corroborate our theoretical ﬁndings.
I NTRODUCTION
In reinforcement learning (RL), an agent interacts with a stochastic environment following a certain policy and receives some reward, and it aims to learn an optimal policy that yields the maximum accumulated reward Sutton & Barto (2018).
In particular, many RL algorithms have been developed to learn the optimal control policy, and they have been widely applied to various practical applications such as ﬁnance, robotics, computer games and recommendation systems Mnih et al.
Silver et al.
; Kober et al.
Conventional RL algorithms such as Q-learning Watkins & Dayan (1992) and SARSA Rummery &
Niranjan (1994) have been well studied and their convergence is guaranteed in the tabular setting.
However, it is known that these algorithms may diverge in the popular off-policy setting under linear function approximation Baird (1995); Gordon (1996).
To address this issue, the two time-scale
Greedy-GQ algorithm was developed in Maei et al.
(2010) for learning the optimal policy.
This algorithm extends the efﬁcient gradient temporal difference (GTD) algorithms for policy evaluation
Sutton et al. (2009b) to policy optimization.
In particular, the asymptotic convergence of GreedyGQ to a stationary point has been established in Maei et al.
More recently, Wang & Zou (2020) studied the ﬁnite-time convergence of Greedy-GQ under linear function approximation and
Markovian sampling, and it is shown that the algorithm achieves an (cid:15) -stationary point of the objective function with a sample complexity in the order of O ( (cid:15) − 3 ) .
Such an undesirable high sample complexity is caused by the large variance induced by the Markovian samples queried from the dynamic environment.
Therefore, we want to ask the following question.
• Q1:
Can we develop a variance reduction scheme for the two time-scale Greedy-GQ algorithm?
In fact, in the existing literature, many recent work proposed to apply the variance reduction techniques developed in the stochastic optimization literature to reduce the variance of various TD learning algorithms for policy evaluation, e.g., Du et al.
; Peng et al.
; Korda & La (2015);
Xu et al.
Some other work applied variance reduction techniques to Q-learning algorithms, e.g., Wainwright (2019); Jia et al.
Hence, it is much desired to develop a variance-reduced
Greedy-GQ algorithm for optimal control.
In particular, as many of the existing variance-reduced
RL algorithms have been shown to achieve an improved sample complexity under variance reduction, it is natural to ask the following fundamental question
• Q2: Can variance-reduced Greedy-GQ achieve an improved sample complexity under Markovian sampling?
In this paper, we provide afﬁrmative answers to these fundamental questions.
Speciﬁcally, we develop a two time-scale variance reduction scheme for the Greedy-GQ algorithm by leveraging the SVRG scheme Johnson & Zhang (2013).
Moreover, under linear function approximation and
Markovian sampling, we prove that the proposed variance-reduced Greedy-GQ algorithm achieves an (cid:15)
-stationary point with an improved sample complexity O ( (cid:15) − 2 ) .
We summarize our technical contributions as follows.
1. O UR C ONTRIBUTIONS
We develop a variance-reduced Greedy-GQ (VR-Greedy-GQ) algorithm for optimal control in reinforcement learning.
Speciﬁcally, the algorithm leverages the SVRG variance reduction scheme
Johnson & Zhang (2013) to construct variance-reduced stochastic updates for updating the parameters in both time-scales.
We study the ﬁnite-time convergence of VR-Greedy-GQ under linear function approximation and
Markovian sampling in the off-policy setting.
Speciﬁcally, we show that VR-Greedy-GQ achieves an (cid:15) -stationary point of the objective function J (i.e., (cid:107)∇ J ( θ ) (cid:107) 2 ≤ (cid:15) ) with a sample complexity in the order of O ( (cid:15) − 2 ) .
Such a complexity result improves that of the original Greedy-GQ by a signiﬁcant factor of O ( (cid:15) − 1 ) Wang & Zou (2020).
In particular, our analysis shows that the bias error caused by the Markovian sampling and the variance error of the stochastic updates are in the order of O ( M − 1 ) , O ( η θ M − 1 ) , respectively, where η θ is the learning rate and M corresponds to the batch size of the SVRG reference batch update.
This shows that the proposed variance reduction scheme can signiﬁcantly reduce the bias and variance errors of the original Greedy-GQ update (by a factor of M ) and lead to an improved overall sample complexity.
The analysis logic of VR-Greedy-GQ partly follows that of the conventional SVRG, but requires substantial new technical developments.
Speciﬁcally, we must address the following challenges.
First, VR-Greedy-GQ involves two time-scale variance-reduced updates that are correlated with each other.
Such an extension of the SVRG scheme to the two time-scale updates is novel and requires new technical developments.
Speciﬁcally, we need to develop tight variance bounds for the two time-scale updates under Markovian sampling.
Second, unlike the convex objective functions of the conventional GTD type of algorithms, the objective function of VR-Greedy-GQ is generally non-convex due to the non-stationary target policy.
Hence, we need to develop new techniques to characterize the per-iteration optimization progress towards a stationary point under nonconvexity.
In particular, to analyze the two time-scale variance reduction updates of the algorithm, we introduce a ‘ﬁne-tuned’ Lyapunov function of the form R mt =
J ( θ ( m ) t ) + c t (cid:107) θ ( m ) t − (cid:101) θ ( m ) (cid:107) 2 , where the parameter c t is ﬁne-tuned to cancel other additional quadratic terms (cid:107) θ ( m ) t − (cid:101) θ ( m ) (cid:107) 2 that are implicitly involved in the tracking error terms.
The design of this special Lyapunov function is critical to establish the formal convergence of the algorithm.
With these technical developments, we are able to establish an improved ﬁnite-time convergence rate and sample complexity for VR-Greedy-GQ.
1. R ELATED W ORK
Q-learning and SARSA with function approximation.
The asymptotic convergence of Q-learning and SARSA under linear function approximation were established in Melo et al.
; Perkins &
Precup (2003), and their ﬁnite-time analysis were developed in Zou et al.
; Chen et al.
However, these algorithms may diverge in off-policy training Baird (1995).
Also, recent works focused on the Markovian setting.
Various analysis techniques have been developed to analyze the ﬁnite-time convergence of TD/Q-learning under Markovian samples.
Speciﬁcally, Wang et al.
(2020) developed a multi-step Lyapunov analysis for addressing the biasedness of the stochastic approximation in Q-learning.
Srikant & Ying (2019) developed a drift analysis to the linear stochas- tic approximation problem.
Besides the linear function approximation, the ﬁnite-time analysis of
Q-learning under neural network function approximation is developed in Xu & Gu (2019).
GTD algorithms.
The GTD2 and TDC algorithms were developed for off-policy TD learning.
Their asymptotic convergence was proved in Sutton et al.
(2009a;b); Yu (2017), and their ﬁnite-time analysis were developed recently in Dalal et al.
; Wang et al.
; Liu et al.
; Gupta et al.
; Xu et al.
The Greedy-GQ algorithm is an extension of these algorithms to optimal control and involves nonlinear updates.
RL with variance reduction: Variance reduction techniques have been applied to various RL algorithms.
In TD learning, Du et al.
(2017) reformulate the MSPBE problem as a convex-concave saddle-point optimization problem and applied SVRG Johnson & Zhang (2013) and SAGA Defazio et al.
(2014) to primal-dual batch gradient algorithm.
In Korda & La (2015), the variance-reduced
TD algorithm was introduced for solving the MSPBE problem, and later Xu et al.
(2020) provided a correct non-asymptotic analysis for this algorithm over Markovian samples.
Recently, some other works applied the SVRG , SARAH Nguyen et al.
(2017) and SPIDER Fang et al.
(2018) variance reduction techniques to develop variance-reduced Q-learning algorithms, e.g., Wainwright (2019);
Jia et al.
In these works, TD or TDC algorithms are in the form of linear stochastic approximation, and Q-learning has only a single time-scale update.
As a comparison, our VR-Greedy-GQ takes nonlinear two time-scale updates to optimization a nonconvex MSPBE.
P RELIMINARIES : P OLICY O PTIMIZATION AND G REEDY -GQ
In this section, we review some preliminaries of reinforcement learning and recap the Greedy-GQ algorithm under linear function approximation.
2. P OLICY O PTIMIZATION IN R EINFORCEMENT L EARNING
In reinforcement learning, an agent takes actions to interact with the environment via a Markov
Decision Process (MDP).
Speciﬁcally, an MDP is speciﬁed by the tuple ( S , A , P , r, γ ) , where S and
A respectively correspond to the state and action spaces that include ﬁnite elements, r : S×A×S → [0 , + ∞ ) denotes a reward function and γ ∈ (0 , 1) is the associated reward discount factor.
At any time t , assume that the agent is in the state s t ∈ S and takes a certain action a t ∈
A following a stationary policy π , i.e., a t ∼ π ( ·| s t ) .
Then, at the subsequent time t + 1 , the current state of the agent transfers to a new state s t +1 according to the transition kernel P ( ·| s t , a t ) .
At the same time, the agent receives a reward r t = r
( s t , a t , s t +1 ) from the environment for this action-state transition.
To evaluate the quality of a given policy π , we often use the action-state value function
Q π : S ×
A → R that accumulates the discounted rewards as follows:
[Equation]
where V π ( s ) is the state value function deﬁned as V π ( s )
=
E
[Equation]
(cid:80) ∞ t =
γ t r t | s
= s (cid:105) .
In particular, deﬁne the Bellman operator T π such that T π Q ( s, a )
=
E s (cid:48) ,a (cid:48)
[ r ( s, a, s (cid:48) ) + γQ ( s (cid:48) , a (cid:48) )] for any
Q ( s, a ) , where a (cid:48) ∼ π ( ·| s (cid:48) ) .
Then, Q π ( s, a ) is a ﬁxed point of T π , i.e.,
T π Q π ( s, a ) =
Q π ( s
[Equation]
a ) , ∀ s, a.
The goal of policy optimization is to learn the optimal policy π ∗ that maximizes the expected total reward E [ (cid:80) ∞ t =
γ t r t | s
=
s ] for any initial state s ∈ S , and this is equivalent to learn the optimal value function Q ∗ ( s, a )
=
sup π Q π ( s, a ) , ∀ s, a .
In particular, Q ∗ is a ﬁxed point of the Bellman operator T that is deﬁned as TQ ( s, a ) =
E s (cid:48) ∼ P ( ·| s,a )
[ r ( s, a, s (cid:48) )
+ γ max b ∈A Q ( s (cid:48) , b )] .
2. G REEDY -GQ WITH L INEAR F UNCTION A PPROXIMATION
The Greedy-GQ algorithm is inspired by the ﬁxed point characterization in eq.
(1), and in the tabular setting it aims to minimize the Bellman error (cid:107) T π Q π − Q π (cid:107) 2 µ s,a .
Here, (cid:107) · (cid:107) 2 µ s,a is induced by the state-action stationary distribution µ s,a (induced by the behavior policy π b ), and is deﬁned as (cid:107) Q (cid:107)
µ s,a = E ( s,a ) ∼ µ s,a [ Q ( s, a ) 2 ] .
In practice, the state and action spaces may include a large number of elements that makes tabular approach infeasible.
To address this issue, function approximation technique is widely applied.
In this paper, we consider approximating the state-action value function Q ( s, a ) by a linear function.
Speciﬁcally, consider a set of basis functions { φ ( i ) :
S ×
A → R , i = 1 , 2 , . . .
, d } , each of which maps a given state-action pair to a certain value.
Deﬁne φ s,a = [ φ (1) ( s, a ); ... ; φ ( d ) ( s, a )] as the feature vector for ( s, a ) .
Then, under linear function approximation, the value function Q ( s, a ) is approximated by Q θ ( s, a )
=
φ (cid:62) s,a θ, where θ ∈ R d denotes the parameter of the linear approximation.
Consequently, Greedy-GQ aims to ﬁnd the optimal θ ∗ that minimizes the following mean squared projected Bellman error (MSPBE).
[Equation]
(2) where µ s,a is the stationary distribution induced by the behavior policy π b , Π is a projection operator that maps an action-value function Q to the space Q spanned by the feature vectors, i.e., Π Q = arg min U ∈Q (cid:107) U − Q (cid:107) µ s,a .
Moreover, the policy π θ is parameterized by θ .
In this paper, we consider the class of Lipschitz and smooth policies (see Assumption 4.2).
Next, we introduce the Greedy-GQ algorithm.
[Equation]
θ ) =
r
[Equation]
s, a, s
[Equation]
+ γV
[Equation]
( θ ) − φ (cid:62) s,a θ and denote (cid:98) φ s ( θ ) = ∇ V s ( θ ) .
Then, the gradient of the objective function J ( θ ) in eq.
(2) is expressed as
[Equation]
where ω ∗ ( θ ) =
E [ φ s,
[Equation]
(cid:62) s,a ] − 1 E
[ δ s,a,s (cid:48)
( θ ) φ s,a ] .
To address the double-sampling issue when estimating the product of expectations involved in E [ (cid:98) φ s (cid:48)
( θ ) φ (cid:62) s,a ] ω ∗ ( θ ) , Sutton et al.
(2009a) applies a weight doubling trick and constructs the following two time-scale update rule for the Greedy-GQ algorithm: for every t = 0 , 1 , 2 , ... , sample ( s t , a t , r t , s t +1 ) using the behavior policy π b and do (Greedy-GQ):
[Equation]
where η θ , η ω > 0 are the learning rates and we denote δ t +1 ( θ )
= δ s t ,a t ,s t +1 ( θ ) , φ t :=
φ s t ,a t , (cid:98) φ t +1 ( θ t ) :
= (cid:98) φ s t +1 ( θ t ) for simplicity.
To elaborate, the ﬁrst two steps correspond to the two timescale updates for updating the value function Q θ , whereas the last step is a policy improvement operation that exploits the updated value function to improve the target policy, e.g., greedy, (cid:15) -greedy, softmax and mellowmax Asadi & Littman (2017).
The above Greedy-GQ algorithm uses a single Markovian sample to perform the two time-scale updates in each iteration.
Such a stochastic Markovian sampling often induces a large variance that signiﬁcantly slows down the overall convergence.
This motivates us to develop variance reduction schemes for the two time-scale Greedy-GQ in the next section.
G REEDY -GQ WITH V ARIANCE R EDUCTION
In this section, we propose a variance-reduced Greedy-GQ (VR-Greedy-GQ) algorithm under
Markovian sampling by leveraging the SVRG variance reduction scheme Johnson & Zhang (2013).
To simplify notations, we deﬁne the stochastic updates regarding a sample x t = ( s t , a t , r t , s t +1 ) used in the Greedy-GQ as follows:
[Equation]
Next, consider a single MDP trajectory { x t } t ≥ 0 obtained by the behavior policy π b .
In particular, we divide the entire trajectory into multiple batches of samples {B m } m ≥ 1 so that B m = { x ( m − 1) M , ..., x mM − 1 } , and our proposed VR-Greedy-GQ uses one batch of samples in every epoch.
To elaborate, in the m -th epoch, we ﬁrst initialize this epoch with a pair of reference points θ ( m )
= (cid:101) θ ( m ) , ω ( m )=
(cid:101) ω ( m ) , where (cid:101) θ ( m ) , (cid:101) ω
( m ) are set to be the output points θ ( m − 1) M , ω ( m − 1)
M of the previous epoch, respectively.
Then, we compute a pair of reference batch updates using the reference points and the batch of samples as follows
[Equation]
In the t -th iteration of the m -th epoch, we ﬁrst query a random sample x ξ mt from the batch
B m uniformly with replacement (i.e., sample ξ mt from { ( m − 1) M, ..., mM − 1 } uniformly).
Then, we use this sample to compute the stochastic updates G x ξmt , H x ξmt at both of the points ( θ ( m ) t , ω ( m ) t ) , ( (cid:101) θ ( m ) , (cid:101) ω ( m ) ) .
After that, we use these stochastic updates and the reference batch updates to construct the variance-reduced updates in Algorithm 1 via the SVRG scheme, where for simplicity we denote the stochastic updates G x ξmt , H x ξmt respectively as G ( m ) t , H ( m ) t .
In particular, we project the two time-scale updates onto the Euclidean ball with radius R to stabilize the algorithm updates, and we assume that R is large enough to include at least one stationary point of J .
Lastly, we further update the policy via the policy improvement operation P .
Algorithm
Input learning rates
, batch
Initialize: (cid:101) θ (1) =
θ 0 , (cid:101) ω (1) =
ω 0 , π (cid:101) θ (1) ← P ( φ (cid:62) (cid:101) θ (1) ) .
for m = 1 , 2 , . . .
do θ ( m )=
(cid:101) θ ( m ) , ω ( m )=
(cid:101) ω ( m ) .
Compute (cid:101) G ( m ) , (cid:101) H ( m ) according to eq.
for t = 0 , 1 , . . .
, M − 1 do
Query a sample from B m with replacement.
θ ( m ) t +1 =
Π R (cid:104) θ ( m ) t − η θ (cid:0) G ( m ) t
( θ ( m ) t , ω ( m ) t ) − G ( m ) t ( (cid:101) θ ( m ) , (cid:101) ω ( m ) )
+ (cid:101) G ( m ) (cid:1)(cid:105) .
ω ( m ) t +1 =
Π R (cid:104) ω
( m ) t − η ω (cid:0) H ( m ) t
( θ ( m ) t , ω ( m ) t ) − H ( m ) t (˜ θ ( m ) , ˜ ω ( m ) )
+ (cid:101) H
( m ) (cid:1)(cid:105) .
Policy improvement : π θ ( m ) t +1 ← P ( φ (cid:62) θ ( m ) t ) .
end
Set (cid:101) θ ( m +1)
= θ ( m ) M , (cid:101) ω ( m +1)
= ω ( m ) M .
end
Output: parameter θ chosen among { θ ( m ) t } t,m uniformly at random.
The above VR-Greedy-GQ algorithm has several advantages and uniqueness.
First, it takes incremental updates that use a single Markovian sample per-iteration.
This makes the algorithm sample efﬁcient.
Second, VR-Greedy-GQ applies variance reduction to both of the two time-scale updates.
As we show later in the analysis, such a two time-scale variance reduction scheme signiﬁcantly reduces the variance error of both of the stochastic updates.
We want to further clarify the incrementalism and online property of VR-Greedy-GQ.
Our VRGreedy-GQ is based on the online-SVRG and can be viewed as an incremental algorithm with regard to the batches of samples used in the outer-loops, i.e., in every outer-loop the algorithm samples a new batch of samples and use them to perform variance reduction in the corresponding inner-loops.
Therefore, VR-Greedy-GQ can be viewed as an online batch-incremental algorithm.
In general, there is a trade-off between incrementalism and variance reduction for SVRG-type algorithms: a larger batch size in the outer-loops enhances the effect of variance reduction, while a smaller batch size makes the algorithm more incremental.
F INITE -T IME A NALYSIS OF VR-G REEDY -GQ
In this section, we analyze the ﬁnite-time convergence rate of VR-Greedy-GQ.
We adopt the following standard technical assumptions from Wang & Zou (2020); Xu et al.
Assumption 4.1 (Feature boundedness) .
The feature vectors are uniformly bounded, i.e., (cid:107) φ s,a (cid:107) ≤ 1 for all ( s, a ) ∈ S × A .
Assumption 4.2 (Policy smoothness) .
The mapping θ
(cid:55)→ π θ is k 1 -Lipschitz and k 2 -smooth.
We note that the above class of smooth policies covers a variety of practical policies, including softmax and mellowmax policies Asadi & Littman (2017);
Wang & Zou (2020).
Assumption 4.3 (Problem solvability) .
The matrix C := E [ φ s,a φ (cid:62) s,a ] is non-singular.
Assumption 4.4 (Geometric uniform ergodicity) .
There exists Λ > 0 and ρ ∈ (0 , 1) such that
[Equation]
for any t > 0 , where d TV is the total-variation distance.
Based on the above assumptions, we obtain the following ﬁnite-time convergence rate result.
Theorem 4.5 (Finite-time convergence) .
Let Assumptions 4.1– 4.4 hold and consider the VRGreedy-GQ algorithm.
Choose learning rates η θ , η ω and the batch size M that satisfy the conditions speciﬁed in eqs.
(15) to (19) .
Then, after T epochs, the output of the algorithm satisﬁes
[Equation]
where ξ, ζ are random indexes that are sampled from { 0 , ..., M − 1 } and { 1 , ..., T } uniformly at random, respectively.
Theorem 4.5 shows that VR-Greedy-GQ asymptotically converges to a neighborhood of a stationary point at a sublinear rate.
In particular, the size of the neighborhood is in the order of O ( M − 1 + η 4 θ η − 4 ω + η 2 ω ) , which can be driven arbitrarily close to zero by choosing a large batch size and sufﬁciently small learning rates that satisfy the two time-scale condition η θ /η ω
→ 0 .
Moreover, the convergence error terms implicitly include a bias error O ( 1 M ) caused by the Markovian sampling and a variance error
O ( η θ M ) caused by the stochastic updates, both of which are substantially reduced by the large batch size M .
This shows that the SVRG scheme can effectively reduce the bias and variance error of the two time-scale stochastic updates.
By further optimizing the choice of hyper-parameters, we obtain the following characterization of sample complexity of VR-Greedy-GQ.
Corollary 4.6 (Sample complexity) .
Under the same conditions as those of Theorem 4.5, choose learning rates so that η θ = O ( 1 M ) , η ω
=
O ( η 2 / 3 θ ) , , and set T,
M = O ( (cid:15) − 1 ) ).
Then, the required sample complexity for achieving E (cid:107)∇ J ( θ ( ζ ) ξ ) (cid:107) 2 ≤ (cid:15) is in the order of TM
=
O ( (cid:15) − 2 ) .
Such a complexity result is orderwise lower than the complexity O ( (cid:15) − 3 ) of the original Greedy-GQ
Wang & Zou (2020).
Therefore, this demonstrates the advantage of applying variance reduction to the two time-scale updates of VR-Greedy-GQ.
We also note that for online stochastic non-convex optimization, the sample complexity of the SVRG algorithm is in the order of O ( (cid:15) − 5 / 3 )
Li & Li (2018), which is slightly better than our result.
This is reasonable as the SVRG in stochastic optimization is unbiased due to the i.i.d.
sampling.
In comparison, VR-Greedy-GQ works on a single
MDP trajectory that induces Markovian noise, and the two-timescale updates of the algorithm also introduces additional tracking error.
S KETCH OF THE T ECHNICAL P ROOF
In this section, we provide an outline of the technical proof of the main Theorem 4.5 and highlight the main technical contributions.
The details of the proof can be found in the appendix.
We note that our proof logic partly follows the that of the conventional SVRG, i.e., exploiting the objective function smoothness and introducing a Lyapunov function.
However, our analysis requires substantial new developments to address the challenges of off-policy control, two time-scale updates of VR-Greedy-GQ and correlation of Markovian samples.
The key step of the proof is to develop a proper Lyapunov function that drives the parameter to a stationary point along the iterations.
In addition, we also need to develop tight bounds for the bias error, variance error and tracking error.
We elaborate the key steps of the proof below.
Step 1: We ﬁrst deﬁne the following Lyapunov function with certain c t > 0 to be determined later.
[Equation]
To explain the motivation, note that unlike the analysis of variance-reduced TD learning Xu et al.
(2020) where the term (cid:107) θ ( m ) t − (cid:101) θ ( m ) (cid:107) 2 can be decomposed into (cid:107) θ ( m ) t − θ ∗ (cid:107) 2 + (cid:107) (cid:101) θ ( m ) − θ ∗ (cid:107) 2 to get the desired upper bound, here we do not have θ ∗ due to the non-convexity of J ( θ ) .
Hence, we need to properly merge this term into the Lyapunov function R mt .
By leveraging the smoothness of
J ( θ ) and the algorithm update rule, we obtain the following bound for the Lyapunov function R mt (see eq.
(10) in the appendix for the details).
[Equation]
In particular, the error term 1 M is due to the noise of Markovian sampling and the variance of the stochastic updates, and the last two terms correspond to tracking errors.
Step 2:
To telescope the Lyapunov function over t based on eq.
(6), one may want to deﬁne
J ( θ ( m ) t
[Equation]
[Equation]
−
[Equation]
[Equation]
= R mt by choosing a proper c t of R mt .
However, note that eq.
(6) involves the last two tracking error terms, which also implicitly depend on E (cid:107) θ ( m ) t − (cid:101) θ ( m ) (cid:107)as we show later in the Step 3.
Therefore, we need to carefully deﬁne the c t of R mt
so that after applying the tracking error bounds developed in the Step 3, the right hand side of eq.
(6) can yield an R mt without involving the term E (cid:107) θ ( m ) t − (cid:101) θ ( m ) (cid:107) 2 .
It turns out that we need to deﬁne c t via the recursion speciﬁed in eq.
(11) in the appendix.
We rigorously show that the sequence { c t } t is uniformly bounded by a small constant (cid:98) c
= 18 .
Then, plugging these bounds into eq.
(6) and summing over one epoch, we obtain the following bound (see eq.
(13) in the appendix for the details).
[Equation]
Step 3: We derive bounds for the tracking error terms (cid:80) M − 1 t =
E (cid:107) ω ( m ) t − ω ∗ ( θ ( m ) t ) (cid:107) 2 and E (cid:107) (cid:101) ω ( m ) − ω ∗ ( (cid:101) θ ( m ) )
(cid:107) 2 in the above bound in Lemma D.7 and Lemma D.8.
Step 4:
Lastly, by substituting the tracking error bounds obtained in Step 3 into the bound obtained in Step 2, the resulting bound does not involve the term (cid:80) M − 1 t =0 E (cid:107) θ ( m ) t − (cid:101) θ ( m ) (cid:107) 2 .
Then, summing this bound over the epochs m = 1 , ..., T , we obtain the desired ﬁnite-time convergence rate result.
E XPERIMENTS
In this section, we conduct two reinforcement learning experiments, namely, Garnet problem
Archibald et al.
and
Frozen Lake game Brockman et al.
(2016), to test the performance of
VR-Greedy-GQ in the off-policy setting, and compare it with Greedy-GQ in the Markovian setting.
6. G ARNET P ROBLEM
For the Garnet problem, we refer to Appendix F for the details of the problem setup.
We set the default learning rates as η θ = 0 .
02 and η ω = 0 .
01 for both VR-Greedy-GQ and Greedy-GQ algorithm.
For VR-Greedy-GQ, we set the default batch size as M = 3000 .
In Figure 1 (left), we plot the minimum gradient norm v.s. the number of pseudo stochastic gradient computations for both algorithms using 40 Garnet MDP trajectories, and each trajectory contains 10 k samples.
The upper and lower envelopes of the curves correspond to the 95% and 5% percentiles of the 40 curves, respectively.
It can be seen that VR-Greedy-GQ outperforms Greedy-GQ and achieves a signiﬁcantly smaller asymptotic gradient norm.
In Figure 1 (middle), we track the estimated variance of the stochastic update for both algorithms along the iterations.
Speciﬁcally, we query 500 Monte Carlo samples per iteration to estimate the pseudo gradient variance E (cid:107) G ( m ) t
[Equation]
[Equation]
It can be seen from the ﬁgure that the stochastic updates of VR-Greedy-GQ induce a much smaller variance than Greedy-GQ.
This demonstrates the effectiveness of the two time-scale variance reduction scheme of VR-Greedy-GQ.
We further study the asymptotic convergence error of VR-Greedy-GQ under different batch sizes M .
We use the default learning rate setting that is mentioned previously and run 100 k iterations for one
Garnet trajectories.
We use the mean of the convergence error of the last 10 k iterations as an estimate of the asymptotic convergence error (the training curves are already saturated and ﬂattened).
Figure 1 (right) shows the asymptotic convergence error of VR-Greedy-GQ under different batch sizes M .
It can be seen that VR-Greedy-GQ achieves a smaller asymptotic convergence error with a larger batch size, which matches our theoretical result.
In Figure 2 (Left), we plot the MSPBE J ( θ ) v.s. number of gradient computations for both GreedyGQ and VR-Greedy-GQ, where one can see that VR-Greedy-GQ achieves a much smaller MSPBE than Greedy-GQ.
In Figure 2 (Middle), we plot the estimated expected maximum reward (see Appendix F for details)
v.s. number of gradient computations for Greedy-GQ, VR-Greedy-GQ and actor-critic, where for actor-critic we set learning rate η θ = 0 . 02 for the actor update and η ω = 0 .for the critic update. One can see that VR-Greedy-GQ achieves a higher reward than the other two algorithms, demonstrating the high quality of its learned policy.
In addition, we also plot the estimated expected maximum reward v.s. number of iterations for Greedy-GQ, VR-Greedy-GQ and policy gradient in Figure 2 (Right).
For the policy gradient, we apply the standard off-policy policy gradient algorithm.
For each update, we sample 30 independent trajectories with a ﬁxed lengthto estimate the expected discounted return.
The learning rate of policy gradient is set as η θ .
We note that each iteration of policy gradient consumes 1800 samples and hence it is very sample inefﬁcient.
Hence we set the x -axis to be number of iterations for a clear presentation (otherwise it becomes a ﬂat curve).
One can see that VR-Greedy-GQ achieves a much higher expected reward than both Greedy-GQ and policy gradient.
6. F ROZEN L AKE G AME
We further test these algorithms in solving the more complex frozen lake game.
We generate a
Gaussian feature matrix with dimension 8 to linearly approximate the value function and we aim to evaluate a target policy based on a behavior policy.
The target policy is generated via the uniform distribution on [0 , 1] with proper normalization and the behavior policy is the uniform policy.
We set the learning rates as η θ = 0 .
2 and η ω = 0 .
1 for both algorithms and set the batch size as M =for the VR-Greedy-GQ.
We run 200 k iterations for each of the 10 trajectories.
Figure 3 shows the comparison between VR-Greedy-GQ and Greedy-GQ, and one can make consistent observations with those made in the Garnet experiment.
Speciﬁcally, Figure 3 (left) shows that VR-Greedy-GQ achieves a much more stationary policy than Greedy-GQ.
Figure 3 (middle) shows that the stochastic updates of VR-Greedy-GQ induce a much smaller variance than those of
Greedy-GQ.
Moreover, Figure 3 (right) veriﬁes our theoretical result that VR-Greedy achieves a smaller asymptotic convergence error with a larger batch size.
We further plot the MSPBE v.s. number of gradient computations for both Greedy-GQ and VRGreedy-GQ in Figure 4 (Left), where one can see that VR-Greedy-GQ outperforms Greedy-GQ.
In Figure 2 (Middle), we plot the estimated expected maximum reward v.s. number of gradient computations for Greedy-GQ, VR-Greedy-GQ and actor-critic, where for actor-critic we set learning rate η θ = 0 . 2 for the actor update and η ω = 0 . 1 for the critic update. It can be seen that VRGreedy-GQ achieves a higher reward than the other two algorithms.
In Figure 2 (Right), we plot the estimated expected maximum reward v.s. number of iterations for Greedy-GQ, VR-Greedy-GQ and policy gradient.
For policy gradient, we use the same parameter settings as before.
One can see that VR-Greedy-GQ achieves a much higher expected reward than both Greedy-GQ and policy gradient. Figure
C ONCLUSION
In this paper, we develop a variance-reduced two time-scale Greedy-GQ algorithm for optimal control by leveraging the SVRG variance reduction scheme.
Under linear function approximation and
Markovian sampling, we establish the sublinear ﬁnite-time convergence rate of the algorithm to a stationary point and prove an improved sample complexity bound over that of the original GreedyGQ.
The RL experiments well demonstrated the effectiveness of the proposed two time-scale variance reduction scheme.
Our algorithm design may inspire new developments of variance reduction for two time-scale RL algorithms.
In the future, we will explore Greedy-GQ with other nonconvex variance reduction schemes to possibly further improve the sample complexity.