C LOSING THE GAP BETWEEN SVRG AND TD-SVRG WITH G RADIENT S PLITTING
Anonymous authors
Paper under double-blind review
A BSTRACT
Temporal difference (TD) learning is a simple algorithm for policy evaluation in reinforcement learning.
The performance of TD learning is affected by high variance and it can be naturally enhanced with variance reduction techniques, such as the Stochastic Variance Reduced Gradient (SVRG) method.
Recently, multiple works have sought to fuse TD learning with SVRG to obtain a policy evaluation method with a geometric rate of convergence.
However, the resulting convergence rate is significantly weaker than what is achieved by SVRG in the setting of convex optimization.
In this work we utilize a recent interpretation of TD-learning as the splitting of the gradient of an appropriately chosen function, thus simplifying the algorithm and fusing TD with SVRG.
We prove a geometric convergence bound with predetermined learning rate of 1/8, that is identical to the convergence bound available for SVRG in the convex setting.
I NTRODUCTION
Reinforcement learning (RL) is a learning paradigm which addresses a class of problems in sequential decision making environments.
Policy evaluation is one of those problems, which consists of determining expected reward agent will achieve if it chooses actions according to stationary policy.
Temporal Difference learning (TD learning, Sutton (1988)) is popular algorithm, since it is simple and might be performed online on single samples or small mini-batches.
TD learning method uses
Bellman equation to bootstrap the estimation process update the value function from each incoming sample or minibatch.
As all methods in RL, TD learning from the “curse of dimensionality” when number of states is large.
To address this issue, in practice linear or nonlinear feature approximation of state values is often used.
Despite its simple formulation, theoretical analysis of approximate TD learning is subtle.
There are few important milestones in this process, one of which is a work of Tsitsiklis & Van Roy (1997), in which asymptotic convergence guarantees were established.
More recently advances were made by Bhandari et al.
(2018), Srikant & Ying (2019) and Liu & Olshevsky (2020).
In particular, the last paper shows that TD learning might be viewed as an example of gradient splitting, a process analogous to gradient descent.
TD-learning has inherent variance problem, which is that the variance of the update does not go to zero as the method converges.
This problem is also present in a class of convex optimization problems where target function is represented as a sum of functions and SGD-type methods are applied
Robbins & Monro (1951).
Such methods proceed incrementally by sampling a single function, or a minibatch of functions, to use for stochastic gradient evaluations.
A few variance reduction techniques were developed to address this problem and make convergence faster, including SAG
Schmidt et al.
(2013), SVRG Johnson & Zhang (2013) and SAGA Defazio et al.
These methods are collectively known as variance-reduced gradient methods .
The distinguishing feature of these methods is that they converge geometrically.
The first attempt to adapt variance reduction to TD learning with online sampling was done by
Korda & La (2015).
Their results were discussed by Dalal et al.
(2018) and Narayanan & Szepesv´ari (2017);
Xu et al.
(2020) performed reanalysis of their results and shown geometric convergence for
Variance Reduction Temporal Difference learning (VRTD) algorithm for both Markovian and i.i.d sampling.
The work of Du et al.
(2017) directly apply SVRG and SAGA to a version of policy evaluations by transforming it into an equivalent convex-concave saddle-point problem.
Since their algorithm uses two sets of parameters, in this paper we call it Primal Dual SVRG or PD SVRG.
All these results obtained geometric convergence of the algorithms, improving the sub-geometric convergence of the standard TD methods.
However, the convergence rates obtained in these papers are significantly worse than the convergence of SVRG in convex setting.
In particular, the resulting convergence times for policy evaluations scaled with the square of the condition number, as opposed to SVRG which retains the linear scaling with the condition number of SGD.
Quadratic scaling makes practical application of theoretically obtain values almost unfeasible, since number of computations becomes very large even for simple problems.
Moreover, the convergence time bounds contained additional terms coming from the condition number of a matrix that diagonalizes some of the matrices appearing the problem formulations, which can be arbitrarily large.
In this paper we analyze the convergence of the SVRG technique applied to TD (TD-SVRG) in two settings: p i q a pre-sampled trajectory of the Markov Decision Process (MDP) (finite sampling), and p ii q when states are sampled directly from the MDP (online sampling).
Our contribution is threefold:
• For the finite sample case we achieve significantly better results with simpler analysis.
We are first to show that TD-SVRG has the same convergence rate as SVRG in the convex optimization setting with a pre-determined learning rate of 1/8.
• For i.i.d.
online sampling, we similarly achieve better results with simpler analysis.
Similarly, we are first to show that TD-SVRG has the same convergence rate as SVRG in the convex optimization setting with a predetermined learning rate of 1/8.
In addition, for Markovian online sampling, we provide convergence guarantees that in most cases are better than state-of-the art results.
• We are the first to develop theoretical guarantees for an algorithm that can be directly applied to practice.
In previous works, batch sizes required to guarantee convergence were very large that made them impractical (see Subsection I.1) and grid search was needed to optimize the learning rate and batch size values.
We include experiments that
show our theoretically obtained batch size and learning rate can be applied in practice and achieve geometric convergence.
P ROBLEM FORMULATION
We consider a discounted reward Markov Decision Process (MDP) p S , A , P , r, γ q , where S is a state space, A is an action space, P “ P p s 1 | s, a q s,s 1 P S ,a P A are the transition probabilities, r “ r p s, s 1 q are the rewards and γ P r 0 , 1 q is a discount factor.
In this MDP agent follows policy π , which is a mapping π : S Ś A Ñ r 0 , 1 s .
Given that policy is fixed, for the remainder of the paper we will consider transition matrix P , such that: P p s, s 1 q “ ř a π p s, a q P p s 1 | s, a q .
We assume, that Markov process produced by transition matrix is irreducible and aperiodic with stationary distribution µ π .
The policy evaluation problem is to compute V π , defined as: V π p s q : “ E “ř 8 t “ 0 γ t r t `‰ .
Here V π is the value function, formally defined to be the unique vector which satisfies the equality T π V π “
V π , where T π is a Bellman operator, defined as: T π V π p s q “ ř s 1 P p s, s 1 q p r p s, s 1 q ` γV π p s 1 qq .
The TD(0) method is defined as follows: one iteration performs a fixed point update on randomly sampled pair of states s, s 1 with learning rate η : V p s q Ð V p s q` η p r p s, s 1 q` γV p s 1 q´ V p s qq .
When the state space size | S | is large, tabular methods which update a value for every state V p s q become impractical.
For this reason linear approximation is often used.
Each state a represented as feature vector ϕ p s q P R d and state value V π p s q is approximated by V π p s q « ϕ p s q T θ , where θ is a tunable parameter vector.
Now a single TD update on randomly sampled transition s, s 1 becomes:
[Equation]
where the second equation should be viewed as a definition of g s,s 1 p θ q .
Our goal is to find parameter vector θ ˚ such that average update vector is zero
[Equation]
This expectation is also called mean-path update ¯ g p θ q and can be written as:
[Table] [Table]
1/ 16 { λ
[Table]
p
[Table]
¯ g p θ q “
E s,s 1 r g s,s 1 p θ qs “
E s,s 1 rp γϕ p s 1 q T θ ´ ϕ p s q T θ q ϕ p
s qs ` E s,s
[Equation]
where the last line should be taken as the definition of A and b .
Finally, the minimum eigenvalue of matrix p
A ` A T q{ 2 plays an important role in our analysis and will be denoted as λ min .
There are few possible setting of the problem: the samples s, s 1 might be drawn from the MDP on-line (Markovian sampling) or independently ( i.i.d.
sampling): first state s is drawn from µ π , then s 1 is drawn from correspondent row of P .
The latter case analysis is covered in 6.
Another possible setting for analysis is the ”finite sample set” setting, in which states a data set
D “ tp s t , a t , r t , s t ` 1 qu Nt “ 1 of size N is drawn ahead of time following Markov sampling, and
TD(0) proceeds by drawing samples from this data set.
We analyze this case in Sections 4 and 5.
We make following standard assumptions:
Assumption 1.
The matrix A is non-singular.
Assumption 2.
||
ϕ p s q|| 2 ď 1 for all s P S .
Assumption 1 needed to guarantee that A ´ 1 b exists and the problem is solvable.
Assumption 2 is introduced for simplicity, it always might be fulfilled by rescaling feature matrix.
2. K EY IDEA OF THE A NALYSIS
In our analysis we often use function f p θ q , defined as:
[Equation]
Function f p θ q is a key characteristic function of TD learning.
In their paper Liu & Olshevsky (2020) introduce function f p θ q as p 1 ´ γ q||
V θ ´ V θ ˚
|| 2 D ` γ || V
θ ´ V θ
˚ || 2 Dir .
Then, authors define gradient splitting (linear function h p x q “ B p x ´ a q is gradient splitting of quadratic function j p x q “ p x ´ a q T Q p x ´ a q , where Q is symmetric positive semi-definite matrix, if B ` B T “ 2 Q ) and show that negation to mean-path update ´ ¯ g p θ q is indeed a gradient splitting of function f p θ q .
In this paper we do not use the fact that function f p θ q might be represented as weighted sum of Dnorm and Dirichlet norm and, for convenience, define function f p θ q based on its gradient splitting properties.
We rely on interpretation of TD learning as splitting of the gradient descent in our analysis.
In
Xu et al.
(2020) authors note: ”In Johnson & Zhang (2013) , the convergence proof relies on the relationship between the gradient and the value of the objective function, but there is not such an objective function in the TD learning problem.”
Well, viewing on TD learning as gradient splitting gives the relationship between the gradient and the value function, which allows implementation of similar analysis as in Johnson & Zhang (2013) to achieve stronger results.
It also gives the objective function f p θ q which is better measure of the distance to optimal solution, rather than || θ ´
θ ˚
|| 2 , and yields tighter convergence bounds.
T HE TD-SVRG ALGORITHM
We next propose a modification of the TD(0) method (TD SVRG) which can attain a geometric rate.
This algorithm is given below as Algorithm 1.
The algorithm works under the “fixed sample set” setting which assumes there already exists a sampled data set D .
This is the same setting was considered in Du et al.
However, the method we propose does not add regularization and does not use dual parameters, which makes it considerably simpler.
Algorithm
Parameters update frequency M and learning rate η
Initialize ˜ θ 0 .
Iterate: for m “ 1 , 2 , . . .
θ “ ˜ θ m ´ 1 , ¯ g p θ q “ 1 N ř s,s 1 P D g s
[Equation]
s 1 p θ q , where g s,s 1 p θ q “ p r p s, s 1 q ` γϕ p s 1 q T θ ´ ϕ p s q T θ q ϕ p s t q , θ
“ θ .
Iterate: for t “ 1 , 2 , . . .
, M
Randomly sample s, s 1 from the dataset and compute update vector v t “ g s,s 1 p θ t ´ 1 q ´ g s,s 1 p θ q ` ¯ g p θ q .
Update parameters θ t “ θ t ´ 1 ´ ηv t .
end
Set ˜ θ m “ θ t for randomly chosen t P p 0 , . . .
, M ´ 1 q .
end
Like the classic SVRG algorithm, our proposed TD-SVRG has two layers of loops.
We refer one step of the outer loop as an epoch and one step of inner loop as an iteration .
TD-SVRG keeps two parameter vectors: current parameter vector θ t , which is being updated every iteration, and the vector ˜ θ , which is updated the end of each epoch.
In the beginning of outer loop, the mean-path TD update vector ¯ g p ˜ θ q is computed with a pass through the entire data set.
This vector is used in inner loop to compute local updates v t “ g s,s 1 p θ t ´ 1 q ´ g s,s 1 p ˜ θ q ` ¯ g p θ q , where g s,s 1 p θ t ´ 1 q is a TD update computed on a uniformly randomly sampled data point from D with current parameter vector θ t and g s,s 1 p ˜ θ q is a TD update computed on the same data point.
Each iteration ends with an update of epoch vector, which is randomly chosen from parameter vectors during the epoch.
C ONVERGENCE ANALYSIS
In this section we show, that under simple assumption Algorithm 1 attain geometric convergence in terms of specially chosen function f p θ q with η is O p 1 q and M is O p 1 { λ min q .
4. P RELIMINARIES
In order to analyze the convergence of the presented algorithm we define expected square norm of difference in current and optimal parameters as w p θ q :
[Equation]
With this notation we provide an technical lemma.
All of our proofs are based on variations of this lemma.
Lemma 1.
If Assumptions 1, 2 hold, epoch parameters of two consecutive epochs m ´ 1 and m are related by the following inequality:
[Equation]
where the expectation is taken with respect to all previous epochs and choices of states s, s 1 during the epoch m .
Proof.
The proof of the lemma generally follows the analysis logic in Johnson & Zhang (2013), it might be found in Appendix A.
Lemma 1 plays an auxiliary role in our analysis and significantly simplifies it.
It introduces a new approach to the convergence proof by carrying iteration to iteration and epoch to epoch bounds to the earlier part of the analysis.
In particular, deriving bounds in terms of some arbitrary function u p θ q is now reduced to deriving upper bounds on || ˜ θ m ´ 1 || 2 and w p θ q and a lower bound on f p θ q in terms of the function u .
In fact, the function f p θ q itself will play the role of u p θ q in our proof.
In addition, it is now easy to demonstrate the point we made in Subsection 2.1.
The main problem of direct application of the SVRG convergence analysis to TD learning is that it requires the target function P p w q to be a sum of convex functions ϕ
i p w q , where all functions ϕ
i are L -Lipschitz and γsmooth (notation here is from Johnson & Zhang (2013)).
Of course, the TD learning problem cannot be represented in this form.
However, the main use of L -Lipschitz and smoothness properties in the original is to derive a bound on the expected norm of the difference between current and optimal parameters.
As will be shown later, in TD learning this expected norm ( w p θ q in our notation) still might be derived, even when a sum representation does not exist.
4. W ARM -U P : C ONVERGENCE IN TERMS OF SQUARED NORM
Firstly, we derive the convergence bound in terms of ||
θ ´
θ ˚
||
and show that they are consistent with previous results.
Proposition 1.
Suppose Assumptions 1, 2 hold.
If we chose learning rate as η “ λ min { 32 and number of iteration as M “ 32 { λ 2min , then Algorithm 1 has a convergence rate of:
[Equation]
Proof.
The proof is given in Appendix B
Note that deriving a convergence rate in terms of squared norm ||
˜
θ m ´
θ ˚ || 2 leads to batch size m to be O p 1 { λ 2 min q , which is better than results in Du et al.
(2017), since their results has complexity
O p κ 2 p C q κ 2 G q , where κ p C q is condition number of matrix C
“ E s P D r ϕ p s q ϕ p s q T s and κ G 9 1 { λ min p
A T C ´ 1 A q .
Experimental comparison of these values is provided in Subsection I.
4. F IRST M AIN R ESULT : C ONVERGENCE IN TERMS OF f p θ q
In this section derive a bound in terms of f p θ q .
For analysis to be simpler and illustrative we introduce one more assumption which we ease in section 4.5.
Assumption 3 (Dataset Balance) .
In the dataset D first state of the first sample and the second state of the last sample is the same state, i.e. s 1 “ s N ` 1 .
We need this assumption to omit dataset bias, so that states s and s 1 have the same distribution, as in the original MDP.
Having this assumption, we can proof theorem 1:
Theorem 1.
Suppose Assumptions 1, 2, 3 hold.
If we choose learning rate η “ 1 { 8 and number of inner loop iterations M “ 16 { λ min , then Algorithm 1 will have a convergence rate of:
[Equation]
Note that ˜ θ m refers to the iterate after m iterations of the outer loop.
Because we choose the length
M of the inner loop to be 16 { λ min , the total number of samples guaranteed by this theorem until
E r f p ˜ θ m qs ď ϵ is actually p 16 { λ min q log p 1 { ϵ q .
Proof of Theorem 1.
The proof is given in Appendix C.
Convergence analysis without Assumption 3 provided in Appendix Section D.
4. S IMILARITY OF SVRG AND TD-SVRG
Liu & Olshevsky (2020) show that negation to mean-path update ´ ¯ g p θ q is a gradient splitting of f p θ q .
In this work we show even greater importance of function f p θ q for TD learning process.
Recall convergence rate obtained in Johnson & Zhang (2013) for sum of convex functions setting:
[Equation]
where γ is a strong convexity parameter and L is Lipschitz smoothness parameter (employing notation from the original paper).
Function f
p θ q “ p θ ´ θ ˚ q T A p θ ´ θ ˚ q is 2 λ min p
A q strongly convex and 2-Lipschitz smooth, which means that convergence rate obtained in this paper is identical to the convergence rate of SVRG in convex setting (we have slightly better bound L instead of 2 L due to strong bound on w p θ q we derived for this setting).
This fact further extends the analogy between TD learning and convex optimization earlier explored by Bhandari et al.
and Liu & Olshevsky (2020).
B ATCHING SVRG C ASE A NALYSIS
In this section we extend our results to inexact mean-path update computation, applying the results of
Babanezhad et al.
(2015) to TD SVRG algorithm.
We show that geometric convergence rate might be achieved with smaller number of computations by estimating mean-path TD-update instead of performing full computation.
This approach is similar to Peng et al.
(2019), but again doesn’t require introduction of dual variables.
In addition, we provide a particular way to compute n m , which might be used in practice.
Since computation of mean-path error is not related to the dataset balance, in this section for simplicity we assume that dataset is balanced.
Theorem 2.
Suppose Assumptions 1, 2, 3 hold, then if learning rate is chosen as η “ 1 { 8 number of inner loop iterations M “ 16 { λ min and batch size n m “ min p N,
[Equation]
where c is a parameter, Algorithm 2 will have a convergence rate of:
[Equation]
where ρ P p 0 , 1 q is convergence rate and C is some constant.
Algorithm
Parameters update frequency M and learning rate η
Initialize ˜ θ 0 .
Iterate: for m “ 1 , 2 , . . .
θ “ ˜ θ m ´ 1 , choose batch size n m , sample batch D m of size n m from D without replacement, compute µ “ 1 n m ř s,s 1 P D
m g s,s 1 p θ q , where g s,s 1 p θ q “ p r p s, s 1 q ` γϕ p s 1 q T θ ´ ϕ p s q T θ q ϕ p s t q , θ
“ ˜ θ .
Iterate: for t “ 1 , 2 , . . .
, M
Randomly sample s, s 1 from D and compute update vector v t “ g s,s 1 p θ t ´ 1 q ´ g s,s 1 p ˜ θ q ` µ ,
Update parameters θ t “ θ t ´ 1 ´ ηv t .
end set ˜ θ m “ θ t for randomly chosen t P p 0 , . . .
, M ´ 1 q .
end
Proof. The proof is given in Appendix E. This theorem shows that during early epochs approximation of mean-path update is good enough to guarantee geometric convergence.
However, the batch size used for approximation increases geometrically with each epoch with rate ρ 2 , where ρ is a desired convergence rate, until it reaches size of the dataset N .
The constant C depends on parameter c and upper bound Z “ max θ p| θ ´ θ ˚ | ), where the max is taken over all parameter vectors seen during the run of the algorithm.
S ECOND MAIN RESULT : O NLINE IID SAMPLING FROM THE MDP
In this section we apply TD learning as gradient splitting analysis to the case of online i.i.d sampling from the MDP each time we need to generate a new state s .
We show that this view of TD learning as gradient splitting might be applied in this case to derive tighter convergence bounds.
One issue of TD-SVRG in i.i.d.
setting is that mean-path update may not be computed directly.
However, this issue might be addressed with sampling technique introduced in Section 5, which makes i.i.d.
case very similar to TD-SVRG with non-exact mean-path computation in finite samples case.
In this setting, geometric convergence is clearly not attainable with variance reduction, which always relies on a pass through the entire dataset.
Since here there is no data set, and one samples from the
MDP at every step, one clearly cannot make a pass through all states of the MDP (or, rather, this is unrealistic to do in practice).
To obtain convergence, one needs to take increasing batch sizes.
Our next theorem does so, while improving the scaling with condition number from quadratic to linear.
TD-SVRG algorithm for iid sampling case is very similar to Algorithm 2, with only difference that states s, s 1 are being sampled from the MDP instead of the dataset D .
Formal definition of Algorithm 3 might be found in section F.
Theorem 3.
Suppose Assumptions 1, 2 hold, then if learning rate is chosen as η “ 1 { 8 , number of inner loop iterations M “ 16 { λ min and batch size n m “
[Equation]
p 2 | r max | 2 ` 8 || θ || 2 q , where c is some arbitrary chosen constant, Algorithm 3 will have a convergence rate of:
[Equation]
where ρ P p 0 , 1 q is convergence rate and C is some constant.
Proof.
The proof is given in Appendix F.
To parse this, observe that as in 5, each epoch requires n m computations to estimate mean-path update and 16 { λ min to perform inner loop iterations.
Thus, as epoch number m grows, n m will dominate 16 { λ min , which results in the total computational complexity of O p max p 1 ϵ , 1 λ min q log p
1 ϵ qq , which is better than O p max p 1 ϵ , 1 λ 2min q log p 1 ϵ qq shown by Xu et al.
(2020) when λ min ą ϵ .
In practice, λ min is determined by the MDP and the feature matrix, while ϵ is a desired accuracy, thus, the former is given and the latter might be chosen.
In most scenarios, λ min is a small number and ϵ is chosen such that ϵ ă λ min .
Even if this is not a case, λ 2min is a very small number and most likely ϵ ă λ 2min .
Thus, in the absolute majority of cases results shown in this paper are stronger.
The same convergence result with predetermined constant learning rate cannot be derived for Markovian sampling case, in which update sampling strategy is different from one in classical SVRG sampling.
However, gradients splitting interpretation of TD learning still allows to achieve better convergence guarantees than in previous works for the absolute majority of problems.
Algorithm, discussion and convergence proof is provided in Appendix Section G.
E XPERIMENTS
A LGORITHMS COMPARISON
In this set of experiments we compare the performance of TD-SVRG with GTD2 Sutton et al.
Vanilla TD learning Sutton (1988) and
PD SVRG Du et al.
(2017) in finite sample setting.
Generally, experiment set-up is similar to Peng et al.
Datasets of size 5000 are generated fromenvironments: Random MDP Dann et al.
(2014) and Acrobot, CartPole and Mountain car OpenAI
Gym environments Brockman et al.
For Random MDP, we construct MDP environment with | S | “ 400 , 21, features and 10 actions, with actions choice probabilities generated from U r 0 , 1 q .
For OpenAI gym environments, agent select states uniformly at random.
Features constructed by applying RBF kernels to original states and then removing highly correlated features (correlation coefficient ą 0 . 5 ).
To produce datasets of similar sizes we resampled dataset if smallest eigen-value of its matrix A was outside the interval r 0 .
54 s ¨ 10 ´ 4 , which corresponds to TD-SVRG batch sizes between 30000 and 50000.
Decay rate γ is set to 0.95.
Hyperparameters for algorithms selected as follows: for TD-SVRG theoretically justified parameters are selected, learning rate η “ 1 { 8 and number of inner loop computations M “ 16 { λ min ; for
GTD2 we used parameters which are suggested for small problems α “ 1 and β “ 1 .
For vanilla
TD decreasing learning rates are set to α “ 1 {? t and α “ 1 { t .
For PD-SVRG setting parameters to theoretically suggested is not feasible, since even for simple problems values of number of inner loop computations M is too large (see Appendix Subsection I.1).
Following original paper Du et al.
(2017) we run a simple grid search are pick best performing values, which are σ θ “ σ w “ 0 .
1 { λ max p ˆ C q .
Results presented on Figure 1.
Each algorithm for each setting was run 10 times, average result is presented.
As theory predicts, TD-SVRG and PD-SVRG converge geometrically, while GTD and vanilla TD converge sub linearly.
[Figure]
In this set of experiments we compare the performance of TD-SVRG, VRTD and three Vanilla TD with fixed and decreasing learning rates in i.i.d.
sampling case.
States and rewards are sampled from the same MDP as in Section 7.1.
Hyperparameters are chosen as follows: for TD-SVRG - learning rate η “ 1 { 8 , number of inner loop computations M “ 16 { λ min .
VRTD - learning rate α “ 0 .and batch size M “ 2000 .
For vanilla TD with constant learning rate its value set to 0 .
1 and for decreasing learning rate it is 1 { t , where t is number of performed update.
Average results overruns presented on Figure 2.
7. R EPRODUCIBILITY
Authors provide a link to anonynous github repository with code and instruction how to reproduce the experiments.
C ONCLUSION
In the paper we utilize a view on TD learning as splitting of gradient descent to show that SVRG technique applied to TD updates attain similar convergence rate as SVRG in convex function setting.
Our analysis addresses both finite sample and i.i.d.
sampling cases, which previously were analyzed separately, and improves state of the art bounds in both cases.
In addition we show that gradient splitting interpretation helps to improve convergence guarantees in Markovian sampling case.
The algorithms based on our analysis have fixed learning rate and small number of inner loop computation, easy to implement and demonstrates good performance during experiments.
A P ROOF OF L EMMA
The proof follows the same logic as in Johnson & Zhang (2013) and organized in four steps.
Step A.1.
In the original paper proof starts with deriving a bound on the squared norm of the difference between current and optimal sets of parameters.
Since the introduction of w p θ q this step in our proof is trivial.
[Equation]
Step A.2.
During Step 2 we derive a bound on the norm of an single iteration t update v t “ g
[Equation]
g p ˜
[Equation]
q , assuming that states s, s 1 were sampled randomly during step t :
[Equation]
The first inequality uses E || a ` b || 2 ď 2 E ||
a || 2 ` 2 E
||
b
|| 2 .
The second inequality uses the face that second central moment is smaller than second moment.
The last equality uses the equality from step 1.
Step A.3.
During this step we derive a bound on the expected squared norm of a distance to optimal parameter vector after a single update t :
[Equation]
The inequality uses the bound obtained in step 2.
After rearranging terms it becomes:
[Equation]
Step A.4.
During this step we sum the inequality obtained in Step 3 over the epoch and take expectations with respect to all choices of pair of states s, s 1 and all previous history and use the random choice property to obtain Equation 1 which relates parameter vectors of two consecutive epochs:
[Equation]
M ÿ t “2 η 2 E w p θ t ´ 1 q ď
M ÿ t “
E ||
θ t ´ 1 ´
θ ˚
|| 2 `
M ÿ t “2 η 2 E w p ˜ θ q
We analyze this expression term-wise.
ř
M t “ 1 E ||
θ t ´ {+
+} ´
θ ˚
||[-
-]
and ř M t “ 1 E ||
θ t ´ θ ˚
|| 2 consist of same terms, except the first term in the first sum and the last term in the last sum, which are E || θ
´ θ ˚
|| 2 and E ||
θ M ´
θ ˚
|| 2 .
Since E ||
θ M ´
θ ˚ ||is always positive and it is on the left hand side of the inequality, we could drop it.
We denote the parameter vector θ chosen for epoch parameters in the end of the epoch ˜ θ m .
Since this vector is chosen uniformly at random among all iteration vectors θ t , ř Mt “ 1 E f p θ t ´ 1 q “ M E f p
˜
θ m q
and ř Mt “ 1 E w p θ t ´ 1 q “ M E w p ˜ θ m q .
At the same time, ˜ θ , which was chosen in the end of the previous epoch remains the same throughout the epoch, therefore, ř Mt “ 1 E w p ˜ θ q “ M E w p ˜ θ q .
Note, that current epoch starts with setting θ
“ ˜ θ .
Also, to underline it is previous epoch, we denote it as ˜ θ m ´ 1 .
Plugging this values in we have 3:
[Equation]
B P ROOF OF P ROPOSITION
To transform inequality 3 from Lemma 1 into convergence rate guarantee [Equation] we need to bound w p θ q and f p θ q in terms of ||
θ ´
θ ˚
|| 2 .
Both bounds are easy to show:
[Equation]
Plugging these bounds into Equation 3 we have:
[Equation]
Which yields epoch to epoch convergence rate of:
[Equation]
For this expression to be ă 1 , we need that ηM to be O p 1 { λ min q , which means that η needs to be
O p λ min q for Mη 2 to be O p 1 q .
Therefore, M need to be O p 1 { λ 2 min q .
Setting η “ λ min { 32 and m “ 32 { λ 2 min yields convergence rate of 5 { 7 .
C P ROOF OF T HEOREM
The same as in the previous section, we start with deriving bounds, but this time we bound ||
θ ´
θ ˚ ||and w p θ q in terms of f p θ q .
First bound is straightforward:
[Equation]
For w p θ q we have:
[Equation]
first inequality uses Assumption 2, third equality uses Assumption 3, ( ř s 1 γ 2 ϕ p s 1 q ϕ p s 1 q T “ ř s γ 2 ϕ p s q ϕ p s q T , since s and s 1 are the same set of states).
The last inequality uses the fact that γ ă 1 .
Plugging these bound into Equation [Equation] we have:
[Equation]
which yields epoch to epoch convergence rate of:
[Equation]
Setting η “ 18 and M “ 16 λ min we have the desired inequality.
D C ONVERGENCE ANALYSIS WITHOUT DATASET BALANCE
In this section we show the convergence bound for the problem without Assumption 3.
In this case, the problem is that after the sampling, s and s 1 in the dataset does not have the same distribution, i.e., the first element of the tuples p s t , a t , r t , s t ` 1 q in our dataset D need not have the same distribution as the last element of this tuple.
Indeed, it could happen that a particular state occurs a different numbers of times as the first element of the tuples in D as compared to the last element, which would not happen under Assumption 3.
When this happens, we will say that the data set is unbalanced .
In that case, ¯ g p θ q need not be a gradient splitting of function f p θ q .
One might hope that, when the size of the data set N is large, this effect has an impact which decays to zero with N .
Our second main result shows something even stronger: we show that the effect of unbalacedness disappears completely for large N .
Thus our next theorem completely recovers the performance attained by Theorem 1 for large N .
The catch is the size of the dataset has to be at least as big as λ ´ 1min for this to happen.
Theorem 4.
Suppose Assumptions 1, 2 hold and dataset is unbalanced.
Define error term J “ 4 γ 2 Nλ min .
Then, if we choose learning rate η “ 1 {p 8 ` J q and number of inner loop iterations
M “ 2 {p λ min η q , Algorithm 1 will have a convergence rate of:
[Equation]
Proof.
The proof is given in Appendix D.1.
Note, that in this case η P O p 1 max p 1 , 1 {p Nλ min qq q and M P O p 1 λ min η q , which always better than parameters required to guarantee convergence of || θ ´
θ ˚
|| 2 (Proposition 1).
Note that the guarantees of this theorem are identical to the guarantees of Theorem 1 in the unbalanced case when N ě λ ´ 1 min .
D. P ROOF OF THEOREM
To proof the theorem we follow the same strategy as in C. For the f p θ q we can use the same bound:
[Equation]
Bound for w p θ q is a little bit more difficult:
[Equation]
The first inequality follows from the assumption about norms of feature vectors.
The third equality is obtained by adding and subtracting γ 2 N p θ ´ θ ˚ q T ϕ p s 1 q ϕ p s 1 q T p θ ´ θ ˚ q .
Second inequality uses the fact that γ 2 ă 1 .
We denote maximum eigen-value of matrix ϕ p
s N ` 1 q ϕ p s N ` 1 q T ´ ϕ p s 1 q ϕ p s 1 q T as
K (note that K ď 1 ).
Thus,
[Equation]
Plugging these bounds into Equation 3 we have:
[Equation]
Which yields convergence rate of:
[Equation]
To achieve constant convergence rate, for example 23 , we set up η such that η p 2 ` γ 2 Nλ min q “ 0 . 25 , thus the second term is equal to 1/3 and η “ 1 8 ` 4 γ 2 Nλmin .
Then, to make the first term equal to 1/3, we need to set
M “
[Equation]
Thus, η is a scale of 1 max p 1 , 1 {
p Nλ min q and M is a scale of
[Equation]
Nλ
[Equation]
E P ROOF OF T HEOREM
In the first part of the proof we derive an inequality which relates model parameters of two consecutive epochs similar to what we achieved in previous proofs, but with error term.
In this part of the proof we follow the same 4 steps logic as while proof of Lemma 1.
In the second part of the proof we show that there are conditions under which error term converges to 0.
Step E.1.
During the first step we use the bound obtained in inequality 4:
[Equation]
Step E.2.
During this step we derive a bound on the squared norm of a single update E r|| v t
|| 2 s .
But now, compared to previous case, we do not compute the exact mean-path updated ¯ g p θ q , but its estimate, and assume our computation has error µ “ ¯ g p θ q ` e .
Thus the single update vector will be
[Equation]
Thus, the bound on the single update might be derived as:
[Equation]
where first inequality uses E || A ` B || 2 ď 2 E
||
A
|| 2 ` 2 E || B
|| 2 , second inequality uses E || A ´ E r A s|| 2 ď E
||
A ||
and the third inequality uses the result of Step E.1.
Step E.3.
During this step, we derive a bound on a vector norm after a single update: E ||
[Equation]
Rearranging terms we obtain:
[Equation]
θ t ´ 1 q ´ 4 η 2 f p θ t ´ 1 q ď || θ t ´
[Equation]
´ θ
[Equation]
|
[Equation]
`
[Equation]
η
[Equation]
f p ˜
[Equation]
´
[Equation]
η p θ t ´ 1 ´ θ ˚ q T e ` 2 η
[Equation]
E
[Equation]
e
[Equation]
|ď
||
θ t ´
[Equation]
´ θ ˚ || 2 ` 4 η
[Equation]
f p ˜ θ q `
[Equation]
η || θ t ´ 1 ´
θ ˚
||
¨
||
e || ` 2 η 2 E
|| e ||
Step E.4.
Now derive a bound on epoch update.
We assume that quantity ||
θ t ´
´
θ
˚
|| might be bounded by constant Z .
Similarly, we denote an error term from previous epoch as e m ´ 1 .
We use the similar logic as during the proof of theorem 1.
Since error term doesn’t change over the epoch, thus, summing over the epoch we have:
[Equation]
Rearranging terms we have the bound:
[Equation]
To obtain convergence, we need to guarantee geometric convergence of first and second term in the sum separately.
The first term is dependent on inner loop updates, its convergence is analyzed in
Theorem 1.
Here we show how to achieve a similar geometric convergence rate of the second term.
Since error term has 0 mean and it is finite sample case with replacement, expected squared norm might be bounded by:
[Equation]
where S 2 is a bound on update vector norm variance.
If we want the error to be bounded by cρ 2 m , we need the number of batch computations n m to satisfy the condition:
[Equation]
Satisfying this condition guarantees that the second term has geometric convergence:
[Equation]
It is only left to derive a bound S 2 for on update vector norm sample variance:
[Equation]
F P ROOF OF T HEOREM
TD-SVRG algorithm for iid sampling case is described as Algorithm 3:
The proof of its convergence is very similar to E, the only difference is that now we derive expectation with respect to MDP instead of fixed dataset.
Parameters update frequency M and learning rate η
Initialize ˜ θ 0 .
Iterate: for m “ 1 , 2 , . . .
θ “ ˜ θ m ´ 1 , choose batch size n m , sample batch D m of size n m , compute µ “ 1 n m ř s,s 1 P D
m g s,s 1 p θ q , where g s,s 1 p θ q “ p r p s, s 1 q ` γϕ p s 1 q T θ ´ ϕ p s q T θ q ϕ p s t q , θ
“ ˜ θ .
Iterate: for t “ 1 , 2 , . . .
, M
Randomly sample s, s 1 and compute update vector v t “ g s,s 1 p θ t ´ 1 q ´ g s,s 1 p ˜ θ q ` µ ,
Update parameters θ t “ θ t ´ 1 ´ ηv t .
end set ˜ θ m “ θ t for randomly chosen t P p 0 , . . .
, M ´ 1 q .
end
Step F.1.
During the first step we use the bound obtained during the proof of theorem 1:
[Equation]
[Equation] first inequality uses Assumption 2, third equality uses the fact that µ π is a stationary distribution of
P ( ř s 1 γ [Equation] µ π p s q P p s, s 1 q ϕ p s [Equation]
q
[Equation]
γ
[Equation] ϕ [Equation] q
[Equation]
q γ
[Equation]
ϕ p s q T ).
The last inequality uses the fact that γ ă 1 .
Step F.2.
During this step we derive a bound on the squared norm of a single update E r|| v t
|| 2 s .
Similarly with E we assume inexact computation of mean-path update µ “ ¯ g p θ q` e .
Thus the single update vector becomes:
[Equation]
Norm of this vector is bounded by:
[Equation]
where first inequality uses E || A ` B || 2 ď 2 E
||
A
|| 2 ` 2 E || B
|| 2 , second inequality uses E || A ´ E r A s|| 2 ď E
||
A ||
and the third inequality uses the result of Step F.1.
Step F.3.
Bound on a vector norm after a single update:
[Equation]
Rearranging terms we obtain:
[Equation]
Step F.4.
Now derive a bound on epoch update.
We assume that quantity ||
θ t ´
´
θ
˚
|| might be bounded by constant Z .
Similarly
[Equation]
we denote an error term from previous epoch as e m ´ 1 .
We use the similar logic as during the proof of Theorem 1.
Since error term doesn’t change over the epoch, thus, summing over the epoch we have:
[Equation]
p ˜ θ q
[Equation]
E |
[Equation]
m ´ 1 |
[Equation]
E ||
e m ´
1 ||
Rearranging terms we have the bound:
[Equation]
Similarly to C convergence for the first term might be obtained by setting learning rate η “ 1 { 8 and number of inner loop iterations M “ 16 { λ min .
To guarantee convergence of the second term, we need to bound E
||
e m ||
In the infinite population with replacement case norm of the error vector is bounded by:
[Equation]
where S 2 is a bound update vector norm variance.
If we want the error to be bounded by cρ 2 m , we need the number of batch computations n m to satisfy the condition:
[Equation]
Satisfying this condition guarantees that the second term has geometric convergence:
[Equation]
Similarly to E [Equation] bound on sample variance S 2 might be derived as follows:
[Equation]
G M ARKOVIAN SAMPLING CASE ALGORITHM AND ANALYSIS .
Markovian sampling case is the hardest to analyse due to its dependence on MDP properties, which makes establishing bounds on various quantities used during the proof much harder.
Applying gradient splitting view helps to improve over existing bounds but derived algorithm does not have a nice property of constant learning rate.
To deal with sample-to-sample dependencies with utilize one more assumption often used in the literature:
Assumption 4.
The considered MDP is irreducible and aperiodic and there exist constant m ąand ρ P p 0 ,
1 q such that
[Equation]
where d TV p P, Q q denotes the total-variation distance between the probability measures P and Q.
Another thing we need to employ is projection, which will help to set a bound on update vector v .
Following Bhandari et al.
(2018) and Xu et al.
(2020) after each iteration we project parameter vector on a ball or radius R (denoted as Π R p θ q “ arg min θ 1 : | θ 1 |ď R | θ ´ θ 1 | 2 .
We assume that | θ ˚ | ď R , choice of R which guarantees it might be found in Bhandari et al.
(2018), Section 8.2.
Adding projection results in Algorithm 4.
Guarantees of convergence of Algorithm 4 are given in Theorem 5.
Theorem 5.
Suppose Assumptions 1, 2, 4 hold, then output of Algorithm 4 will satisfy: [Equation] where C “ 4 p 1 `p m ´ 1 q ρ q p 1 ´ ρ q r 4 R 2 ` r 2 max s . Proof.
The proof is given in Appendix ?? .
Theorem 5 implies that if we choose s “ O p log p 1 { ϵ qq , n m “ O p 1 {p λ min ϵ qq and η “ O p ϵ { log p 1 { ϵ q and M “ O p log p 1 { ϵ q ϵλ min q , total sample complexity to achieve accuracy of ϵ is:
Algorithm
Parameters update frequency M , learning rate η , projection radius R and batch size n m
Initialize ˜ θ 0 .
Iterate: for m “ 1 , 2 , . . .
θ “ ˜ θ m ´ 1 , sample trajectory D m of length n m , compute µ “ 1 n m ř s,s 1 P D m g s,s 1 p θ q , where g s,s 1 p θ q “ p r p s, s 1 q ` γϕ p s 1 q T θ ´ ϕ p s q T θ q ϕ p s t q , θ
“ ˜ θ .
Iterate: for t “ 1 , 2 , . . .
, M
Randomly sample s, s 1 and compute update vector v t “ g s,s 1 p θ t ´ 1 q ´ g s,s 1 p ˜ θ q ` µ ,
Update parameters θ t “ Π R p θ t ´ 1 ´ ηv t q . end set ˜ θ m “ θ t for randomly chosen t P p 0 , . .
, M ´ 1 q .
end
O p log 2 p 1 { ϵ q ϵλ min q
In the most practical application his result is better than O p 1 ϵλ 2min log p 1 { ϵ qq , since log p 1 { ϵ q{ λ min ąfor practical values of ϵ and λ min .
H P ROOF OF T HEOREM
In the Markovian sampling case, we cannot simply apply Lemma 1; due to high estimation bias the bounds on f p θ q and w p θ q will not be derived based on current value of θ , but based on global constraints on the updates guaranteed by applying projection.
First, we analyse a single iteration on step t of epoch m , during which we apply the update vector v t “ g t p θ q ´ g t p ˜ θ q ` µ p ˜ θ q .
The update takes the form:
[Equation]
where the expectation is taken with respect to s, s 1 sampled during iteration t .
Recall that under Markovian sampling, E r g t p θ t ´ 1 qs ‰ ¯ g p θ t ´ 1 q and that for the expectation of the estimated meanpath update we have E r µ p ˜ θ
q| s m ´
1 s ‰ ¯ g p ˜ θ q , where s m ´ 1 is the last state of epoch m ´ 1 .
To tackle this issue, we follow the approach introduced in a previous works (Bhandari et al. (2018), Xu et al. and rewrite the expectation as a sum of mean-path update and error terms.
Similar to Bhandari et al.
(2018), we denote the error term on a single update as ζ :
[Equation]
For an error term on the trajectory we follow Xu et al.
and denote it as ξ :
[Equation]
Applying this notation, 6 can be rewritten as:
[Equation]
Error terms can be bounded by slightly modified lemmas from the original papers.
For ζ p θ q , we apply a bound from Lemma 11 in Bhandari et al.
[Equation]
In the original lemma, a bound on E r ζ t p θ qs is stated, however, in the proof a bound on absolute value of the expectation is also derived.
For mean-path estimation error term, we use a modified version of Lemma 1 in Xu et al.
The proof of this lemma in the original paper starts by applying the inequality
[Equation]
to the expression p θ ´ θ ˚ q T p µ p θ q ´ ¯ g p θ qq , with k “ λ A { 2 (using the notation in Xu et al.
For the purposes of our proof we use k “ λ min .
Thus, we will have the expression:
[Equation]
Also, note, that the term E
|| v t
|| 22 might be bounded as E || v t
||
22 ď 18 R 2 .
Plugging these bounds into 7 we obtain:
[Equation]
Summing the inequality over the epoch and taking expectation with respect to all previous history [Equation] we have:
[Equation]
Then we divide both sides by 2 ηM and use || ˜ θ s ´
1 ´ θ ˚
||
22 ď 1 { λ min f p ˜ θ s ´ 1 q to obtain:
[Equation]
We choose η and M such that ηMλ min “ 2 .
We then apply this inequality to the value of the function f in the first term in the right-hand side recursively [Equation] which yields the desired result:
[Equation]
I
D ETAILS ON ALGORITHM IMPLEMENTATION I. C OMPARISON OF THEORETIC BATCH SIZES
In this subsection we compare the values of batch sizes which are theoretically required to guarantee convergence.
We compare batch sizes of three algorithms: TD-SVRG, PDSVRG (Du et al.
[Table]
and VRTD (Xu et al.
Note that PDSVRG and VRTD are algorithms for different settings, but for TD-SVRG the batch size value is the same: 16 { λ min , thus, we compare two algorithms in the same table.
We compare the batch sizes required by algorithm for an MDP with 400 states (similar to one used for experiments in Subsections 7.1 and 7.2).
Since a batch size is dependent on the smallest eigenvalue of the matrix A , which is, in turn, is dependent on the dimensionality of the feature vector, we do the comparison for different feature vector sizes: 5, 10, 20 and 40 randomly generated features + 1 constant feature for each state.
We generate 10 datasets and environments for each feature size.
Our results are summarized in table 2.
I. B ATCHED SVRG PERFORMANCE
In this set of experiments we compare the performance of TD-SVRG and batched TD-SVRG in finite-sample case.
We generate 10 datasets of size 50000 from the similar MDP as in Section 7.1.
Algorithms also run with the same hyperparameters.
Average results over 10 runs presented on
Figure 3 and show, that batched TD-SVRG saves a lot of computations during the earlier epochs, which provides faster convergence. θ ´ θ ˚ |q