P OLICY O PTIMIZATION VIA S TOCHASTIC R ECURSIVE G RADIENT A LGORITHM
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we propose the StochAstic Recursive grAdient Policy Optimization (SARAPO) algorithm which is a novel variance reduction method on Trust
Region Policy Optimization (TRPO).
The algorithm incorporates the StochAstic
Recursive grAdient algoritHm(SARAH) into the TRPO framework.
Compared with the existing Stochastic Variance Reduced Policy Optimization (SVRPO), our algorithm is more stable in the variance.
Furthermore, by theoretical analysis the ordinary differential equation and the stochastic differential equation (ODE/SDE) of SARAH, we analyze its convergence property and stability.
Our experiments demonstrate its performance on a variety of benchmark tasks.
We show that our algorithm gets better improvement in each iteration and matches or even outperforms SVRPO and TRPO.
I NTRODUCTION
Reinforcement Learning (RL) (Sutton et al., 1998) is a dynamic learning approach to interact with the environment and make actions so certain measure of cumulative rewards is maximized.
It achieves remarkable performance in several tasks including continuous control, video games, etc.
Among all existing RL algorithms, policy gradient (Sutton et al., 1999) is the most fundamental method for RL.
However, policy gradient suffers from sample inefﬁciency and high variance during the training phase.
We could use reward shaping techniques such as, REINFORCE(Williams, 1992),
GAE(Schulman et al., 2015b), etc., as an variance reduction.
Besides, trust region policy optimization (TRPO) was proposed by (Schulman et al., 2015a) to remedy this problem.
It merges the trust region method (Nocedal & Wright, 2006) and the natural gradient theory (Amari, 1998) into the policy gradient framework.
The basic idea is to make the policy move toward the direction that improves mean episode rewards under the constraint of the Kullback-Leibler (KL) divergence with the old policy distribution.
Compared to traditional gradient descent which corresponds to the L 2 constraint, constraints that uses KL distance has been widely adopted as an alternative choice.
The KL constraint results a natural gradient update ﬁrstly proposed by Amari (1998).
Later Kakade (2001) merges the natural gradient into the policy gradient (See Martens (2014) for further extensions).
Natural gradient attempts to solve the problem of vanishing gradients in the so-called plateau areas.
TRPO which strictly constrain the KL distance achieves a more robust performance on continuous control tasks.
Policy gradient optimization method shares similar structure with traditional optimization thus can be powerfully augmented by variance reduced gradient methods.
For example, SVRG (Johnson & Zhang, 2013) is a kind of stochastic variance reduced method that overcomes the fallback of
SGD of large variance and potentially accelerates the RL training.
Previous works use SVRG to accelerate policy gradient (SVRPG) and its TRPO variants (SVRPO)
(Xu et al., 2017; Papini et al., 2018).
The variance reduced version of policy gradient allows a multi-step optimization for each batch of data, while the vanilla policy gradient optimizes the policy once each iteration.
In theory,
SVRG accommodates with larger stepsize for each iteration and observably outperforms traditional
PG and TRPO, respectively.
Furthermore, the introduction of stochastic variance reduction not only improves sample efﬁciency but also stabilizes the training process by reducing the variance and accommodate larger stepsize.
In this paper, we focus on studying variance reduction method on TRPO.
We discussed an alternative variance reduction method called SARAH, recently proposed by (Nguyen et al., 2017) whose convergence rate matches that of SVRG in the convex case and has the inner-loop convergence property that SVRG does not possess.
We propose a new algorithm named SARAPO which hybrids the SARAH method with TRPO and analyze the dynamics of SVRG/SARAH on natural gradient updates via its approximating ordinary and stochastic differential equation (ODE/SDE).
In experiments, we test the performance and sample efﬁciency of SARAPO, SVRPO, and TRPO.
To allow larger steps in SVRPO/SARAPO, we remove the KL constraint in the line search of TRPO.
We verify that variance reduction enables a larger leap towards the optima and ﬁnd that SARAPO outperforms TRPO and matches the performance of SVRPO in most of the tasks.
The rest of this paper is organized as the following: we introduce some background, propose and analyze our algorithm in Section 2.
In Section 3, we theoretically analyze our proposed SARAPO algorithm from a dynamical system viewpoint.
We present the numerical results of experiments in
Section 4 and conclude our paper in Section 5.
P RELIMINARIES AND M ETHODOLOGY
The reinforcement learning task can be considered as solving a discrete time Markov Decision
Process (MDP) M = {S , A , P , R , γ, ρ } , where S is the set of states, A is the set of actions,
R : S ×
A → R is the reward function, γ is the discount factor and ρ the initial state distribution.
The policy gradient method take steps by directly maximizing the expected sum of discounted rewards:
[Equation]
with respect to the parameters w of the stochastic policy π w ( a | s ) .
The gradient of the objective function ∇ w L is given by:
[Equation]
where Q π is the state-action function (i.e., the Q function) deﬁned as the expected return when taking action a t from state s t .
The expectatioin is calculated by Monte-Carlo simulation by sampling n timesteps { s t , a t , r t } nt
=1 of policy π w .
2. T RUST R EGION P OLICY G RADIENT
As the policy gradient is highly unstable and sample inefﬁcient, TRPO introduces the trust region method to policy gradient to limit how far the new policy is deviated from the old policy.
Instead of the L 2 norm in trust region method, TRPO takes KL as the measure of distance.
Furthermore, to make computation easy, the algorithm turns the KL constraint problem into a penalized problem.
TRPO optimizes the surrogate loss at every iteration:
[Equation]
At every iteration, TRPO takes an update of:
[Equation]
where v t = ∇ w L and F is the Fisher Information Matrix (FIM)
[Equation]
The initial step size η is calculated by limiting the KL distance within the range δ :
[Equation]
Algorithm
Initialization:
Initialize w 0 =
w 1 to be a randomized parameter.
for t = 1 to T do if t % m =
=
then
Set ˜ w = w t
Run policy π w t for N timesteps.
Store a batch of trajectories D .
Calculate the baseline gradients:
[Equation]
else
Draw a mini-batch D t of size M from D
Calculate the estimated gradient w t :
[Equation]
end if
Update policy parameter with Algorithmend for
Ensure: w ← ˜ w T
2. SARAH AND SVRG FOR N ATURAL G RADIENT
SARAH (Nguyen et al., 2017) and SVRG (Johnson & Zhang, 2013) are among the recently popularized variance reduced gradient algorithms in optimization.
They reduce the variance during the optimization process and achieve a faster convergence rate.
Instead of calculating the full gradient ∇ L .
Recall that SVRG calculate a mini batch of size m in each inner loop, and it updates the weights as follows:
[Equation]
where ˜ v and ˜ w stores the full gradient and weight parameters as the output of the previous loop, and
L m is the objective function estimated using m samples.
Here the update rule of w t involves the inverse of the Fisher information matrix as in TRPO, which is computed via CG method.
Also, v t provides an unbiased estimation of the true gradient while the variance is much reduced when w t is close to ˜ w
SARAH, on the other hand, estimates each gradients using a recursive gradient update rule.
It adds the gradient estimated one step before v t by the difference of gradients between the current parameter and the previous parameter ∇ w L m ( w t ) − ∇ w L m ( w t − 1 ) estimated over a mini batch.
The core update rule is as follows:
[Equation]
In SARAH, the algorithm exchanges unbiasedness of the gradient with a much reduced variance of such.
We will analyze the iteration in Section 3.
2. S TOCHASTIC G RADIENT R ECURSIVE P OLICY O PTIMIZATION
Our goal in this section is to combine the SARAH with TRPO for natural gradient and formally propose the SARAPO algorithm.
The details of calculating the estimated gradients are listed in
Algorithm 1.
There, we use ˜ w to denote the old policy parameters saved at the beginning of each outer loop.
After calculating the estimate of the gradient, we conduct a TRPO step.
We note that as the Fisher Information Matrix can be approximated by Hessian matrix of the KL divergence when
Estimate the Fisher Information Matrix ˆ
[Equation]
=
[Equation]
Update policy parameter with mini-batch I j :
[Equation]
where the ˆ F − 1 ( w t ) · ˆ v t is computed using the conjugate gradient method
Initialize η t according to equation
Line-search η j , η j / 2 , η j / 4 , · · · for improving L π ˜ w ( π w ) the current distribution exactly matches that of the base distribution.
As a consequence, we should use the second derivative of the KL distance between the current parameters and the parameters one step before to estimate ˆ F , which is to penalize the KL between each inner loop updates, instead of the outer loop updates in TRPO.
The Fisher information matrix is only estimated using a mini batch of the data.
And the inverse Fisher vector product is calculated by the conjugate gradients algorithm.
Different from the line-search process in TRPO, we do line-search only to ﬁnd the stepsize η that can improve the policy loss.
Due to the fact that the variance is reduced, we can take a larger step away.
So we remove the KL condition in the line-search and let the algorithm to ﬁnd the KL distance of the outer loop updates itself.
Detailed description is in Sec 4.
T HEORY OF D YNAMICAL S YSTEMS
Recall that the core inner-loop update rule of Stochastic Recursive Gradient Method (SARAH)
(Nguyen et al., 2017) is, assuming the minibatch size is 1,
[Equation]
Here we provide a simple alternative viewpoint of (random) dynamical systems.
The method of using dynamical systems and continuous-time framework have been adopted to characterize variants of gradient descent algorithms (Su et al., 2016;
Li et al., 2017), but to our best knowledge, it has never been used to analyze variance reduced gradient methods before this work.
As we could see from Algorithm 1, we applied a scaled-form of SARAH in combination with Trust
Region Policy Optimization by the inverse Fisher Information matrix F − 1 .
Seeing this, the inner loop update has the essential form of
[Equation]
For simplicity of analysis, we consider the quadratic approximation approach.
Let the quadratic objective function L ( w ) :
= 1 n (cid:80) ni
=1 L i ( w ) , with
[Equation]
We sketch the analysis using ODE and SDE, in order to better understand the dynamics.
3. ODE ANALYSIS
By rescaling the time t = η − 1 s according to the stepsize η , we deﬁne a new process
[Equation]
and X ( s )
=
w (cid:98) η − 1 s (cid:99) .
Then the newly deﬁned process is converted from equation 5 and has the form
[Equation]
where e s + η is a martingale difference noise term of O (1) , and
[Equation]
Substituting X ( s ) − X ( s − η ) with − ηF − 1 V ( s ) , we derive the update rule for V ( s )
[Equation]
η .
From the standard weak convergence theory, one easily conclude the following:
Theorem 1 When η → 0 + , the scaled SARAH update rule (5) can be approximated by the following
ODE system
[Equation]
Note in (6), the dynamics of V ( s ) forms an autonomous ODE, which is independent of the dynamics of X ( s ) .
Left multiplying both sides of the ﬁrst line of equation 6 by F − 1 , one obtains
[Equation]
Noticing V (0)
=
HX 0 , we can easily verify that the V ( s ) can be solved as
[Equation]
By combining V ( s ) ’s solution with the update rule for X ( s ) , we have
[Equation]
and has the expressed solution
[Equation]
As a result, we reach the following solution to (6):
[Equation]
From (7) and (8), both the vectors X ( s ) and V ( s ) exponentially decays with rate matrix F − 1 H which is consistent with the theory of natural gradient ascent (Kakade, 2001).
3. SDE ANALYSIS
The ODE analysis simply neglects the effect of noise.
Here we turns to develop a stochastic differential equation approximation tool to exploit the effect of noise.
Denote H i
= H + E i , H being the Hessian for L ( w ) .
Further assume that the noise term E i is σdiag ( χ 1 , . . .
, χ d ) where χ i are i.i.d.
standard normal, for simplicity.
Deﬁne the coordinate-wise product of two vectors ( x 1 , . . .
, x d ) (cid:62) • ( y 1 , . . .
, y d ) (cid:62) =
( x 1 y 1 , . . .
, x d y d ) (cid:62) .
Let W ( s ) =
( W 1 ( s ) , . . .
, W d ( s )) (cid:62) be a d -dimensional standard Wiener processes.
Then under standard assumptions, the inner loop has an update rule for V ( s ) which can be written as
[Equation]
As η → 0 + , it becomes a SDE system
[Equation]
which can be simpliﬁed as
[Equation]
From the above analysis, one can verify that the approximating SDE in (9) introduces a small (size of O ( η 1 / 2 ) ) noise deviation from its ODE curve, and hence enjoys similar property as the natural gradient descent update.
In the case where H is convex, the dynamics validates the property of innerloop convergence in (Nguyen et al., 2017)
[Theorem 1a & 1b], and hence the multi-loop geometric convergence [Theorem 2].
The per-step cost, however, turns to be evaluations of two individual gradients instead of the whole-batched gradients.
Remark 1 One may also conduct a similar analysis for SVRG and conclude the same approximating ODE for scaled SVRG.
The approximating SDE, nevertheless, shall differ by an extra noise term that is proportional to the distance between the current iterate and the control point.
This well-explains the inner-loop convergence and stability properties that SARAH enjoy upon SVRG.
E XPERIMENTS
In this section, we design a set of experiments to investigate the following questions:
• Does the performance of SARAPO matches the performance of SVRPO or even better?
Which algorithm trains faster and achieves better mean episode reward?
• What happens in the inner loop?
Does the variance reduction methods reaches a better optimum than gradient descent?
• What is the actual KL divergence between the updated policy and the old policy?
Our implementation is based on the modular rl .
We choose the commonly used MuJoCo (Todorov et al., 2012) environment on several tasks:
Swimmer, Half-Cheetah, Walker, and Hopper, and we also test the algorithms on two classic continuous control tasks: Pendulum and CartPole.
We compare our implementation of SARAPO with SVRPO and the original TRPO.
The result is shown in
Figure 1.
4. E XPERIMENTAL S ETTINGS
We follow the settings of Xu et al.
(2017) and Schulman et al.
(2015a) to ﬁx part of our parameters.
We set the number of steps per iteration to be 50000 and the discount factor to be 0.995.
We use a gaussian policy with a diagonal covariance matrix, whose mean is parameterized by a multi-layer perceptron (MLP).
For all tasks, we use two hidden layers of size (64 , 64) with tanh activation function.
We ﬁne-tuned other parameters as follows: minibatch size chosen in { 1000 , 5000 } , the number of inner loop iteration in { 20 , 50 } , inner loop max Kullback-Leibler divergence being in [0 . 005 , 0 . 01] , and the Conjugate Gradient damping factor is 0 .
We list the detailed parameters of each experiment in the Appendix.
For comparing fairly, we chose ﬁve different random seeds and plotted our result as the average of the ﬁve runs.
As the number of the outer loop iteration is ﬁxed, the x -axis is proportional to the number of samples we have drawn and the curve corresponds to the sample efﬁciency.
4. C OMPARISON BETWEEN SARAPO, SVRPO AND TRPO
In Figure 1, we compared the learning curve of three algorithms which are SARAPO, SVRPO and TRPO.
On the classic continuous control tasks such as Pendunlum and CartPole which are 2D balancing tasks, SARAPO outperforms other two algorithms.
Speciﬁcally, SARAPO converges faster than SVRPO and TRPO and gets high mean episode reward at the end stage of the training process.
We see that in Half-Cheetah environment which is a 3D continuous-control locomotion task, SARAPO observably performs better than other two algorithms and achieves a high score in 500 epochs.
In Walker and Hopper (3D continuous-control locomotion task), the three algorithms exhibits comparable performance.
SARAPO converges faster in Swimmer but SVRPO reaches a better score.
We make two more tables to get a clearer view of our result.
Table 1 presents the number of iterations needed to cross the threshold of 90 percent of the best score.
In four of the tasks, SARAPO reaches
[Table]
[Table]
the threshold the fastest and in the other two tasks, SVRPO outperforms the other two algorithms.
In Table 2, on HalfCheetah task, SARAPO outperforms other two algorithms.
Table 2 shows the ﬁnal performance of policy which is the average of best reward over ﬁve different seeds.
4. D ETAILS
We perform further experiments to investigate the phenomenon during the training process.
By analyzing on statistics of the small step updates and the outer loop updates respectively, we summarize three meaningful statistics:
The norm of estimated gradients in each small step gives us a glimpse on how the gradient value is changing.
The policy loss improvement is the absolute difference between the objective function value before and after an outer loop update, which intuitively visualize the expected increase in the mean episode reward.
And the KL divergence between outer loop updates illustrates the difference between the old policy and the updated policy.
Larger KL divergence and loss improvement are indicators of a good policy update.
Figure 4.3 incorporates the result of all three statistics of three different tasks.
The ﬁrst row shows the training process of Swimmer, the second row is the Hopper task and the
In the ﬁrst column, we plot the L 2 norm of the estimated gradient of SVRG and SARAH.
Then the second column is the loss function both before and after an outer loop update.
For SARAPO and SVRPO, this statistics is the sum of the increase in each small step update.
We compare the result with the policy loss improvement in TRPO.
From this, we verify our conjecture that both
SARAH and SVRG reach a better optimum than SGD.
Generally, a more signiﬁcant increase in loss corresponds to a higher mean episode rewards.
The third column shows the KL divergence between the old policy and the updated policy.
We observe that although we do not constrain the KL strictly in the line search, the KL of TRPO stays under the given threshold.
In the interval to plot the result, KL constraint is set to be 0.01, TRPO stays under 0.01, SARAPO stays under 0.02, and SVRPO stays under 0.03.
This is a good illustration that KL divergence won’t step two far away after several inner loop updates.
The variance reduced gradient ﬁnds a better solution slightly outside the KL range.
C ONCLUSION
In this work, we propose a sample-efﬁcient and variance reduction method for reinforcement learning called Stochastic Recursive Gradient Policy Optimization (SARAPO) algorithm.
We hybrid the recently proposed SARAH algorithm which is a kind of variance reduction method with TRPO, which is the state-of-the-art model-free policy gradient methods in continuous control tasks together.
We also provide theory which presents our algorithm enjoys the better properties by using ordinary and stochastic differential equations.
Our experiments show its advantage over existing policy optimization methods such as TRPO and SVRPO.
We hope this work can inspire future works and ultimately achieve higher performance in the reinforcement learning tasks.
A H YPERPARAMETERS
Here, we present the hyper-parameters that get the best results for all tasks.
We tune hyper-parameter by using grid search and record the hyper-parameter in Table 3 and Table 4.
Speciﬁcally, the minibatch size is searched in { 1000 , 5000 } , the inner loop iterations is in { 20 , 50 } , the max KL is between 0 .
005 and 0 .
01 , and the Conjugate Gradient (CG) damping factor is 0 .
[Table]
[Table]
B T HE M EAN E PISODE R EWARD OF THE EXPERIMENTS IN F IGURE 4.
We provided the learning curve for the experiments in Figure 4.3.
We only plot the statistics using a single run.
We see that for this random seed, SARAPO achieve best performance on Swimmer and HalfCheetah tasks.
SARAPO and SVRPO get similar performance on Walk2d task and both outperform than TRPO except Hopper task.
C E XPERIMENT ON AN ADAPTIVE PENALTY VARIANT OF THE PROXIMAL POLICY MAPPING
To ﬁnd out more about applying stochastic gradient in policy gradient algorithms and its variants.
We also proposed an algorithm on the Proximal Policy Gradient (PPO).
We use the adaptive KL penalty version of PPO because the clipped objective function is not differentiable.
We list our algorithm that combined SVRG and PPO in our appendix.
And did experiments on Swimmer to compare the performance of SARAPPO and SVRPPO.
Both SARAPPO and SVRPPO reaches a better reward than PPO.