Improving Actor-Critic Reinforcement Learning via Hamiltonian Policy
Abstract
Approximating optimal policies in reinforcement learning (RL) is often necessary in many real-world scenarios, which is termed as policy optimization.
By viewing the reinforcement learning from the perspective of variational inference (VI), the policy network is trained to obtain the approximate posterior of actions given the optimality criteria.
However, in practice, the policy optimization may lead to suboptimal policy estimates due to the amortization gap and insufﬁcient exploration.
In this work, inspired by the previous use of Hamiltonian Monte Carlo (HMC) in
VI, we propose to integrate policy optimization with HMC.
As such we choose evolving actions from the base policy according to HMC.
First, HMC can improve the policy distribution to better approximate the posterior and hence reduces the amortization gap.
Second, HMC can also guide the exploration more to the regions with higher action values, enhancing the exploration efﬁciency.
Instead of directly applying HMC into RL, we propose a new leapfrog operator to simulate the Hamiltonian dynamics.
With comprehensive empirical experiments on continuous control baselines, including MuJoCo, PyBullet Roboschool and DeepMind Control Suite, we show that the proposed approach is a data-efﬁcient, and an easy-toimplement improvement over previous policy optimization methods.
Besides, the proposed approach can also outperform previous methods on DeepMind Control Suite, which has image-based high-dimensional observation space.
Introduction
Reinforcement learning (RL) algorithms involve policy evaluation and policy optimization [38].
In continuous control the policy optimization can be challenging due to instability and poor asymptotic performance.
In deep RL, where policies over continuous actions are often realized by deep neural networks, such issues are typically tackled by the regularization based on past learned policies [35, 36] or by maximizing the policy entropy [27, 11].
These techniques essentially solve RL in the framework of variational inference (VI) [22], using optimization to infer a policy that yields high expected return while satisfying prior policy constraints.
However, from this perspective, when used with entropy or KL regularization, policy networks essentially perform amortized optimization [12, 22].
It means that, many deep RL algorithms, such as soft actor-critic (SAC) [14], optimize a network to directly output the parameters of policy distribution, approximating the posterior given the input state and optimality.
While these schemes have improved the efﬁciency of variational inference as encoder networks [19, 31, 26], the learned policy distribution can be sub-optimal and far away from the posterior, due to the insufﬁcient expressivity of the policy network [8, 17].
This suboptimality is typically deﬁned as the amortization gap [8], resulting into a gap in the RL objective.
The HMC has been used in VI in many previous works [3, 48].
Starting Hamiltonian dynamics with initial values of variables sampled from an optimized variational distribution, we can break the expressive limitation of the variational distribution and hence ﬁll in the amortization gap, leveraging the advantages of both VI and HMC [33, 46].
In this work, we propose to use Hamiltonian dynamics (HD) to evolve the actions sampled from the policy network, so as to better approximate the posterior, i.e., the action with highest Q value, and make the exploration to be more directionally informed.
We call this new policy integrated with HD as Hamiltonian policy .
The conventional Gaussian policies in SAC [14] are directionally uninformed, where actions are sampled in arbitrary directions with equal probabilities, wasting a lot of samples.
Moreover the expressivity of Gaussian policies is quite limited.
However, the gradient information in Hamiltonian policy can make the exploration more directionally informed, avoiding sampling too many actions in opposite directions.
Moreover, the randomness of momentum vectors therein can help sampled actions to jump over the local optima and make the agent to explore more unknown parts in the state space.
The proposed leapfrog operator in
Hamiltonian policy, which generalizes HMC by neural networks, can also increase the expressivity of the base policy network and adapt to the target distribution deﬁned by Q function which is changing during the learning process.
In this work we only consider stochastic policy, where the policy optimization is regularized by the policy entropy or KL distance [38, 14].
Hence the density of policy distribution should be tractable to compute.
However, in most MCMC algorithms such as SGLD [45], the density of output variables is intractable or very expensive to compute, which cannot be used in policy optimization here.
Because of the tractable Jacobian determinant, the density of output distribution of HMC is tractable to compute [29].
Hence the policy entropy of the Hamiltonian policy can be easily regularized in the objective, even though the policy distribution evolved by HD is non-Gaussian with complex shape.
Using empirical experiments, we evaluated the proposed method across a variety of benchmark continuous control tasks such as OpenAI Gym using the MuJoCo simulator [42], the realistic Bullet
Roboschool tasks [6], and the DeepMind Control Suite [40] with high-dimensional observations.
We show that the proposed method improves upon a already high performing method such as SAC and some other related methods, achieving better both convergence rate and performance.
Additionally, we also conduct ablation study and sensitivity analysis, which shows the effects of the proposed neural architecture and hyper-parameters.
Moreover, we also empirically evaluate the shape of the policy distribution produced by the Hamiltonian policy, verifying its non-Gaussianity and expressivity.
Preliminary
In this section, we are going to introduce reinforcement learning (RL) as an Markov Decision Process (MDP), and formulate the problem in the framework of variational inference.
Then we brieﬂy review the Soft Actor-Critic (SAC) [14] and Hamiltonian Monte Carlo (HMC) [29] as building blocks of the proposed method.
2.1 Markov Decision Process
We investigate Markov decision processes (MDP), where s t ∈ S and a t ∈
A are the state and action at time step t , with the corresponding reward r t = r
( s t , a t ) .
The state transition of the environment is governed by s t
+1 ∼ p env ( s t +1 | s t , a t ) , and the action is sampled from the policy distribution, given by the policy network π θ ( a t | s t ) with parameters θ .
The discounted sum of rewards is denoted as
R ( τ ) =
(cid:80) t γ t r t , where γ ∈ (0 , 1] is the discounted factor, and τ = ( s 1 , a 1 , . . . ) is a trajectory.
Thus, the distribution over the trajectory is
[Equation]
where the initial state is drawn from the distribution ρ ( s 1 ) .
The objective of RL is to maximize the expected discounted return E p ( τ ) [ R ( τ )] .
At a given time step t , one can optimize this objective by estimating the accumulated future returns in the summation using an action-value network [26, 14], denoted as Q π ( s, a ) in terms of a policy π .
2.2 Reinforcement Learning via Variational Inference
Recently a surge of works have formulated reinforcement learning and control as probabilistic inference [9, 43, 41, 2, 22].
In these works, the agent-environment interaction process is formulated as a probabilistic graphical model, then reward maximization is converted into maximum marginal likelihood estimation, where the policy resulting the maximal reward is learned via probabilistic inference.
This conversion is accomplished by introducing one or more binary observed variables O , whose probability conditioned on the trajectory can be expressed as
[Equation]
where α is the temperature hyper-parameter.
By referring variables O as optimality [22], our target is to learn the policy π θ which can produce actions maximizing the likelihood of optimality.
However evaluating this likelihood, i.e., p ( O = 1) =
(cid:82) p ( O = 1 | τ ) p ( τ ) dτ , needs the averaging over all the trajectories, which is computationally intractable especially in high-dimensions.
Hence variational inference (VI) is adopted to lower bound the objective, where a variational distribution q ( τ |O ) is learned to approximate the posterior of trajectory given the optimality, i.e.,
[Equation]
Since we focus on model-free RL here and the environment dynamics is unknown, what we can learn is the posterior of actions given the input state and optimality, i.e., q ( a t | s t , O ) , to maximize the evidence lower bound (ELBO) of the objective log p ( O = 1) , shown as below,
[Equation]
Simplifying this ELBO with (1) and (3), we can get the objective of policy optimization as below
[Equation]
Speciﬁcally, at time step t , this objective can be written as
[Equation]
where D KL ( ·(cid:107)· ) denotes the KL divergence.
Hence, with π θ as action prior, policy optimization in the framework of VI [14, 22] is to ﬁnd optimal q maximizing the objective J ( q, θ ) in (6).
2.3 Soft Actor-Critic
Soft Actor-Critic (SAC) [14] is a state-of-art off-policy RL algorithm widely achieving success in many applications, especially in robotic problems with continuous actions and states.
It updates the policy using gradient descent, minimizing the KL divergence between the policy and the Boltzmann distribution using the Q-function as its negative energy function.
SAC can also be formulated from the perspective of variational inference.
When using uniform distribution U =
( − 1 , 1) as the action prior π θ in (6), the objective of SAC can be formulated as the state-action value function regularized with a maximum entropy term, shown as below,
[Equation]
Here ρ q is the state distribution induced by policy q , and α is the temperature parameter same as (2).
In this work, we are going to build the proposed method upon SAC.
2.4 Hamiltonian Monte Carlo
Hamiltonian Monte Carlo (HMC) is a popular MCMC method for generating sequence of samples, which converge to being distributed according to the target distribution [29].
Inspired by physics, the key idea of HMC is to propose new points by simulating the dynamics of a frictionless particle on a potential energy landscape U ( x ) induced by a desired target distribution p ( x ) , where p ( x ) ∝ exp( − U ( x )) .
This simulation is done in the formulation of Hamiltonian dynamics (HD).
Speciﬁcally, HD is a reformulation of physical dynamics, and the state of the physical system can be described by a pair ( x, v ) of d -dimensional vectors, where x is the position vector and v is the momentum vector.
The dynamics of the system over time, i.e., the HD, is described by the
Hamiltonian equations:
[Equation]
where H ( x, v ) is the Hamiltonian of the system, deﬁned as the total energy of the system.
In the physical context of HMC, the motion of the frictionless particle is governed by the potential energy U ( x ) and kinetic energy K ( v ) .
Since the Hamiltonian is the total energy here, we have
H ( x, v ) =
U ( x ) + K ( v ) , which is independent of time step due to the conservation of energy.
Since the kinetic energy can be described as K ( v ) =
βv T v/ 2 where β is the mass of the particle, and the momentum vector is distributed as p ( v ) ∝ exp( − βv T v/ 2) [46].
The analytic solutions of HD (8) can determine three important properties of HMC algorithm, i.e., reversibility, volume preservation and Hamiltonian conservation .
The reversibility means that the mapping T s from the state ( x t , v t ) at time t to some future state at time t + s ( s > 0) is one-to-one and reversible.
The volume preservation refers that the transformation based on HD conserve the volume in state space, i.e., applying T s to some region results in another region with the same volume.
Finally, the Hamiltonian H ( x, v ) keeps constant with time, i.e., dH/dt = 0 , which is called Hamiltonian conservation.
The HD described in (8) is typically simulated by the leapfrog operator [21, 29], of which the single time step can be described as
[Equation]
which transforms ( x, v ) to ( x (cid:48) , v (cid:48) )
We can see that transformations in (9) are all volume-preserving shear transformations, where in every step only one of variables ( x or v ) changes, by an amount determined by the other one.
Hence the Jacobian determinant of (9) is simply 1 and the density of transformed distribution p ( x (cid:48) , v (cid:48) ) is tractable to compute.
Related Work
There have been a lot of previous works on improve policy optimization in recent years.
To optimize the Q-value estimator with an iterative derivative-free optimizer, Qt-opt [16] uses the cross-entropy method (CEM) [32] to train robots to grasp things.
To improve the model-predictive control, CEM and related methods are also used in model-based RL [28, 4, 30, 34].
However, there are less recent works on gradient-based policy optimization [15, 37, 1, 24].
They are speciﬁcally designed for model-based RL [15, 37, 1].
Normalizing ﬂow [13, 39, 25] is another method to improve the policy optimization, by increasing the expressivity of the policy network.
But none of them include gradient information, so that exploration is not sufﬁcient in some environments.
Another signiﬁcant challenge with this approach is the Jacobian determinant in the objective, which is generally expensive to compute.
Previous methods make the Jacobian determinant easy-to-evaluate at the sacriﬁce of the expressivity of the transformation, where the determinant only depends on the diagonal [20, 39, 25], limiting the exploration in the RL process.
Another approach is to apply iterative amortization in policy optimization, which uses gradients of
Q function to iteratively update the parameters of the policy distribution [24].
However, especially when the estimation bias of Q functions is signiﬁcant [5], directly using gradients to improve policy distributions without additional randomness may make the policy search stuck at local optima which limits the exploration, and the policy distribution therein is still Gaussian.
That is why the performance in high-dimensional environments is not satisfactory [24].
Finally, one more related work is Optimistic Actor Critic (OAC) [5] using gradient of value function to update actions in exploration.
However, their updated policy distribution is still Gaussian without enough expressivity.
Further one-step update with gradient is not enough to have signiﬁcant performance improvement.
Methodology
In this work we only consider stochastic policy, where the policy optimization is regularized by the policy entropy or KL distance [38, 14].
Hence the density of policy distribution should be tractable to compute.
However, in most MCMC algorithms such as SGLD [45], the proposal distribution is intractable or very expensive to compute, which cannot be used in policy optimization of stochastic policy.
Because of the property of the tractable Jacobian determinant, the proposal distribution of HMC is tractable to compute [29], and Hamiltonian dynamics (HD) can be applied into policy optimization.
In this work, we propose to use HD to evolve actions produced by the base policy network, so as to make the policy distribution better approximate the maximizer of the ELBO (5), i.e., the posterior of action given the state and optimality.
Here we focus on the time-inhomogeneous HD [29, 3].
This method uses reverse kernels which are optimal for reducing variance of the likelihood estimators and allows for simple calculation of the approximate posteriors.
In this section, we ﬁrst formulate the proposed policy optimization method in the framework of variational inference (VI), and then propose a new leapfrog operator which is generalized by additional neural networks to quickly adapt to the changes of the target distribution during the learning.
Additional considerations in implementation are also introduced.
4.1 Hamiltonian Policy Optimization
We call the proposed method as Hamiltonian policy optimization (HPO).
Similar as HMC, we introduce momentum vector ρ to pair with the action a in dimension d a , extending the Markov chain to work in a state space ( a, ρ ) ∈ R d a × R d a .
Speciﬁcally, the momentum vector ρ has Gaussian prior
N ( ρ | 0 , β − 1 0 I ) , and the action a follows the uniform prior U ( − 1 , 1) d a , where β 0 is a hyper-parameter.
Therefore, in the framework of VI, the target distribution in policy optimization, i.e., unnormalized posterior of action and momentum vector, can be written as
[Equation]
Following the formulation in Section 2.4, we can have the potential function of the target U
θ ( s, a ) :
= − Q π θ ( s, a ) /α and the kinetic energy for the momentum vector K ( ρ ) :=
β 0 2 ρ T ρ .
Essentially the variational distribution is the policy distribution being updated together with the momentum prior, i.e., π θ ( a | s ) N ( ρ | 0 , β − 1 0 I ) .
The core idea of HPO is to improve the variational distribution via HD, where HD is simulated by steps of deterministic transitions (leapfrog), so that we can make the policy distribution better approximate the posterior.
Essentially we evolve the action and momentum vector via deterministic transitions q [Equation] [Equation]
[Equation] ) , where θ denotes the parameters of policy π θ in (10) and h represents trainable parameters in simulating HD, so that we can have
[Equation]
) where ( a 0 , ρ 0 ) ∼ π θ ( ·| s ) N (0 , β − 1 0 I ) , and { Φ kθ,h } Kk =1 deﬁne diffeomorphisms corresponding to a time-discretized and inhomogeneous Hamiltonian dynamics [3].
Different from normalizing ﬂow policies in previous works [39, 44, 25], the gradient information of Q-function is incorporated, and the log density of output actions is easy to compute due to the tractable Jabcobian determinant, facilitating the policy entropy regularization in policy optimization.
Every transition Φ kθ,h here consists of two steps: a leapfrog step, which discretizes the Hamiltonian dynamics, and a tempering step, which adds inhomogeneity to the dynamics and allows us to explore isolated modes of the target distribution [3, 29].
The leapfrog transforming ( a, ρ ) to ( a (cid:48) , ρ (cid:48) ) can be described as the following transformations:
[Equation]
) where ∇ is the differentiation taken with respect to a , and step size (cid:15) ∈ R .
We can see that the leapfrog (12) still has unit Jacobian.
For the tempering step, the momentum output ρ (cid:48) of each leapfrog step is multiplied by a scalar α k ∈ (0 , 1) for k = 1 , . . .
, K .
Since the target distribution deﬁned by Q-function is varying during the learning process and Q-function may have estimation bias, we adopt ﬁxed tempering scheme proposed in [3], which can keep the learning process numerically stable.
Speciﬁcally α k = (cid:112)
β
k − 1 /β k and β k is a deterministic function of β 0 shown in (17), where 0 < β 0 ≤ β 1 ≤ . . .
≤ β K
= 1 .
Then we can easily have that the Jacobian of every transition Φ kθ,h is given by | det ∇ Φ kθ,h ( a k , ρ k ) | =
α d a k = ( β k − 1 /β k ) d a / 2 .
Therefore, the joint distribution of action and momentum variables at K -th step of leapfrog Φ Kθ,h can be expressed as
[Equation]
Hence the density of output action and momentum vector is tractable to compute, facilitating the policy entropy regularization in SAC-style algorithms.
Since HPO is in the formulation of VI, the policy optimization objective is just the ELBO (4).
According to foundations of VI, the ELBO can be written as the difference between the log target and log variational distribution, i.e.,
[Equation]
(14) where ( a K , ρ K ) :
= f K HMC ( a 0 , ρ 0 ; θ, h ) in (11), and parameters to be trained consist of policy network parameters θ and parameters in HMC h .
Finally, combining (13) and (14) together and ignoring terms not related with θ and h , the objective of HPO, i.e., the expectation of EBLO over all the visited states, can be written as
[Equation]
where ρ π θ is the state distribution induced by the policy π θ , ( a 0 , ρ 0 ) ∼ π θ ( ·| s ) N ( ·| 0 , β − 1 0 I ) and ( a K , ρ K )
=
f K HMC ( a 0 , ρ 0 ; θ, h ) deﬁned in (11).
Speciﬁcally α is the temperature parameter tuned in the same way as SAC [14].
4.2 Theoretical Justiﬁcation
By regarding the joint distribution of action and momentum π θ ( a | s ) N (0 , β − 1 I ) as the variational distribution, we essentially improve it by running HD initialized by ( a 0 , ρ 0 ) ∼ π θ ( a | s ) N (0 , β − 1 I ) , and the stationary distribution of action is the posterior of the target (10) with ρ marginalized which is denoted as ¯ p a,α ( ·| s ) .
Assume we run the leapfrog for k steps, by summing out ρ , the distribution of action is denoted as q ka,θ ( · ) .
It is provable that given state s , the distribution q ka,θ ( · ) is an improvement over the variational distribution π θ ( ·| s ) and it is closer to the posterior of the target (10), i.e., ¯ p a,α ( ·| s ) , in terms of KL divergence [7],
[Equation]
( ·| s ))
Hence it reduces the amortization gap .
According to [29], as the step k → ∞ , the distribution of action q ka,θ ( · ) will tend to the posterior ¯ p a,α ( ·| s ) in terms of KL divergence.
We empirically ﬁnd that using K = 2 , 3 , 4 leapfrog steps is enough to yield satisfactory improvement.
4.3 Proposed Leapfrog Operator
In the practical policy optimization, the action value given by the Q network may have estimation bias, and directly incorporating gradient information like (12) may mislead the learning to wrong regions in the policy space, leading to suboptimal policy estimate.
Since the Q value is changing in the learning process, how to make the policy distribution quickly adapt to updated Q networks in a numerically stable way is the desiderata here.
The key is to improve policy expressivity without [Figure] [Equation] [Figure]
[Equation]
[Figure] [Equation] scarifying numerical stability. The previous work using neural network to generalize HMC [23] cannot be applied here, because based on our empirical study the direction variable, binary mask and exp operation therein can make the policy optimization unstable, degrading the RL performance.
Therefore, we use gating-based operation to generalize the conventional leapfrog operator(12), and propose a new leapfrog operator, which integrates gradient information via both explicit and implicit approaches.
The explicit approach is to directly use the primitive gradient same as (12), whereas the implicit approach is to use an MLP T h to transform the primitive gradient, state and action together.
Then the gradient information from both explicit and implicit approaches are combined via a gate σ h .
The motivation behind is to improve the policy expressivity by MLP T h and control the numerical stability by gate σ h , making the policy distribution quickly adapt to the changes of Q networks during the learning process.
According to empirical experiments we ﬁnd that the gradient normalized by its l 2 norm can make the model perform best.
The inputs of T h and σ h include normalized gradients, action and state, where the state is optional and can be ignored in some environments.
Therefore, the proposed leapfrog operation, transforming from ( a, ρ ) to ( a (cid:48) , ρ (cid:48) ) , can be described as
[Equation]
where g := ∇ U θ ( s,a ) (cid:107)∇ U θ ( s,a
[Equation] , g (cid:48) :
= ∇ U θ ( s,a (cid:48) ) ∇ U θ ( s,a (cid:48) ) and a (cid:48)
= a + (cid:15) (cid:12) ˜ ρ .
Speciﬁcally, T h and σ h have different parameters contained in h , but they should have the same architecture based on empirical study.
We ﬁnd that the architecture of T h and σ h should be simple, having one hidden layer with 32 or 64 units.
The output activations of T h and σ h are sigmoid and tanh respectively.
The transformations in (16) are visualized in Figure 2.
Obviously the proposed leapfrog operator (16) still keeps the properties of reversibility and tractable Jacobian determinant.
4.4 Implementation Considerations
In implementation, we build the Hamiltonian dynamics (HD) simulated by leapfrog steps on top of the policy network in both exploration and policy optimization.
This new policy is termed as
Hamiltonian policy , whose working mechanism is shown in Figure 1.
The randomness of momentum vector ρ can boost the exploration and improve the generalization of the policy trained in the policy optimization.
The neural networks T h and σ h in the proposed leapfrog operator can increase the expressivity of policy distribution, veriﬁed in the experiment section.
In exploration, the policy distribution is expanded towards regions with higher Q values, which makes the exploration more directed and hence improves the sample efﬁciency.
Speciﬁcally, we ﬁnd that the HD has different effects on performance in the exploration and policy optimization, which is analyzed in the experiment section.
Using different momentum variances β 0 for exploration and policy optimization, denoted as β expand β tr 0 respectively, can yield the best empirical performance.
And in general we set β exp> β tr 0 , since the exploration needs more randomness than policy optimization.
In order to achieve the best empirical performance, we make some speciﬁc changes in the implementation of RL algorithms.
First, it is better to omit the last term of the objective (15), i.e., α 2 ρ TK ρ K , in the practical implementation of policy optimization.
Second, different from previous works on
HMC [33, 46, 23], the Metropolis-Hastings accept/reject step is omitted here.
That is because the Q values are dramatically varying during the training of RL.
If accept/reject step was applied here, the acceptance rate could be very low, which could limit the exploration and hence degrade the learning performance signiﬁcantly.
The chart of the proposed algorithm is shown in Algorithm 1, and HD simulated by leapfrog steps is described in Algorithm 2.
1 Sample ρ 0 ∼ N[-(0 , β − 1 
 I ; 2 for k =-]
I do
Obtain ˜ ρ k by the ﬁrst equation in (
Update a k − 1 + (cid:15) (cid:12) ˜
ρ k ;
Obtain ρ k by the second equation in
Compute β k via
[Equation]
Set ρ k
[Equation]
ρ k · (cid:112) [Equation]
[Equation]
7 end
Output: a K , ρ K
[Figure]
Experiment
In experiments, the proposed method is built on top of soft actor critic (SAC), having the same setup as [14], denoted as "SAC-HPO".
In this section, we present the empirical evaluation of the proposed method from many perspectives.
First, we compare SAC-HPO against the primitive SAC on a set of various continuous control tasks, including OpenAI Gym MuJoCo [42], the realistic Roboschool
PyBullet suit [6] and DeepMind Control Suite [40].
Then we conduct ablation study on the proposed leapfrog operator and the speciﬁc effects of Hamiltonian dynamics (HD) simulated by the proposed leapfrog operator on the exploration and policy optimization analysis.
Further the sensitivity analysis of hyper-parameters is also presented, which includes the number of leapfrog steps and variances of momentum vectors in exploration and policy optimization.
Finally, we analyze the shape of policy distribution evolved by HD, verifying its non-Gaussianity and multi-modality.
5.1 Continuous Control Tasks 5.1.1 OpenAI Gym MuJoCo Benchmarks
First, the proposed method, SAC-HPO, is compared against the SAC on ﬁve continuous control tasks from the MuJoCo suite, shown in Figure 3.
All result curves show the performance in evaluation, which, in the case of both methods, is equivalent to setting the noise (variance) to 0 at the policy network output.
Evaluation happens every 10,000 steps, where each evaluation score (accumulated rewards in one episode) is averaged over 10 runs.
The values reported in the plots are smoothed by exponential moving averaging (EMA) with a window size of 5, equivalent to averaging every 50, steps to improve comparability.
The best obtained accumulated rewards reported in the tables are primitive values without being smoothed.
We use uniform action prior p ( a | s ) ∈ U ( − 1 , 1) for both SAC-HPO and SAC.
Both methods use the same architecture for Q networks, hyper-parameters such as batch size and learning rate, and tuning scheme for the temperature α .
The policy network in SAC consists of two feed-forward hidden layers with 256 units.
In SAC-HPO, we tried two kinds of policy networks, having one or two hidden layers, with details shown in Table 1.
The output of policy network in SAC-HPO is further improved by leapfrog operations (16) for K ∈ { 2 , 3 , 4 } steps, where the neural networks σ h , T h are chosen to be small, consisting of one feed-forward hidden layer with 32 or 64 units.
The detailed architectures of σ h , T h used throughout the paper are shown in Table 2.
And the output activation functions for T h and σ h are tanh and sigmoid, respectively.
Since the number of parameters of σ h , T h networks is only about 1% of that of the base policy network, the performance improvement of SAC-HPO does not come from the increase of trainable parameters, i.e., overparameterization.
The critic (Q) networks follows the same architecture as [14], i.e., two-layer fully connected neural networks with 256 units in each layer, and two Q networks are implemented and trained by bootstrapping.
All networks are updated by Adam optimizer [18] with the learning rate of 3e-4.
Both SAC-HPO and SAC have same choices for α , the temperature for entropy regularization.
And the momentum random variables have different variance for training β tr and exploration β exp .
The update rate (cid:15) in the leapfrog operator (16) is also carefully tuned for every task.
All the hyperparameters for simulating HD are shown in Table 3.
All the other important hyperparameters for running these experiments can be found in Table 4, where l represents the number of hidden layers in the policy network, h is the number of units in the hidden layer in T h and σ h networks, m denotes the batch size in training, and B represents the replay buffer.
[Table]
[Table]
[Table]
[Table]
The learning performance of SAC-HPO and SAC are displayed in Figure 3.
The Table ?? presents the best observed accumulated reward in evaluation.
We can see that the SAC-HPO outperforms SAC in terms of both convergence rate and performance, achieving better sample complexity and higher accumulated rewards.
There are primarily two reasons for that.
First, the HD incorporating gradient information of Q networks reduces the amortization gap in policy optimization and hence breaks the expressivity limitation of the base policy network.
Second, the HD simulated by the proposed leapfrog steps boost the exploration by the randomness of momentum vectors ρ and also inform the agent directions to regions with higher Q values.
We can see that SAC-HPO has smaller regions of standard deviation for every task in Figure [Table] and it shows that SAC-HPO achieves better learning stability than SAC.
That is because the randomness of momentum vectors ρ in HD can make
the
[Figure]
[Table]
policy optimization (training of the policy network) cover more state-action pairs, improving the generalization capability of the policy In particular, SAC-HPO can perform better than SAC on difﬁcult tasks with high-dimensional states, including Ant-v2 and Humanoid-v2, even though previous methods such as iterative amortization policy optimization [24] fail in these tasks. We can also see that HalfCheetah-v2 is exceptional, where the momentum variance for policy optimization (training) ρ tr needs more randomness than that for exploration ρ exp .
That is because in HalfCheetah-v2 exploration is less important than policy optimization for performance improvement.
5.1.2 Roboschool PyBullet Benchmarks
In addition to the classical tasks in MuJoCo suit, we also evaluate the proposed method, SAC-HPO, in the PyBullet implementation of Roboschool tasks [6].
The Bullet library is one of the most realistic collision detection and multi-physics simulation engines available up to now, which is widely used in many robotic tasks [10].
In this section, we introduce another baseline, SAC-NF, which builds a normalizing ﬂow on top of the base policy network [39, 25].
According to the previous evaluations [25], we adopt the radial normalizing ﬂow.
Although authors did not release their codes, we try our best to implement to match their reported results.
We compare SAC-HPO with SAC and SAC-NF in three representative
PyBullet environments, including HumanoidPyBulletEnv-v0, HumanoidFlagrunPyBulletEnv-v0 and
HumanoidFlagrunHarderPyBulletEnv-v0.
The learning performance is shown in Figure 4, where the evaluation scheme and smoothing method are same as the previous section.
And the best observed accumulated rewards in evaluations are reported in Table 5.1.2.
We can see that SAC-HPO performs best in terms of both convergence rate and obtained rewards.
All these environments are more realistic and have high-dimensional state and action spaces, where the exploration capability is important.
However, although the normalizing ﬂows in method SAC-NF improves the expressivity of the policy distribution, in SAC-HPO the gradient information can make the exploration more directionally informed and the expressivity can be also improved by momentum vector ρ and neural networks T h , σ h in the proposed leapfrog operator.
5.1.3 DeepMind Control Suite
We also conduct experiments on DeepMind Control Suite [40], which have high-dimensional observations.
The baseline is SAC-AE model [47], where the input to the policy and critic is the latent representation learned by an autoencoder.
We apply HD on top of the policy in SAC-AE, termed as SAC-AE-HPO.
And the gradients of critics are conditioned on the latent representation, [Figure] rather than raw observations.
For all three environments, we select the hyper-parameters of HPO as β tr ∈ { 0 . 1 , 0 . 15 , 0 . 2 } , β exp ∈ { 1 . 0 , 1 . 5 , 2 . 0 } , (cid:15) = 0 .
And the hyper-parameters of the baseline is set same as [47], except that the replay buffer capacity is 100000 .
The screenshots and performance comparison are shown in Figure 5.
We can see that adding HD into SAC-AE can consistently improve its learning speed and performance.
Since the gradient of critic is only conditioned on the low-dimensional latent representation, the leapfrog steps do not have signiﬁcant computation overhead.
5.2 Analysis
In this section, we are going to conduct ablation study, sensitivity analysis and investigate the shape of the policy distributions evolved by HD.
5.2.1 Ablation Study
In ablation study, we ﬁrst verify the effect of the proposed leapfrog operator (16), compared with the conventional leapfrog in (12).
Then we study the difference between the effects of HD in exploration and policy optimization, where the policy optimization refers to the training step of the policy network and neural networks T h , σ h in HD.
Speciﬁcally, we introduce three baselines adopting different leapfrog operators in exploration and policy optimization, which are summarized in Table 6.
Here "Gaussian Policy" means that the same policy network as SAC is directly used without evolving actions by simulating HD.
"Conventional Leapfrog" refers to that actions are evolved by HD simulated by conventional leapfrog steps (12), and "Proposed Leapfrog" means that HD evolving actions is simulated by the proposed leapfrog steps (16).
We cannot use HD simulated by the proposed leapfrog only in exploration, since the neural networks T h , σ h in (16) need to be trained.
The SAC-HPO and baselines are evaluated over Ant-v2 and HumanoidPyBulletEnv-v0, and learning curves are shown in Figure 6, where we use the same hyper-parameters as Section 5.1.1 and 5.1.2 for every task here.
It can be seen that in Figure 6(a) and 6(c), SAC-HPO outperforms the baseline-1 in terms of both performance and learning stability, showing the advantage of the proposed leapfrog operator over the conventional one.
In Figure 6(b) and 6(d), the baseline-1 outperforms both baseline2 and baseline-3, showing the effects of HD in both exploration and policy optimization. Further, we Exploration Policy Optimization
SAC-HPO
Proposed Leapfrog
Proposed Leapfrog BaselineConventional Leapfrog Conventional Leapfrog BaselineConventional Leapfrog
Gaussian Policy
BaselineGaussian Policy
Proposed Leapfrog
SAC
Gaussian Policy
Gaussian Policy
[Figure]
can see that the improvement of baseline-2 over SAC is higher that of baseline-3, meaning that using
HD in exploration can have more performance improvement more than that in policy optimization.
5.2.2 Sensitivity Analysis
Here we conducted sensitivity analysis on three important hyper-parameters.
The ﬁrst is the number of leapfrog steps in simulating HD, i.e., K = 1 , 2 , 3 , shown in Figure 7(a).
We can see that in
Hopper-v2, the cases of K = 2 and K = 3 perform similar, but much better than that of K = 1 , showing that it is not meaningful to have large K .
The second parameter analyzed here is the variance of momentum vector ρ in exploration, i.e., β exp 0 − 1 = 0 .
5 , 1 . 0 , 1 . 5 , shown in Figure 7(b).
We [Figure] [Figure] can see that the performance is sensitive to the choice of β exp , and not the highest choice leads to best performance.
That is because although ρ with larger variance can boost the exploration, too much variance of ρ may make the sampled state-action pairs deviate too much from the optimal path.
In Figure 7(c), we analyze three different the variance for momentum vector ρ in training, i.e., β tr 0 − 1 = 0 .
However, we can see that the performance is not sensitive to β tr 0 .
According to more evaluations, larger β tr 0 can not lead to better performance, since when β tr 0 = 1 , the performance degrades signiﬁcantly.
5.2.3 Shape of Policy Distribution
We visualize the policy distributions at different environmental steps of Ant-v2 in Figure 8, where x and y axes represent two different dimensions of sampled actions in exploration.
Speciﬁcally, the red dots represent 1000 actions sampled from the policy distribution produced by HD simulated by leapfrog steps.
For comparison the blue dots represent 1000 actions sampled from the Gaussian distribution at the output of base policy network.
The contour of Q value is shown as background for reference.
From Figure 8 we can see that HD can evolve actions from the base policy to regions with higher Q values.
Compared with actions sampled from the base policy, those evolved by HD can be non-Gaussian and have larger variance with much larger effective support.
Besides, there are still some actions evolved to regions with similar or lower Q values, reaching a reasonable trade-off of exploration and exploitation.
That is why the exploration can be boosted by HPO and the performance can be improved.
Conclusion
In this work, we propose to integrate policy optimization with HMC, evolving actions from the base policy network by Hamiltonian dynamics simulated leapfrog steps.
In order to adapt to the changes of the target distribution deﬁned by Q function, we propose a new leapfrog operator which generalizes HMC by neural networks.
The proposed method can reduce the amortization gap in policy optimization and make the exploration more directionally informed.
In empirical experiments, the proposed method can outperform baselines in terms of both convergence rate and performance.
In the future, we are going to apply the proposed method in the real-world robotic tasks