Improving Actor-Critic Reinforcement Learning via Hamiltonian Monte Carlo Method
Abstract
The actor-critic RL is widely used in various robotic control tasks.
By viewing the actor-critic RL from the perspective of variational inference (VI), the policy network is trained to obtain the approximate posterior of actions given the optimality criteria.
However, in practice, the actor-critic RL may yield suboptimal policy estimates due to the amortization gap and insufﬁcient exploration.
In this work, inspired by the previous use of Hamiltonian Monte Carlo (HMC) in VI, we propose to integrate the policy network of actor-critic RL with HMC, which is termed as Hamiltonian
Policy .
As such we propose to evolve actions from the base policy according to
HMC, and our proposed method has many beneﬁts.
First, HMC can improve the policy distribution to better approximate the posterior and hence reduce the amortization gap.
Second, HMC can also guide the exploration more to the regions of action spaces with higher Q values, enhancing the exploration efﬁciency.
Further, instead of directly applying HMC into RL, we propose a new leapfrog operator to simulate the Hamiltonian dynamics.
Finally, in safe RL problems, we ﬁnd that the proposed method can not only improve the achieved return, but also reduce safety constraint violations by discarding potentially unsafe actions.
With comprehensive empirical experiments on continuous control baselines, including MuJoCo and
PyBullet Roboschool, we show that the proposed approach is a data-efﬁcient and easy-to-implement improvement over previous actor-critic methods.
Introduction
In continuous control, actor-critic RL algorithms are widely used in solving practical problems.
However, searching optimal policies can be challenging due to instability and poor asymptotic performance.
Speciﬁcally, most previous methods of actor-critic RL, such as KL regularization [37, 38] and maximum policy entropy [29, 12], essentially solve RL in the framework of variational inference (VI) [24], which infers a policy that yields high expected return while satisfying prior policy constraints.
However, from this perspective, the policy network essentially performs amortized optimization [13, 24].
It means that most actor-critic RL algorithms, such as soft actor-critic (SAC) [16], optimize a network to directly output the parameters of policy distribution which approximate the posterior given the input state and optimality.
While these schemes have improved the efﬁciency of VI by encoder networks [21, 33, 28], the output distribution of learned policy can be sub-optimal and far away from the target posterior, due to the insufﬁcient expressivity of the policy network [9, 19].
This suboptimality is typically deﬁned as the amortization gap [9], resulting into a gap in the RL objective.
The Hamiltonian Monte Carlo (HMC) has been used to improve VI in statistics [3, 46].
In this work, by leveraging the advantages of both VI and HMC [35, 45], we propose to initialize Hamiltonian dynamics (HD) with samples from an optimized variational distribution, so that we can break the expressive limitation of the variational distribution and hence ﬁll in the amortization gap.
Speciﬁcally, we propose to use HD to evolve the actions sampled from the policy network, so as to better approximate the target posterior and sample the actions with higher Q values, improving the efﬁciency of the exploration.
We call this new policy integrated with HD as Hamiltonian policy .
The proposed method offers several beneﬁts.
First, the gradient information in Hamiltonian policy can make the exploration more directionally informed, avoiding sampling too many actions in opposite directions.
Moreover, the randomness of momentum vectors in HD can help sampled actions to jump over the local optima and make the agent to explore more unknown parts of the state space.
Further, the proposed leapfrog operator in Hamiltonian policy, which generalizes HMC via gated neural networks, can also increase the expressivity of the base policy network and adapt to the changing target distribution deﬁned by Q function.
Finally, in safe RL tasks, we ﬁnd that the Hamiltonian policy can not only improve the achieved return by boosting the exploration, but also reduce the safety constraint violations by discarding potentially unsafe actions according to Lyapunov constraints [4, 5].
Using empirical experiments, we evaluated the proposed method across a variety of benchmark continuous control tasks such as OpenAI Gym using the MuJoCo simulator [43] and the realistic
PyBullet Roboschool tasks [8].
We show that the proposed method improves upon representative previous methods such as SAC [14] and SAC with normalizing ﬂow policy [27], achieving both a better convergence rate and expected return.
Additionally, we also empirically verify the advantage of our method in safe RL problems.
In experiments, we conduct ablation study of the proposed leapfrog and sensitivity analysis on hyper-parameters.
Additionally, we also compare the proposed method with iterative amortization policy optimization [26].
And the action distribution of Hamiltonian policy is also visualized to show the improvement of expressivity.
Preliminary
In this section, we are going to introduce reinforcement learning (RL) as an Markov Decision Process (MDP). Then the constrained MDP and the solution based on Lagrangian method are introduced. We also formulate the RL problem in the framework of variational inference.
Finally we brieﬂy review the Soft Actor-Critic (SAC) [16] and Hamiltonian Monte Carlo (HMC) [31] as building blocks of the proposed method.
2.1 Markov Decision Process
We consider Markov decision processes (MDP) as ( S , A , p env , r ) , where s t ∈ S and a t ∈ A are the state and action at time step t , with the corresponding reward r t = r
( s t , a t ) .
The state transition of the environment is governed by s t
+1 ∼ p env ( s t +1 | s t , a t ) , and the action is sampled from the policy distribution, given by the policy network π θ ( a t | s t ) with parameters θ .
The discounted sum of rewards is denoted as R ( τ ) =
(cid:80) t γ t r t , where γ ∈ (0 , 1] is the discounted factor, and τ = ( s 1 , a 1 , . . . ) is a trajectory.
Thus, the distribution over the trajectory is
[Equation]
where the initial state is drawn from the distribution ρ ( s 1 ) .
The objective of RL is to maximize the expected discounted return E p ( τ ) [ R ( τ )] .
At a given time step t , one can optimize this objective by estimating the accumulated future returns in the summation using an action-value network [28, 16], denoted as Q π ( s, a ) in terms of a policy π .
2.2 Constrained MDP and Lagrangian Method
Safety is an important issue in RL problems.
We use constrained MDP (CMDP) to model RL problems in which there are constraints on the cumulative cost.
The CMDP extends MDP by introducing a safety cost function and the associated constrained threshold, which is deﬁned as ( S , A , p env , r, c, d 0 ) where c ( s ) ∈ [0 , C max ] is a state-dependent cost function and d 0 ∈ R > 0 is an upper-bound on the expected cumulative safety cost in one episode.
In addition to Q π , we use another action-value network to approximate the accumulated future safety costs C ( γ ) :
= E [ (cid:80) t γ t c t ] , denoted as Q C,π .
The Lagrangian method is a straightforward method to solve CMDP, by transforming it to a penalty form, i.e., max θ min λ E [ (cid:80) t γ t ( r ( s t , a t ) − λc ( s t ) | π θ , s 0 ] .
The parameters θ and λ are jointly optimized to a saddle-point.
The policy parameters θ are optimized by a policy gradient algorithm, while the multiplier λ is updated iteratively as λ ←− [ λ + η ( J πC − d 0 )]
+ , where J πC is the discounted sum (or average sum) of safety costs in previous episodes.
2.3 Reinforcement Learning via Variational Inference
Recently a surge of works have formulated reinforcement learning and control as probabilistic inference [10, 44, 42, 2, 24].
In these works, the agent-environment interaction process is formulated as a probabilistic graphical model, then reward maximization is converted into maximum marginal likelihood estimation, where the policy resulting the maximal reward is learned via probabilistic inference.
This conversion is accomplished by introducing one or more binary optimality variables
O .
Since calculating the likelihood of optimality O requires intractable integral over all the possible trajectories, variational inference (VI) is adopted to lower bound the objective, where a variational distribution q ( τ |O ) is learned to approximate the posterior of trajectory given the optimality, yielding the evidence lower bound (ELBO) [24].
The ELBO of the likelihood of optimality O can be written as below,
[Equation]
[Equation] where D KL ( ·(cid:107)· ) denotes the KL divergence.
Only model-free RL is considered here.
We can simplify the EBLO in (2) by cancelling the probability of environmental dynamics.
Then we can get the objective of policy optimization as below [24],
[Equation]
Speciﬁcally, at time step t , this objective can be written as
[Equation]
Hence, with π θ as action prior, policy optimization in the framework of VI [16, 24] is to ﬁnd optimal q maximizing the objective J ( q, θ ) in (4).
2.4 Soft Actor-Critic
Soft Actor-Critic (SAC) [16] is a state-of-art off-policy RL algorithm widely used in many applications, especially in robotic problems with continuous actions and states.
SAC can also be formulated from the perspective of variational inference.
When using uniform distribution U =
( − 1 , 1) as the action prior π θ in (4), the objective of SAC can be formulated as the state-action value function regularized with a maximum entropy,
[Equation]
where q is the variational distribution of action.
Here ρ q is the state distribution induced by policy q , and α is the temperature parameter which is introduced to improve the exploration.
In this work, we are going to build the proposed method upon SAC.
The optimal solution of (5) is ¯ p α ( a | s ) ∝ exp( Q ( s, a ) /α ) which is also the target policy distribution.
2.5 Hamiltonian Monte Carlo
Hamiltonian Monte Carlo (HMC) is a popular Markov chain Monte Carlo (MCMC) method for generating sequence of samples, which converge to being distributed according to the target distribution [31].
Inspired by physics, the key idea of HMC is to propose new points by simulating the dynamics of a frictionless particle on a potential energy landscape U ( x ) induced by a desired target distribution p ( x ) , where p ( x ) ∝ exp( − U ( x )) .
This simulation is done in the formulation of Hamiltonian dynamics (HD).
Speciﬁcally, HD is a reformulation of physical dynamics whose states can be described by a pair ( x, v ) of d -dimensional vectors, where x is the position vector and v is the momentum vector.
The dynamics of the system over time, i.e., the HD, is described by the Hamiltonian equations:
[Equation]
where H ( x, v ) is the Hamiltonian of the system, deﬁned as the total energy of the system.
In the physical context of HMC, the motion of the frictionless particle is governed by the potential energy U ( x ) and kinetic energy K ( v ) .
Since the Hamiltonian is the total energy here, we have
H ( x, v ) =
U ( x ) + K ( v ) , which is independent of time step due to the conservation of energy.
The kinetic energy can be described as K ( v ) =
βv T v/ 2 where β is the mass of the particle, and the momentum vector is distributed as p ( v ) ∝ exp( − βv T v/ 2) [45].
The analytic solutions of HD (6) can determine three important properties of HMC algorithm, i.e., reversibility, volume preservation and Hamiltonian conservation .
The reversibility means that the mapping T s from the state ( x t , v t ) at time t to some future state at time t + s ( s > 0) is one-to-one and reversible.
The volume preservation implies that the transformation based on HD conserves the volume in state space, i.e., applying T s to some region results in another region with the same volume.
Finally, the Hamiltonian H ( x, v ) stays constant with time, i.e., dH/dt = 0 , which is called
Hamiltonian conservation.
The HD described in (6) is typically simulated by the leapfrog operator [23, 31], of which the single time step can be described as [Equation]
[Equation]
which transforms ( x, v ) to ( x (cid:48) , v (cid:48) )
We can see that transformations in (7) are all volume-preserving shear transformations, where in every step only one of variables ( x or v ) changes, by an amount determined by the other one.
Hence the Jacobian determinant of (7) is simply 1 and the density of transformed distribution p ( x (cid:48) , v (cid:48) ) is tractable to compute.
Related Work
There have been a lot of previous works on improve policy optimization in recent years.
To optimize the Q-value estimator with an iterative derivative-free optimizer, Qt-opt [18] uses the cross-entropy method (CEM) [34] to train robots to grasp things.
To improve the model-predictive control, CEM and related methods are also used in model-based RL [30, 6, 32, 36].
However, there are less recent works on gradient-based policy optimization [17, 39, 1, 26].
They are speciﬁcally designed for model-based RL [17, 39, 1].
Normalizing ﬂow [14, 40, 27] is another method to improve the policy optimization, by increasing the expressivity of the policy network.
But none of them include gradient information, so that exploration is not sufﬁcient in some environments.
Another signiﬁcant challenge with this approach is the Jacobian determinant in the objective, which is generally expensive to compute.
Previous methods make the Jacobian determinant easy-to-evaluate at the sacriﬁce of the expressivity of the transformation, where the determinant only depends on the diagonal [22, 40, 27], limiting the exploration in the RL process.
Another approach is to apply iterative amortization in policy optimization, which uses gradients of Q function to iteratively update the parameters of the policy distribution [15, 26].
However, especially when the estimation bias of Q functions is signiﬁcant [7], directly using gradients to improve policy distributions without additional randomness may make the policy search stuck at local optima which limits the exploration, and the policy distribution therein is still Gaussian.
That is why methods in [26] cannot outperform SAC in many MuJoCo environments.
Finally, one more related work is Optimistic Actor Critic (OAC) [7] using gradient of value function to update actions in exploration.
However, their updated policy distribution is still Gaussian which has limited expressivity and one-step update with gradient is not enough to have signiﬁcant performance improvement.
Methodology
In this section, we ﬁrst formulate the proposed policy optimization method in the framework of variational inference (VI), and then propose a new leapfrog operator to quickly adapt to the changes of the target distribution during the learning.
Finally implementation considerations and its application into safe RL are introduced.
4.1 Hamiltonian Policy Optimization
We call the method of using Hamiltonian policy into RL as Hamiltonian policy optimization (HPO).
A feature of HPO is that a momentum vector ρ is introduced to pair with the action a in dimension d a , extending the Markov chain to work in a state space ( a, ρ ) ∈ R d a × R d a .
Speciﬁcally, the momentum vector ρ has Gaussian prior N ( ρ | 0 , β − 1 0 I ) , and the action a follows the uniform prior U ( − 1 , 1) d a , where β 0 is a hyper-parameter which determines variance of ρ .
According to the discussion in Section 2.3 and 2.4, the target distribution in HPO, i.e., target posterior of action and momentum vector, can be written as
[Equation]
Therefore, following Section 2.5, the target potential function and momentum kinetic energy can be written as U θ
( s, a ) : = − Q π θ ( s, a ) /α and K ( ρ ) :
=
β 0 2 ρ T ρ .
The core innovation of HPO is to evolve a and ρ via Hamiltonian dynamics (HD) in (6), where HD is approximated by steps of deterministic transitions (leapfrog in (7)), so that the evolved actions would likely better approximate the target posterior.
In Hamiltonian policy, given input state s , the initial action a 0 and momentum vector ρ 0 are sampled as ( a 0 , ρ 0 ) ∼ π θ ( ·| s ) N (0 , β − 1
I ) where π θ is the base policy network.
Then, by using leapfrog operator in (7) to simulate HD, since U θ ( s, a ) := − Q π θ ( s, a ) /α , action and momentum are evolved iteratively by the leapfrog (HMC step) as below,
[Equation]
where ∇ is the differentiation taken with respect to a ,
(cid:15) ∈ R is the learning step size, and k = 0 , . . .
, K − 1 .
Then by evolving initial action a 0 for K leapfrog steps, the action a K is applied to the environment ﬁnally.
The working process is shown in Figure 1(a)
Denote the k -th leapfrog step described above as ( a k , ρ k ) :
=
Φ kθ,h ( a k − 1 , ρ k − 1 ) .
We can see that each leapfrog step still has unit Jacobian.
Therefore, based on the change of variable formula in probability distribution, the joint distribution of action and momentum variables after K steps of leapfrog can be expressed as
[Equation]
where ( a K , ρ K ) are action and momentum evolved by K HMC steps.
Hence the density of output action and momentum vector becomes tractable to compute, facilitating the policy entropy regularization in SAC-style algorithms.
In the framework of VI, the policy optimization objective of HPO is the ELBO (2).
Since ELBO can be written as the difference between the log of target distribution and log of variational distribution [21], we can write the ELBO for HPO as below,
[Equation]
The policy network parameters are denoted as θ and parameters in HMC are denoted as h .
Finally, combining (8), (10) and (11) together and ignoring terms not related with θ and h , the objective of HPO, i.e., the expectation of EBLO over all the visited states, can be written as
[Equation]
where ρ π θ is the state distribution induced by the policy π θ .
Note that α is the temperature parameter tuned in the same way as SAC [16].
And Q π θ is approximated by a target critic network which is not related with θ and is periodically updated in the learning process [14].
s [Figure] 0 ∼ N ( ·| µ ( s ) , Σ( s ))
ρ 0 ∼ N (0 ,β − 1 0 I ) aρaρ 2 ...
HMC HMC π θ (a) Hamiltonian Policy
T h σ h g
T h
σ h g ρ k + × ˜ ρ
k ρ k a k +1 a (cid:48)
k a k 1 − × + (cid:12) (cid:15)+ (cid:12) (cid:15)
+ × 1 − + (cid:12) (cid:15)× +
[Figure]
4.2 Proposed Leapfrog Operator
Since HMC with conventional leapfrog (9) converges and mixes slowly, some past works proposed to use neural networks to generalize HMC [24, 25].
However, since Q networks are changing in RL, the techniques proposed in [24, 25] cannot be used here.
Based on our empirical study, the direction variable, binary mask and exp operation therein [24, 25] can make the policy optimization unstable, degrading the RL performance.
Instead, we propose to use a gating-based mechanism to generalize the conventional leapfrog operator (9) and design a new leapfrog operator, which integrates gradient information via both explicit and implicit approaches.
The explicit approach is to directly use the primitive gradient same as (9), whereas the implicit approach is to use an MLP T h to transform the primitive gradient, state and action together.
Then the gradient information from both explicit and implicit approaches are combined by a gate σ h .
The motivation behind is to improve the policy expressivity by MLP T h and control the numerical stability by gate σ h , making the policy distribution quickly adapt to the changes of Q networks during the learning process.
The inputs of T h and σ h include normalized gradients, action and state, where the state is optional and can be ignored in some environments.
Therefore, the proposed leapfrog operation, transforming from ( a k , ρ k ) to ( a k +1 , ρ k +1 ) , can be described as [Equation]
[Equation]
where g := −∇ Q θ ( s,a k ) (cid:107)∇ Q θ ( s,a k ) (cid:107) , g (cid:48) :
= −∇ Q θ ( s,a k +1 ) (cid:107)∇ Q θ ( s,a k +1 ) (cid:107) and a k +1 = a k + (cid:15) (cid:12) ρ k +1 / 2 .
This process is shown in
Figure 1(b).
Obviously the proposed leapfrog operator (13) still keeps the properties of reversibility and unit Jacobian, so that the distribution of ( a K , ρ K ) in (10) is still tractable.
4.3 Implementation and Algorithms
In implementation, we build the Hamiltonian dynamics (HD) simulated by leapfrog steps on top of the policy network.
Speciﬁcally, we only use one hidden layer for the base policy network π θ , so the number of parameters of our model is much smaller than that of models in previous papers [14, 15, 26] which use two hidden layers in the policy network.
The proposed RL algorithm is built on top of SAC, where the Gaussian policy is replaced by Hamiltonian policy and the policy optimization objective in (12) is used.
It is termed as SAC-HPO.
The process of producing actions from Hamiltonian policy is shown in Figure 1(a).
The details of the algorithm are presented in Algorithm 2 and Algorithm 3 in Appendix.
4.4 Safe Reinforcement Learning with Hamiltonian Policy
In addition to regular RL, we also ﬁnd the proposed method can be used in safe RL to reduce the safety violations.
It is based on Lyapunov constraint [4, 5].
Under this constraint, based on their assumptions, the updated policy is provably guaranteed to satisfy the safety constraint.
It transforms the trajectorywise safety constraint to a state-wise constraint [5].
Speciﬁcally, the Lyapunov constraint is expressed as
[Equation]
where
[Equation]
where π B is the reference policy which is the updated policy in last iteration, Q C,π B is the accumulated safety costs in terms of policy π B , s 0 is the initial state and γ is the discounting factor of the MDP.
However, in previous work [4, 5], the sampled actions are made to satisfy the Lyapunov constraints based on the linear approximation of the cost critic Q C,π B , which is inaccurate in practice since the action may not have a linear relationship with the future costs.
In this paper, we propose to use
Hamiltonian policy to iteratively sample actions until the Lyapunov constraint is satisﬁed, so that potentially unsafe actions can be discarded in this process.
When interacting with the environment, for any sampled action, the agent ﬁrst uses Lyapunovconstraint to predict its safety violation.
If the Lyapunov-constraint is satisﬁed, it is applied into the environment. Otherwise, the sampled action will be updated by the leapfrog operator to get the next sampled actions, until the sampled actions satisfy the Lyapunov constraint.
So, HMC here can not only boost the exploration by randomness and gradient information, but also improve safety in exploration by sampling actions iteratively until the safety (Lyapunov) constraint is satisﬁed.
The application of Hamiltonian policy in safe RL is summarized in Algorithm 1.
Note that compared with regular RL, a difference of Hamiltonian policy in safe RL is that the random noise is injected into momentum variables in every leapfrog step, rather than initial step only.
Experiment
In experiments, the Hamiltonian policy is applied into soft actor critic (SAC), so the proposed method is denoted as "SAC-HPO".
The environments in our experiments are diverse, ranging from OpenAI Gym MuJoCo [43] to the realistic Roboschool PyBullet suit [8, 11].
We empirically evaluate the proposed method from many perspectives.
First, SAC-HPO is compared with the primitive SAC [16] and SAC-NF [27] to show our advantage over classical SAC and normalizing ﬂow policy.
Second, we show the advantage of Hamiltonian policy in two MuJoCo environments with safety constraints, comparing with SAC-Lagrangian [41, 5].
In addition, we conduct ablation study on the proposed leapfrog operator and the sensitivity analysis of hyper-parameters in Section C.
The effect of random momentum variables in HMC is also evaluated in Section C.3.
Finally, in Section C.4, we also analyze the shape of action distribution after leapfrog steps, verifying its non-Gaussianity and improvement of expressivity.
5.1 Continuous Control Tasks
We compare SAC-HPO with SAC and SAC-NF on eight continuous control tasks.
SAC is chosen because it is a fundamental learning method in actor-critic RL.
SAC-NF is selected since it is a representative and widely-used method which adopts normalizing ﬂow policy to improve the exploration.
We use the ofﬁcial implementation of SAC [16].
And we try our best to implement
SAC-NF according to [27], where the policy network is one-layer MLP with 256 hidden units and
ReLU activation and radial normalizing ﬂow is adopted.
The learning curves are shown in Figure 2, where ﬁrst ﬁve tasks, corresponding from Figure 2(a) to Figure 2(e), are from the MuJoCo suite and the other three are from Roboschool PyBullet.
[Figure]
All the methods use the same architecture for Q networks, hyper-parameters, and tuning scheme for the temperature α .
The critic (Q) networks follows the same architecture as [16], i.e., two-layer fully-connected neural networks with 256 units and ReLU activation in each layer, where two Q networks are implemented and trained by bootstrapping.
All networks are updated by Adam optimizer [20] with the learning rate of 3e-4.
The batch size for updating policies and critics is 256, and the size of replay buffer is 10 6 .
In SAC, the policy network consists of two fully-connected hidden layers with 256 units and ReLU activation.
In SAC-HPO, actions are evolved by HD simulated by leapfrog operations (13) for K ∈ { 1 , 2 , 3 } steps.
The base policy only has one hidden layer with 256 units and ReLU activation.
Neural networks in proposed leapfrog ( T h and σ h ) are simple MLPs having one hidden layer with h n hidden units and ELU activation.
The variances of the momentum vector ( β 0 ) should be different for training and exploration, denoted as β tr 0 and β exprespectively.
In most experiments, we ﬁnd the variance of momentum ρ 0 in exploration should be larger than that in training, i.e., β tr 0 < β exp 0 , which can improve exploration efﬁciency.
Besides, it is important to make networks T h and σ h in leapfrog small, which can stabilize the learning process.
Hyper-parameters used in SAC-HPO are shown in Table 1 in Appendix, including number of leapfrog steps ( K ), number of hidden units in T h and σ h ( h n ), momentum variances for training and exploration ( β tr 0 and β exp 0 ), update rate in leapfrog ( (cid:15) ), and temperature parameter ( α ).
All results in Figure 2 show the evaluation performance.
Evaluation happens every 10,000 environmental steps, where each evaluation score (accumulated rewards in one episode) is averaged over 10 runs.
The values reported in the plots are smoothed by exponential moving averaging (EMA) with a window size of 5, equivalent to averaging every 50,000 steps to improve comparability.
We can see that the SAC-HPO outperforms SAC and SAC-NF in terms of both convergence rate and performance.
5.2 Safe Reinforcement Learning
In this section, we evaluate the performance of Hamiltonian policy in safe RL problems.
The environments in this section are Ant-v2 and HalfCheetah-v2 in MuJoCo suite.
At each step the robot selects the amount of torque to apply to each joint.
In Ant-v2, the safety constraint is on the amount of torque the robot decided to apply at each time step [41].
Since the target of the task is to prolong the motor life of the robot, the robot is constrained from using high torque values.
This is accomplished by deﬁning the constraint d 0 as the average torque the agent has applied to each motor.
The constraint threshold on torques is set to be 25 in each episode.
In HalfCheetah-v2, the safety constraint is that the speed of the robot should be less than 1, i.e., the constraint cost is 1 [ | v | > 1] at each time step [5].
The constraint threshold d 0 is on the discounted sum of safety costs in each episode, which is set to be 10 .
[Figure] In experiments, we learn critic networks for both return and accumulated costs (safety violations), denoted as Q ( s, a ) and Q C ( s, a ) respectively.
Both Q and Q C are realized by two-layer MLP with 256 hidden units and ReLU activation in each layer.
The baseline is the Lagrangian-based SAC [5] which introduces a Lagrangian multiplier λ to balance between return and safety costs, shorted as SAC-Lagrangian.
The policy learning objective is max θ E s ∼B ,a ∼ π θ ( ·| s )
[( Q ( s, a ) − λQ C ( s, a )) − α log π θ ( a | s )] .
And the multiplier is updated as λ ←− [ λ + η
( J π θ C − d 0 )] + where η = 0 .
Speciﬁcally, in Ant-v2, J π θ C is the average sum of safety costs averaged in recent episodes, while in HalfCheetah-v2, J π θ C is the discounted sum of safety costs averaged in recent episodes.
In SAC-HPO, the Lyapunov constraint is written as (14), where reference policy π B is the policy updated in the last iteration, and Q π B , Q C,π B are target value networks of return and safety costs which are periodically updated in typical actor-critic RL algorithms.
The working process of
Hamiltonian policy in safe RL is summarized in Algorithm 1 in Section 4.4.
In SAC-HPO, the policy learning objective is in the same form as (12), where Q π θ is replaced by Q π B + λQ C,π B and λ is updated in the same way as SAC-Lagrangian.
The policy network and hyper-parameters of the
SAC-HPO are same as Section 5.1, except that the maximum number of leapfrog K is set to bewhich is larger than that in regular RL problems, and in practice the number of leapfrog steps taken actually is usually much smaller than K .
The performance comparison is presented in Figure 3, showing that our method not only improves the average return, but also reduces the safety violations.
In safe RL, the learning objective contains
Q + λQ C instead of Q , where λ is changing in every learning iteration.
Hence, since the target posterior of actions in (8) is deﬁned in terms of Q + λQ C , the amortization gap in policy optimization is more signiﬁcant than that in regular RL due to the rapid changes of λ . So HMC is more necessary here to make the sampled actions better approximate the target posterior.
Moreover, iterative HMC sampling can discard potentially unsafe actions until safe actions are sampled.
So, Hamiltonian policy can achieve more signiﬁcant performance improvement in safe RL tasks than that in regular RL tasks.
Conclusion
In this work, we propose to integrate policy optimization with HMC, evolving actions from the base policy network by Hamiltonian dynamics simulated by leapfrog steps.
In order to adapt to the changes of Q functions which deﬁne the target posterior, we propose a new leapfrog operator which generalizes HMC via gated neural networks.
The proposed method can improve the efﬁciency of policy optimization and make the exploration more directionally informed.
In empirical experiments, the proposed method can outperform baselines in terms of both convergence rate and performance.
In safe RL problems, the Hamiltonian policy cannot only improve the achieved return but also reduce the number of safety constraint violations.