Stationary Deep Reinforcement Learning with Quantum K-spin Hamiltonian Equation
Anonymous Author(s)
AffiliationAddress email
Abstract
A foundational issue in deep reinforcement learning (DRL) is that Bellman’s optimality equation has multiple fixed points —failing to return a consistent one.
Adirect evidence is the instability of existing DRL algorithms, namely, the highvariance of cumulative rewards over multiple runs.
As a fix of this problem, wepropose a quantum K-spin Hamiltonian regularization term (H-term) to help apolicy network stably find a stationary policy, which represents the lowest energyconfiguration of a system.
First, we make a novel analogy between a Markov
Decision Process (MDP) and a quantum K-spin Ising model and reformulate theobjective function into a quantum K-spin Hamiltonian equation, a functional ofpolicy that measures its energy.
Then, we propose a generic actor-critic algorithmthat utilizes the H-term to regularize the policy/actor network and provide Hamiltonian policy gradient calculations.
Finally, on six challenging MuJoCo tasks over20 runs, the proposed algorithm reduces the variance of cumulative rewards by65 . 2% ∼ 85 .
6% compared with those of existing algorithms.
Introduction
Deep reinforcement learning (DRL) [33] algorithms are quite unstable, namely, agents trained withdifferent random seeds may have dramatically different performance.
Existing works [1, 13, 6, 22]reported a high variance over multiple runs, thus it requires to train tens of agents and then pick thebest one.
Such a high variance puts a challenge on reliability and reproducibility [15, 14], limitingthe wider adoption in real-world tasks.
The difficulty of guaranteeing stability lies in a foundational issue that Bellman’s optimality equationhas multiple fixed points [4, 28, 20, 18] —failing to return a consistent one.
Consider MDP exampleswith an terminal state 0 , as shown in Fig.
1 (we adapt dynamic programming examples [4, 28] intoreinforcement learning settings.
Observational experiments are given in Section 2.2),• Shortest path problem (deterministic) in Fig.
1(a):
At state 1 ,
an agent transits to either state1 or 0 with reward 0 or b , respectively.
Assume the value function for state 0 is V (0)
= 0 .
TheBellman’s optimality equation for state 1 is V (1) =
max { V (1) , b } , where any V (1) ≥ b is afeasible solution.
If initialize V 0 (1) ≥ b , a resulting policy is that an agent at state 1 always transitsback to state 1 ; otherwise, drives to terminal state
(always returns back to itself with reward 0 ).
• Blackmailer’s problem (stochastic) in Fig.
1(b):
At state 1 , a profit maximizing blackmailerdemands a cash amount a ∈ (0 , 1] ; a victim transits to state 1 with probability a or state 0 withprobability 1 − a , respectively.
At state 0 , a victim always refuses to yield, i.e., V (0) = 0 .
TheBellman’s optimality equation for state 1 is V (1) = max a { a +(1 − a ) V (1) } , where any V (1) ≥ 1is a feasible solution.
If initialize V 0 (1) > 1 , the blackmailer’s policy is demanding a → 0 to keepthe victim at state 1 ; otherwise, demanding a = 1 that drives the victim to terminal state 0 .
[Figure]
The instability problem has been partially addressed, such as ensemble methods [2, 8], regularizationapproaches [35, 9], and baseline-correction approaches [30, 39].
In particular, Generalized Advantage
Estimation (GAE) [30] is a widely used one that significantly reduces the variance of the advantagefunction.
However, they did NOT fix the foundational issue of Bellman’s optimality equation in Fig.1, thus the objective function inherently fails to search for a consistent policy.
For practical usage, weoften expect a DRL algorithm stably converges to a certain policy independent of initialization andnoises.
As a fix of the problem, we make a novel analogy between an MDP and a quantum K-spin Ising model[24, 12], and reformulate a reward-maximization RL task into an energy-minimization problem,namely, finding the lowest-energy configuration of a quantum spin system.
We hypothesize thata physically stationary policy would have the lowest energy .
Different from our quantum K-spinperspective, several recent papers utilized the (classical) Hamiltonian equation to endow RL agentsthe capability of inductive biases.
For example, [19, 37] used Hamiltonian mechanics to train anagent that learns and respects conservation laws ; [40] applied a Hamiltonian Monte Carlo (HMC)simulator to approximate the posterior action probability; and [26] proposed an unbiased estimatorfor the stochastic Hamiltonian gradient methods for min-max optimization problems.
In this paper, we propose a quantum-inspired K-spin Hamiltonian regularization term (H-term) thathelps policy gradient algorithms stably converge to a physically stationary policy.
Our contributionsare as follows: 1) we derive a novel regularizer, H-term , by a novel reformulation of the RL’s objectiveas a K-spin Hamiltonian equation that measures the energy of a policy; 2) we propose a genericactor-critic algorithm that utilizes the H-term to regularize the policy/actor network and provide
Hamiltonian policy gradient calculations on both deterministic and stochastic cases; and 3) on sixchallenging MuJoCo tasks over 20 runs, we show that the proposed algorithm reduces the variance ofcumulative rewards by 65 . 2% ∼ 85 .
6% compared with those of existing algorithms.
Bellman’s Optimality Equation and Multiple Fixed Points
We describe Bellman’s optimaility equation and provide observational experiments to verify theexistence of multiple policies.
2.1 Bellman’s Optimality Equation
A reinforcement learning (RL) [33] agent interacts with an unknown environment and learns anoptimal policy that maximizes the cumulative reward.
Mathematically, the environment can beformulated as a Markov Decision Process (MDP) with the five-tuple ⟨S , A , P , R, γ ⟩ .
Here S and
A denote the state and action spaces; P : S ×
A → ∆( S ) denotes a transition probability function,where ∆ is a probability simplex; R : S ×
A × S → R denotes a reward function; and γ ∈ (0 , 1] denotes a discount factor.
The objective is to find an optimal policy π ∗ : S → ∆( A ) that maximizes(discounted) expected reward.
Consider a discrete, finite, discounted MDP with infinite horizon, one can define the Q-value functionof a state-action pair ( s, a ) under policy π as follows
[Equation]
where R ( S k , A k , S k +1 ) denotes the immediate reward when taking action A k at state S k and
arrivingat state S k +1 , capital letters denote random variables and lowercase letters denote values.
Theconventional objective function J ( θ ) of reinforcement learning [33] takes the following form
[Equation]
where τ is a trajectory, i.e., τ = ( S 0 , A 0 , · · · ) , and P ( τ | π )
=
d 0 ( s 0 ) · (cid:81) Tk =
P ( S k +
| s k , a k ) π ( a k | s k ) .
The Bellman equation [33] converts (1) into a recursive form as follows
[Equation]
which expresses the expected reward as a summation of immediate reward R ( s, a ) and discounted future rewards, and the immediate reward R ( s, a ) is defined as R ( s, a )
= (cid:80) s ′ ∈S P ( s ′ | s, a ) R ( s, a, s ′ ) .
The Bellman’s optimality equation [33] is
[Equation]
The optimal policy π ∗ is given by a greedy strategy such that π ∗ =
arg max π Q π ( s, a ) .
2.2 Observational Experiments for Multiple Policies and Physically Stationary Policy
As we pointed out in Fig.
1, theoretically there exists multiple policies.
Here, we perform observational experiments on four challenging MuJoCo tasks [36], namely, Humanoid, Hopper, HalfCheetah,and Ant (details given in Appx.
B.1), which are typical examples of the locomotion control of a robot.
We render the obtained policies over multiple runs and then identify physically stationary ones.
Weobserve various types of moving strategies, as shown in Fig.
2, which verifies that multiple policiesare very common.
For example, the Humanoid agent learns either jumping with a single leg or
[Table]
running with two legs, as shown in Fig.
2 (top-left); another interesting example is HalfCheetah, inwhich an agent can run normally or in a flipped manner, as shown in Fig.
2 (bottom-left).
Amongthe obtained policies, one can easily identify the physically stationary polices that control the robotmoving forward with a stable gait (defined as gait that does not lead to fall).
Reinforcement Learning as Quantum K-spin Hamiltonian Equation
First, we make a novel analogy between the MDP and a quantum K-spin Ising model, and reformulatethe objective function (2) into a quantum K-spin Hamiltonian equation.
Then, we show that this
Hamiltonian equation helps fix the issue of multiple fixed points in Fig.
3.1 Motivation through Analogy with Quantum K-spin Ising Model
Given the multiple policies in Section 2.2, it is critical to employ a proper criteria to search for aphysically stationary policy.
In Table 1, we make a novel analogy between an MDP and a quantum
K-spin Ising model [24, 12].
The Hamiltonian equation for a quantum K-spin Ising model [24, 12] measures the energy of aparticular configuration of a quantum K-spin system and takes the following form
[Equation]
where N is the number of spins in the k -th configuration, σ j k = ± 1 are spin variables, and L j 0 ...
j k isan energy density function for k nearest spins’ configuration ( σ j 0 , . . .
, σ j k ) .
Analogy in Table 1 .
Starting from an analogy between a state-action pair µ k = ( S k , A k ) anda spin j k , we can map an optimal policy π ∗ ( µ k ) ∈ { 0 , 1 } in (4) to a single-qubit spin operatorσ j k ∈ {− 1 , 1 } via π ∗ ( µ k ) ←→ ( 1 µ k − σ µ k ) / 2 , where π θ ( µ k ) denotes the probability of takingaction A k at state S k , following policy π θ .
The energy density function L j 0 ...
j k can be defined as thediscounted reward on a path ( µ 0 , · · · , µ k − 1 ) of length k , i.e.,
[Equation]
(obtained via Monte Carlo simulation) (6) where d 0 ( S 0 ) denotes the distribution of initial state S 0 .
Analogy to the quantum K-spin Ising model,we can derive a functional of policy H ( π µ 0 , ..., π µ K − 1 ) in the context of MDP, which measures theenergy of an RL policy.
In this case, a physically stationary policy satisfies the stationary conditionδH ( π µ 0 , · · · , π µ K − 1 ) /δπ µ = 0 .
3.2 Reformulation into Quantum K-spin Hamiltonian Equation
Inspired by [17], we formally reformulate (2) into a K -spin Hamiltonian equation
[Equation]
where K → ∞ , the expectation is taken over S 0 ∼ d 0 ( · ) , A 0 ∼ π θ ( S 0 , · ) , and the density function
L µ 0 ,...,µ k is given in (6).
Physical interpretation : Analogy to a quantum K-spin system, H ( θ ) in (7) measures the energy ofpolicy π .
We hypothesize that the energy of a policy is a favorable criteria, since a stationary policywith minimum energy: 1).
achieves a relative high reward independent of the initialization ; and 2).
isrobust to interference/noise in the inference stage .
K-step truncation in practice .
Minimizing (7) is NP-hard [11].
Since γ ∈ (0 , 1) , γ K monotonicallydecreased with look-ahead steps K , therefore, we truncate (7) to finite K terms.
One can showthat these K terms in (7) is a geometric sequence with a truncation error ratio 1 − γ K .
Assuming
− γ K ≤ 1 − ϵ , where ϵ > 0 is small, thus we have the look-ahead steps satisfies K ≥ log γ ϵ .
3 Revisiting Examples in Fig.
We elaborate how adding the energy measured by (7) onto each state can help drive to the terminalstate (a stationary policy), which fixes the foundational issue of multiple fixed points in Fig.
1 whereγ = 1 .
We have H (0) =
for the terminal state 0 .
• (a) Shortest path problem (deterministic) : H (1) = − (cid:80) ∞ k =1 b = −∞ .
At state 1 , the Bellman’soptimality equation becomes V (1) =
max { V (1) + λH (1) , b } .
Independent of the initial value
V 0 (1) , an agent obtains a policy that always transits back to terminal state 0 .
• (b) Blackmailer’s problem (stochastic) : H (1) =
−∞ .
The Bellman’s optimality equationbecomes V (1) = max a { a +(1 − a )( V (1)+ λH (1)) } for state 1 .
For any V 0 (1) < ∞ , the optimal
policy becomes a = 1 that drives to the terminal state 0 .
• (c) Optimal stopping problem (terminating policies) : any policy that takes infinite
steps willhave H ( x ) =
−∞ , since at each step number k , there are always trajectories that jump to point 0with reward − c ; and a direct jumping policy will have H ( x ) = − c .
Therefore, adding H ( x ) toeach point x ̸ = 0 will lead to a policy of jumping back to point 0 .
Actor-Critic Algorithm with Quantum K-spin Hamiltonian Regularization
We propose a generic actor-critic algorithm with a H-term, derive two Hamiltonian’s policy gradienttheorems for both deterministic and stochastic cases, and present the Monte Carlo gradient estimation.
4.1 Stationary Actor-Critic Algorithm with H-term
Actor-critic algorithms in reinforcement learning perform a bilevel optimization, namely alternatingbetween approximating a value function and optimizing a policy.
In practice, a critic network withparameter ϕ approximates the Q -value function, and an actor network with parameter θ approximatesthe policy π , details given in Appx.
E.
However, since the critic’s update is governed by the Bellman’soptimality equation, actor-critic algorithms suffer the multiple fixed points problem.
Motivated by Section 3.3, we propose a novel H-term for both deterministic and stochastic actor-criticalgorithms.
Similar to the entropy term in [21], the proposed H-term is an add-on term to regularizethe actor network and help it converge to a stationary policy.
Specifically, the objective functions ofactor and critic networks become:
[Equation]
(8) where a target Q-value is y ( S k
[Equation]
A
[Equation]
A k +1 ) , and λ > 0 is a temperatureparameter.
As an interpretation, the second term − λH ( θ ) in the maximization objective function ofactor network aims to find a minimum energy configuration for the MDP problem, namely, a policyπ that will add a minimum amount of energy to each state’s value function (as in Section 3.3).
New algorithm .
In Alg. 1, an agent interacts with an environment and alternatively updates its actornetwork and critic network.
The algorithm has M episodes and each episode consists of a (Monte
Carlo) simulation process and a learning process (gradient estimation) as follows:
• During the (Monte Carlo) simulation process (lines 5 - 10 of Alg. 1), an agent takes action a t ac-
cording to a policy π θ ( ·| s t ) , t = 0 , · · · , T − 1 , generating a trajectory of T steps/transitions.
Algorithm
1: Input : learning rate α , temperature λ , look-ahead step K , and parameters M, T, G, B, B ′ 2: Initialize actor network π and critic network Q with parameters θ , ϕ , and replay buffers D 1 , D3: for episode = 1 , · · · , M do 4:
Initialize state s5: for t = 0 , · · · , T − 1 do 6:
Select action a t ∼ π θ ( ·| s t ) 7:
Execute action a t , receive reward r t , and observe new state s t + 8:
Store a transition ( s t , a t , r t , s t +1 ) in D9: end 10:
Store a trajectory τ of length T in D11: for g = 1 , · · · , G do 12:
Randomly sample a mini-batch of B transitions { ( s i , a i , r i , s i +1 ) }
Bi
=1 from D13:
Randomly sample a mini-batch of B ′ trajectories (of length K ) { τ j } B ′ j =1 from D14:
Update critic network using a conventional method 15:
Update actor network as θ ← θ + α (cid:16) ∇ θ (cid:98)
J ( θ ) − λ ∇ θ
(cid:98) H ( θ ) (cid:17) .
16: end 17: end
Then, these T transitions are stored into a replay buffer D 1 , while the full trajectory τ =( s 0 , a 0 , r 0 , s 1 , · · · , s T − 1 , a T − 1 , r T − 1 , s T ) is stored in replay buffer D 2 .•
During the learning process ( G ≥ 1 updates in one episode) (lines 11 - 16 of Alg. 1), a minibatch of B transitions { ( s i , a i , r i , s i +1 ) }
Bi =1 and a mini-batch of B ′ trajectories (of length K ){ τ j = ( s j 0 , a j 0 , r j 0 , s j 1 , · · · , s jK − 1 , a jK − 1 , r jK − 1 , s jK ) } B ′ j =1 are sampled from D 1 and D 2 , respectively.
The critic network is updated by a conventional method, e.g., minimizing the mean squared error(MSE) between an estimated Q-value and a target value.
The actor is updated by a Monte Carlogradient estimator over B transitions and B ′ trajectories.
Two new hyperparameters .
We introduce two hyperparameters: a temperature λ > 0 that is arelative weight of the H-term, and a look-ahead step K ≤ T that defines the horizon of the H-term.
Implementation of replay buffer D 2 .
After a full trajectory τ of length T is generated, it ispartitioned into T − K +1 trajectories of length K .
We rank them according to the cumulative rewardand store the top portion, say 80% , into a new replay buffer D 2 (line 10 of Alg. 1).
We randomlysample a mini-batch of B ′ trajectories from D 2 (line 13 of Alg. 1) to compute the H-term.
4.2 Hamiltonian Policy Gradient and Monte Carlo-based Gradient Estimator
We provide the policy gradient of the quantum K-spin Hamiltonian equation in (7), which are variantsof the well-known policy gradient theorem [33].
We provide detailed derivations in Appx.
D.
Stochastic version .
The Hamiltonian stochastic gradient of (7) w.r.t.
parameter θ is
[Equation] [Equation]
Deterministic version .
Let η θ ( · ) :
S → A denote a deterministic policy, while we use (cid:101) π θ,δ ( µ ) torepresent that a Gaussian noise (a.k.a, an exploration noise) with standard deviation δ > 0 is added inthe exploration process.
The Hamiltonian deterministic gradient of (7) w.r.t.
parameter θ is
[Equation]
The quantum K-spin Hamiltonian equation in (7) is a reformulation of (2).
We verify the gradientcalculation by showing that: when K → ∞ , the Hamiltonian stochastic and deterministic policygradient ∇ θ H ( θ ) and ∇ θ H ′ ( θ ) are equal to the stochastic policy gradient ∇ θ J ( θ ) in [34] anddeterministic policy gradient ∇ θ J ′ ( θ ) in [32], respectively.
Note that the gradient ∇ θ H ( θ ) in (9) and ∇ θ H ′ ( θ ) in (10) w.r.t.
a distributional parameter θ takes anexpectation form.
Thus, a Monte Carlo gradient estimator is practically useful.
We obtain the Monte
Carlo gradient estimator of ∇ θ H ( θ ) , illustrated in Fig.
3 (right), as follows
[Equation]
As a contrast, we provide the Monte Carlo gradient estimator of REINFORCE’s [34] policy gradient,as illustrated in Fig.
3 (left), as follows
[Equation]
where G
[Equation]
=
[Equation]
γ
[Equation]
R
[Equation] [Equation]
An interesting observation is that both gradient calculations follow a similar pattern as shown in Fig.3.
REINFORCE’s policy gradient [34] in Fig.
3 (left) employs an estimate of future rewards, while
Hamiltonian policy gradient in Fig.
3 (right) uses trajectories in replay buffer D 2 .
Computational complexity : we measure the computation complexity by the times of computing one∇ θ log π θ ( µ ) .
Assume N = B and N ′ =
B ′ , since most DRL algorithms use a mini-batch stochasticgradient decent methods.
REINFORCE’s
[34] policy gradient in (12) takes O ( BT ) computations,while Alg.
adds O ( B ′ K ( K + 1) / 2) computations in each gradient update step, thus a totalcomplexity of O ( BT + B ′ K ( K + 1) / 2) .
Performance Evaluation
We evaluate the proposed H-term from four aspects: 1) increasing cumulative reward, 2) reducingvariance, 3) driving to physically stationary policy, and 4) the impact of trajectory length K .
Allexperiments were executed on an NVIDIA DGX-2 server [10].
The server contains 8 A100 GPUs,320 GB GPU memory, and 128 CPU cores running at 2.25 GHz.
5.1 Experimental Settings
Environments (tasks) .
We consider six challenging MuJoCo tasks [36] as in Section 2.2.
The agentlearns to control the locomotion of a robot and aims to move forward as quickly as possible.
Thesetasks have high-dimensional continuous state and action spaces, in which there exists multiple locallyoptimal polices as revealed in Section 2.2.
Compared algorithms .
To evaluate deterministic and stochastic algorithms, we choose Deep
Deterministic Policy Gradient (DDPG) [25] and Proximal Policy Optimization (PPO) [31].
Since the
H-term is compatible with existing variance reduction techniques, we implement the PPO algorithmwith GAE [30].
For a fair comparison, we keep the hyperparameters (listed in Appx.
G) the sameand make sure that the obtained results reproduce existing benchmark tests [13].
Performance metrics .
We employ two performance metrics, the cumulative rewards and variance,while in Section 5.4, we further consider different policies and report the number of convergence.
Werun each experiment with 20 random seeds and in each run we test 100 episodes.
5.2 H-term Increases Cumulative Reward
Experience replay is crucial in improving performance in terms of cumulative reward.
The proposed
H-term in (11) can be viewed as a novel experience replay technique for an actor network.
Here, we
[Table]
[Figure]
[Table]
[Figure]
[Table]
m
[Table]
l a t i v
[Table]
r
[Table]
w a r d s
HalfCheetah
[Table]
Humanoid
[Table]
[Figure]
[Table]
[Figure]
l a t i v
[Figure]
r
[Figure]
[Table]
a r d s
[Table]
[Figure]
[Table]
[Figure]
u m u l a t
i v e r e w a r d s
[Figure]
[Table]
[Figure]
t
i
[Figure]
e r
[Figure]
Humanoid
[Figure]
u l a t i v
[Figure]
r
[Figure]
a r
[Figure]
add a compared algorithm, DDPG with Prioritized Experience Replay [29] (DDPG+PER), where
PER prioritizes experience by the TD error to update a critic network.
In Fig.
4, both DDPG+PER and DDPG+H achieve a substantial improvement of cumulative reward.
Inparticular, DDPG+H achieves the highest cumulative rewards in all six tasks, which are comparable to
PPO’s performance in Fig.
It is worthwhile to discuss the advantage of DDPG+H over DDPG+PER.
DDPG+PER utilizes a prioritized replay strategy to obtain a more accurate critic network, however,it is updated via the Bellman equation with the trouble of multiple fixed points.
In contrast, the
H-term in DDPG+H is performed on the actor network.
Our results indicate that an experience replaytechnique on actor network may be much more powerful.
5.3 H-term Reduces Variance
The PPO algorithm with GAE is regarded as the state-of-the-art algorithm in MuJoCo environments.
However, it still has a very high variance (the shaded area) after the policies have converged, asshown in Fig.
We observe that, at the end of training, the PPO algorithm has a variance of 969 . 2 ,1563 . 4 , 2513 . 5 , 905 . 3 , 60 . 7 , 1290 .
1 in the six tasks, respectively.
Such a high variance is mainly dueto the fact that the agent may converge to a random one of multiple policies.
[Table]
In Fig.
[Table]
the shaded areas of PPO+H ( K = 16 ) are dramatically smaller, i.e., a variance of 228 . 4 ,225 . 4 , 683 . 7 , 184 . 2 , 31 . 6 , and 296 . 8 , respectively.
The variance has been reduced by 65 . 2% ∼ 85 .
6% ,which verifies the effectiveness of the proposed H-term.
In Fig.
4, we also observe that the H-termcan help the DDPG algorithm reduce variances, namely, the variances of DDPG+H are much smallerthan those of vanilla DDPG and DDPG+PER.
Therefore, we may conclude that the H-term guidesthe agent to search for a stationary policy among multiple feasible ones.
More experimental results are given in Appx.
G due to the space limit, including the cases of K = 8and K
=
24 , and the curves of the H-term value during the training process.
One may verify that thestationary policies have relative lower H-values.
5.4 H-term Drives to Physically Stationary Policy
A key question needs to be answered: is H-term really guiding the agent converge to a physicallystationary policy?
Similar to Section 2.2, we perform observational experiments on the six MuJoCotasks and measure the number of convergence to the different policies over 20 runs.
As shown in
Table 2, the vanilla PPO algorithm converges to the physically stationary policy ( bold ) with 13 , 17 , 7 ,10 , 14 , and 5 times for the six tasks, while the PPO+H ( K = 16 ) converges to the stationary policywith 20 , 20 , 16 , 20 , 20 , and 16 times, respectively.
From the empirical observation, we find that the
PPO gets stuck in locally optimal policies, failing to find a consistent one.
As expected, PPO+H canconverge to the stationary policy with a substantially higher ratio, which verifies the effectiveness ofthe proposed H-term in finding a physically stationary policy.
5.5 Impact of Trajectory Length K
We investigate the impact of trajectory length K that defines the horizon of the H-term.
Based on (11),we foresee that a large K means a more accurate estimation of ∇ θ H ( θ ) but at a price of computations.
Here, we evaluate the PPO+H with K = 8 , 16 and set the size of replay buffer D 2 to 1 , 000 .
In Table2, we observe that the cumulative reward increases and the variance decreases as K increases from 8to 16 .
However, for the case K = 24 , both metrics get worse due to the out-of-memory issue.
For
K = 24 , we reduce the replay buffer size to 800 to meet the memory limit.
The smaller replay buffersize hurts the diversity of the trajectories and may lead to a small performance drop.
This hypothesizeis verified in Appx.
G.2, where all trials use a consistent replay buffer size 800 .
Conclusions
In this paper, we have addressed a foundational issue of existing deep reinforcement learningalgorithms that Bellman’s optimality equation has multiple fixed points .
This issue leads to theinstability of DRL algorithms, puts a challenge on their reliability and reproducibility, and thuslimits the wider adoption in real-world tasks.
As a fix of the problem, we propose a novel H-term, aphysically inspired regularizer, by making a novel analogy between a MDP and a quantum K-spin
Ising model.
Experimentally, we show that the H-term helps DRL algorithms find a stationarypolicy that has the lowest energy in a system and reduces the variance of cumulative rewards by65 . 2% ∼ 85 .
6% compared with those of existing algorithms.
For future works, we will explore the potential of directly training a policy network using (7) as in
Appx.
H, quantum simulator [23] and quantum reinforcement learning [5][16].
It is interesting toapply Monte Carlo estimator for unbiased policy gradient calculations.
We would like to show thatthe proposed H-term can help distributional RL algorithms [3] find a stationary policy, since thedistributional Bellman optimality operator is not a contraction and thus there is also no unique policy.
Broader Impact Statement
In the field of DRL research, this paper points out the issue of multiple fixed points and may attractmore attention from various perspectives.
In terms of applications, this paper moves toward thereliable and reproducible research by improving the stability of existing DRL algorithms.
On the otherhand, the obtained stationary policy may have broad practical impact in many real-world applicationareas, including but not limited to robotics, transportation, and finance.
From an interdisciplinary perspective, our approach lies at the intersection of (quantum) Hamiltonianmechanics and DRL, espcially the explicit analogy between an MDP and a quantum K-spin Isingmodel.
We hope that our study will attract more attention to bring together the strengths of bothapproaches and yield new insights in both fields.
In terms of broader societal impact of this work, we do not see any foreseeable strongly negativeimpacts.
However, this work essentially makes a tradeoff between the computational resource andstability, which may lead to future works with higher computational cost.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] More computational cost.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] May lead
to future works with higher computational cost.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes] In the appendix.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes](b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes](c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We reported the variance over 20 random seeds as shadedarea in the figures, and also list it in a table.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[Yes](b) Did you mention the license of the assets?
[N/A](c)
Did you include any new assets either in the supplemental material or as a URL?
[Yes](d) Did you discuss whether
and how consent was obtained from people whose data you’reusing/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]
[Table]
[Figure]
A More Examples with Multiple Fixed Points
First, we consider the discounted formulations of the three examples (shown in Fig. 1), as shown in
Fig.
5 where γ ∈ (0 , 1) .
The differences are marked in red.
• (a) Shortest path problem (deterministic, discounted case) :
Given two states 1 and 0 , an agentat state 1 transits to either state 1 or 0 with rewards r = c or r = b , respectively.
c > (1 − γ ) · b .At state 0 , the value function is V (0) = 0 .
At state 1 , the Bellman’s optimality equation isV (1) =
max { c + γ · V (1) , b } , where any V (1) ≥ ( b − c ) /γ is a solution.
If initialize V 0 (1) ≥ b ,an agent obtains a policy that always transits back to state 1 ; otherwise, a result policy drives toterminal state 0 .•
(b) Blackmailer’s problem (stochastic, discounted case) :
Different from (a), a profit maximizingblackmailer/agent at 1 demands a cash amount a ∈ (0 , 1] (an action), while a victim transits to state1 with probability a or to state 0 with probability 1 − a , respectively.
At state 0 , a victim alwaysrefuses to yield to the blackmailer’s demand, i.e., V (0) = 0 .
The Bellman’s optimality equationis V (1) = max a { a + γ · (1 − a ) V (1) } for state 1 , where any V (1) ≥ 1 is a feasible solution.
Ifinitialize V 0 (1) =
c > 1 , the blackmailer’s policy is demanding a → 0 at the k -th step to keepthe victim stay at state 1 , for any k ≤ K 0 = −⌊ log γ c ⌋ ; and taking a = 1 to transit to terminalstate 0 at the k -th step, for any k ≥ K 0 + 1 ; otherwise initialize V 0 (1)
=
c ≤ 1 , the result policy isdemanding the maximum a = 1 that drives the victim to a refusal state 0 (a terminal state).
• (c) Optimal stopping problem (terminating policies, discounted case) :
In a space R 2 withterminating state at point 0 , at point x ̸ = 0 an agent moves to either point 0 with negative reward − cor point αx with reward −|| x || , respectively, where α ∈ (0 , 1) .
The Bellman’s optimality equationis V ( x ) = max {− c, −|| x || + γ · V ( αx ) } and the optimal policy is to continue inside the sphere ofradius (1 − α ) c and to stop outside.
If add a cone region C within which an agent always receivesa reward − c , a second policy is jumping to point 0 at any point in region C .
Then, we elaborate how the proposed H-term fixes the problems in Fig.
(a) Shortest path problem (deterministic, discounted case)
Assume V 0 (1) ≥ b and c > (1 − γ ) b , we have
V 1 (1) =
c
+ γ · V 0 (1) ≥ c + γ · b > b
V 2 (1) =
c
+ γ · c + γ 2 · V 0 (1) ≥ (1 + γ ) c
+ γ 2 b > b
V 3 (1) =
c + γ · c + γ 2 c + γ 3 · V 0 (1) ≥ (1 + γ + γ 2 ) c
+ γ 3 b > b · · ·
V k (1) =
k − 1 (cid:88) i
= γ
i · c + γ k · V 0 (1) ≥ k − 1 (cid:88) i
= γ
i · c + γ k b > b · · ·
V ∗ (1) = ∞ (cid:88) i = γ i · c = 1 − γ c > b (13)
The values of H (0) and H (1) are as follows:
[Equation]
Adding the above H-values to state 1 and 0 , respectively, we have
[Equation]
Therefore, V ∗ (1) + H (1) <
V ∗ (0) + H (0) , independent of the initial value V 0 (1) .
That is, an agentalways obtains a policy that drives to terminal state 0 at step 1.
(b) Blackmailer’s problem (stochastic, discounted case)
If initialize V 0 (1) = c > 1 , the blackmailer’s policy is demanding a → 0 at the k -th step to keep thevictim stay at state 1 , for any k ≤ K 0 = −⌊ log γ c ⌋ ; and taking a = 1 to transit to terminal state
atthe k -th step, for any k ≥ K
+ 1 ; otherwise initialize V
(1) =
c ≤ 1 , the result policy is demandingthe maximum a = 1 that drives the victim to a refusal state
(a terminal state).
The values of H (0) and H (1) are as follows:
[Equation]
For arbitrary initial value of V 0 (1) , V 1 (1) =
a +(1 − a ) · γ ( V 0 (1)+ H (1)) take maximum V 1 (1)
=
1when a = 1 .
Therefore, the policy always drives to terminal state
at step 1 .
c
Optimal stopping problem (terminating policies, discounted case)
Any policy that takes infinite steps will have
[Equation]
and a direct jumping policy will have H ( x ) =
− c .
Therefore, the H-term drives to a terminatingpolicy.
B MuJoCo Tasks with Multiple Policies
B.1 Description of MuJoCo Taks
We selected six challenging robotic locomotion tasks from MuJoCo, namely, Swimmer-v3, Hopperv3, Walker2D-v3, HalfCheetah-v3, Ant-v3, Humanoid-v3.
Table 3 lists the action space and statespace of each task.
[Table]
B.2 Multiple policies in MuJoCo tasks
[Table]
In the supplementary files
[Table]
we includes rendered videos of different policies
[Table]
as given in Table 4.
• Different policies are obtained over 20 runs of the PPO algorithm.
We rendered theses polices and
classified them by physical gaits.
• The policies in bold texts are physically stationary.
[Table]
_standing.mp
C Quantum K-Spin Hamiltonian Formulation of Reinforcement Learning
We provide the detailed steps of reformulating (1) into a K -spin Hamiltonian equation
[Equation]
where K → ∞ , and the density function is
[Equation]
[Table]
D Derivation Steps for Section 4.2: Hamiltonian’s Policy Gradients
We provide the policy gradient of the quantum K-spin Hamiltonian equation in (7) for both stochasticand deterministic cases, which are variants of the well-known policy gradient theorem [33].
Theorem 1.
( Hamiltonian’s stochastic policy gradient )
The stochastic gradient of the K-spin
Hamiltonian equation (7) w.r.t.
parameter θ is
[Equation]
Corollary 1.
When K → ∞ , the Hamiltonian’s stochastic policy gradient ∇ θ H ( θ ) in (20) is equalto the stochastic policy gradient ∇ θ J ( θ ) in [34],
[Equation]
Let η θ ( · ) :
S → A denote a deterministic policy, while we use (cid:101) π θ,δ ( µ ) to represent that a Gaussiannoise (a.k.a, an exploration noise) with standard deviation δ > 0 is added in the exploration process.
Theorem 2.
( Hamiltonian’s deterministic policy gradient )
The deterministic gradient of the K-spin
Hamiltonian equation (7) w.r.t.
parameter θ is
[Equation]
Corollary 2.
When K → ∞ , the Hamiltonian’s deterministic policy gradient ∇ θ H ′ ( θ ) in (22) isequal to the deterministic policy gradient ∇ θ J ′ ( θ ) in [32],
[Equation]
[Equation]
Corollary 3.
When the variance of the exploration noise approaches zero, i.e., δ → 0 , the deterministic policy gradient ∇ θ H
( θ ) is the limiting case of the stochastic policy gradient ∇ θ H ( θ ) ,
[Equation]
[Equation]
D.1 Proof of Theorem 1: Hamiltonian’s Stochastic Policy GradientProof.
[Equation] [Equation]
= −
[Equation]
µ·
[Equation]
S×A (cid:88) µ k
L µ
[Equation] [Equation]
k
[ π
[Equation]
( µ 0 ) · · · π θ ( µ k )] ∇ θ log [ π θ ( µ 0 ) · · · π θ ( µ k )]
= −
K − 1 (cid:88) k
=
S×A (cid:88) µ· · ·
S×A (cid:88) µ k γ k · R ( µ k ) · d 0 ( S 0 ) · π θ ( µ 0 )
k − 1 (cid:89)
[Equation]
where µ k =
( S k , A k ) , S 0 ∼ d 0 ( · ) , A k ∼ π θ ( S k , · ) , S k +1 ∼ P ( · | S k , A k ) for k = 0 · · · K .
D.2 Proof of Corollary
Proof.
[Table]
[Equation]
where ρ ( S, S ′ , i ) denotes the probability of state S transfer to S ′
in i steps.
We provide detailed explanations step-by-step:
• Equality ( a ) holds by definition.•
In equality ( b ) , using the chain rule, we take derivative of ∇ θ
[ π θ ( µ 0 ) · · · π θ ( µ k )] with respect toπ θ ( µ i ) , i = 1 , ..., k .•
In equality ( c ), we plug in L µ 0 , ··· ,µ k in (6).• In equality ( d ), we insert P ( S i +1 | µ i ) P ( S i +1 | µ i ) between π θ ( µ i ) and π θ ( µ i +1 ) , i = 1 , ..., k .•
In equality ( e ), we split trajectory µ 0 , · · · , µ i , · · · , µ k into two trajectories µ 0 , · · · , µ i andµ i , · · · , µ k .
Therefore, we can classify all trajectories µ 0 , · · · , µ k by µ 0 , µ i , µ k , and i .•
In equality ( f ), we reorganize (cid:80) K − 1 k =0 (cid:80) ki =0 into (cid:80) K − 1 i =
(cid:80) K − 1 k
= i .
The former one first traverses the
length k of a trajeoctory, and then traverses the i -th step on it.
The latter one first traverses the i -th
step of a trajectory, and then traverses the length k of it.
• In equality ( g ), we calculate the limit of ( f ) when K approaches ∞ .
• In equality ( h ), we normalize (cid:80) S S
d 0 ( S 0 ) (cid:80) ∞
i =
ρ ( S 0 , S, i ) to be a probability distribution.
• In equality ( i ), we remove the constant (cid:80) S S (cid:80) S S
d 0 ( S 0 ) (cid:80) ∞
i =
ρ ( S 0 , S, i ) and replace the fraction
with d π θ ( S ) , the stationary distribution of state S under policy π θ .
• In equality ( j ), we reformulate ( i ) as expectation.
D.3 Proof of Theorem 2: Hamiltonian’s Deterministic Policy Gradient
Proof.
Let η θ ( · ) :
S → A denote a deterministic policy, while we use (cid:101) π θ,δ ( µ ) to represent that
a
Gaussian noise (a.k.a, an exploration noise) with standard deviation δ > 0 is added in the explorationprocess.
In the inference stage, there is no exploration noise, the policy is deterministic, i.e., δ = 0and
A k = η θ ( S k ) .
[Equation]
where K → ∞ , and
[Equation]
D.
Proof.
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Equation]
where µ k =
( S k , A k ) , S 0 ∼ d 0 ( · ) , A k ∼ π θ ( S k , · ) , S k +1 ∼ P ( · | S k , A k ) , for k = 0 · · · K .
D.5 Proof of Corollary
Proof.
In Corollary 2 and Corollary 1, we have
[Equation]
when K → ∞ .[32] proved that
[Equation]
31) where δ is the standard deviation of the Gaussian noise of stochastic policy π θ .
Therefore,
[Equation]
E Conventional Actor-Critic Algorithms for Deep Reinforcement Learning
The gradient of (2) is [33]
[Equation]
Since Q θ in (33) is unknown [38] ( the stationary distribution d θ is unknown), one can plug in a criticnetwork with parameter ϕ as an estimator of Q θ and obtain
[Equation]
[Equation]
where d S ,θ ∈ R |S||A|× 1 + denotes the stationary distribution over the states instead of state-actionpairs.(34) is a bi-level optimization problem [7], and a natural solution is an iterative algorithm thatalternates between estimating Q ϕ with parameter ϕ and improving policy π θ with parameter θ .
Therefore, a family of actor-critic algorithms are proposed with following objective functions:
[Equation] [Equation]
The gradient of (35) can be estimated as follows
[Equation]
The parameters ϕ and θ are updated as follows:
[Equation]
F
Stationary Deterministic Policy Gradient Algorithm with H-term
For completeness, we present the details of the deterministic actor-critic algorithm with H-term.
Algorithm do 6:
Take action a t =
η θ s
) + ϵ ,
∼ N δ
Execute action a t , receive reward r t , and observe new state s t + 8:
Store a transition
D
9: end 10:
Store
D do 12:
Randomly sample mini batch
B
Randomly sample a mini batch of B trajectories length
K j j
Update critic method 15:
Update actor network as
←
J
−
16: end 17: end
We apply the proposed Hamiltonian equation (7) to regularize the actor network.
Specifically, H ′ ( θ )in (7) is added to the actor’s objective with weight λ > 0 .
The objective functions of actor and criticnetworks become:
[Equation]
The gradient of (38) is
[Equation]
[Equation]
To estimate ∇ θ H ′ ( θ ) , the Monte Carlo gradient estimator in (11) is used.
Therefore, (39) and (40)can be estimated as follows:
[Equation]
[Equation]
Hyperparameters and More
[Table] [Table]
G.2 More Results
Fig.
6 shows the H-value (average over 20 runs) during the training process, which verified that thetrained agents have converged to policies with small H-values.
4 #samples 1e
H v a l u e s
HalfCheetah 0.
2.0 #samples 1e
H v a l u e s
Ant 0.
2.0 #samples 1e 1.
H v a l u e s
Humanoid   3 #samples 1e
H v a l u e s
Hopper
[Table]
#samples 1e 0.
H v a l u e s
Swimmer 0.
2.0 #samples 1e
H v a l u e s
Walker2D
PPO
PPO + H, K = PPO + H, K =
PPO + H, K =
4 #samples 1e 1k 2k 3k 4k 5k
C u m u l a t
i v e r e w a r d s
HalfCheetah
K = 8 K =
K =    4 #samples 1e 1k 2k 3k 4k 5k
C u m u l a t
i v e r e w a r d s
Ant
K = 8 K =
K =0.
2.0 #samples 1e 2k 4k 6k 8k 10k 12k
C u m u l a t
i v e r e w a r d s
Humanoid
K = 8 K =
K =   3 #samples 1e 1k 2k 3k 4k 5k
C u m u l a t
i v e r e w a r d s
Hopper
K = 8 K =
K =    8 #samples 1e
C u m u l a t
i v e r e w a r d s
Swimmer
K = 8 K =
K =0.
2.0 #samples 1e 1.5k 3k 4.5k 6k 7.5k 9k
C u m u l a t
i v e r e w a r d s
Walker2D
K = 8 K =
K =
Tasks
Policies
PPO
PPO+H ( K = 8 )
PPO+H ( K = 16 )
PPO+H ( K = 24 ) running    flipping    diving
HalfCheetah balancing
[Table]
Fig.
7 shows more performance of the PPO+H algorithm, for K = 8 , 16 , 24 .
We run each experimentwith 20 random seeds and each run we test 100 episodes.
To verify the hypothesize that smaller replay buffer hurts the performance, we rerun the trials of
K = 8 , 16 with a replay buffer size [Table]
H.1 Hamiltonian Policy Network
Since Hamiltonian equation in (7) is a functional of policy π θ , a natural question would be: canwe use the Hamiltonian equation replace existing Bellman’s equation (3) or the policy gradient’sobjective function (2)?
As a verification, we test the capability of Hamiltonian equation in (7) as a loss function to train apolicy network.
The algorithm is first given as follows.
do 6:
Select action a
Execute action a t , receive reward r t , and observe new state s t
8: end 9:
Store a trajectory τ of length T in D 10: for g = 1 , · · · , G do 11:
Randomly sample mini-batch
B
Update pocliy network
13: end 14: end
In Alg. 3, an agent interacts with an environment and updates its policy network.
The algorithm has
M episodes and each episode consists of a (Monte Carlo) simulation process and a learning process(gradient estimation) as follows:
• During the (Monte Carlo) simulation process (lines 5 - 9 of Alg. 3), an agent takes action a taccording to a policy π θ ( ·| s t ) , t = 0 , · · · , T − 1 , generating a trajectory of T steps/transitions.
Then, the full trajectory τ = ( s 0 , a 0 , r 0 , s 1 , · · · , s T − 1 , a T − 1 , r T − 1 , s T ) is stored in replay buffer
D .
• During the learning process ( G ≥ 1 updates in one episode) (lines 10 - 12 of Alg. 1), a mini-batch of
B trajectories (of length K ) { τ j = ( s j 0 , a j 0 , r j 0 , s j 1 , · · · , s jK − 1 , a jK − 1 , r jK − 1 , s jK ) }
Bj =1 are sampledfrom D , respectively.
The policy network is updated by a Monte Carlo gradient estimator over B
trajectories.
Implementation of replay buffer D .
After a full trajectory τ of length T is generated, it is partitionedinto T − K + 1 trajectories of length K .
We rank them according to the cumulative reward andstore the top portion, say 80% , into a new replay buffer D (line 9 of Alg. 3).
We randomly sample amini-batch of B trajectories from D (line 11 of Alg. 3) to compute the H-term.
H.2 Frozenlake Task
Environment : Frozenlake 8 × 8 , a game in OpenAI Gym.
Rules : As shown in Fig.
8 (left), the Frozenlake task has 8 × 8 states with 4 optional actions to movearound.
The agent needs to go from the start point and find the way to the destination in limited steps.
There are 8 holes which can cause the agent to fail the game.
Experiment settings : We take Deep Q-learning (DQN) [27] as our baseline and use the implementation from the ElegantRL library.
We use a 4 -layer fully connected neural network as the deep policynetwork both in DQN and DHN.
We use the Adam optimizer with a learning rate 1 × 10 − 3 and abatch size 100 .
Evaluation : We evaluate the performance of policy by computing the success rate, in which we use50 agents to walk 100 steps and compute the rates of agents who successfully arrive the destination.
Results for the Frozenlake task : Fig.
9 (left) shows the success rate of agents with increasing thenumber of transitions learned by the network.
compared with DQN, DHN has a more stable trainingprocess.
It is easy for DQN to quickly obtain a good policy to win the game.
But with increasing thenumber of transitions fed to the network, the performance of DQN shows a large and frequent shockwhile the performance of DHN shows the strong stability.
H.3 Gridworld Task
[Figure]
Environment : a Gridworld of size 10 × 10 , a game available in our code.
[Figure]
Rules : As shown in the Fig.
8 (right), the Gridworld has 10 × 10 states with 4 optional actions tomove around.
The agent will initialize at a random locations and it needs to find the smiley as manyas possible which has 10 reward in turn.
It should be noted that there are some endpoints which maycause the agent game over and some transfer-points which transfer the agent to certain location.
Experiment settings and evaluation : Both the experiment settings and evaluation method are thesame with that on Frozenlake 8 × 8 game.
Results for the Gridworld task : Fig.
9 (rigt) shows the mean reward obtained by the agents withincreasing the training time.
Compared with DQN, DHN has a faster training process.
It only needsmassive random parallel samples of trajectories and do not need any policy for guided sampling while
DQN needs guided exploration in the training process which costs a large time consumption.