Stationary Deep Reinforcement Learning with Quantum K-spin Hamiltonian Equation
Anonymous Author(s)
AffiliationAddress email
Abstract
A foundational issue in deep reinforcement learning (DRL) is that Bellman’s optimality equation has multiple fixed points —failing to return a consistent one.
Adirect evidence is the instability of existing DRL algorithms, namely, the highvariance of cumulative rewards over multiple runs.
As a fix of this problem, wepropose a quantum K-spin Hamiltonian regularization term (H-term) to help apolicy network stably find a stationary policy, which represents the lowest energyconfiguration of a system.
First, we make a novel analogy between a Markov
Decision Process (MDP) and a quantum K-spin Ising model and reformulate theobjective function into a quantum K-spin Hamiltonian equation, a functional ofpolicy that measures its energy.
Then, we propose a generic actor-critic algorithmthat utilizes the H-term to regularize the policy/actor network and provide Hamiltonian policy gradient calculations.
Finally, on six challenging MuJoCo tasks over20 runs, the proposed algorithm reduces the variance of cumulative rewards by65 . 2% ∼ 85 .
6% compared with those of existing algorithms.
Introduction
Deep reinforcement learning (DRL) [31] algorithms are quite unstable, namely, agents trained withdifferent random seeds may have dramatically different performance.
Existing works [1, 12, 6, 20]reported a high variance over multiple runs, thus it requires to train tens of agents and then pick thebest one.
Such a high variance puts a challenge on reliability and reproducibility [14, 13], limitingthe wider adoption in real-world tasks.
The difficulty of guaranteeing stability lies in a foundational issue that Bellman’s optimality equationhas multiple fixed points [4, 26, 18, 16] —failing to return a consistent one.
Consider MDP exampleswith an terminal state 0 , as shown in Fig.
1 (we adapt dynamic programming examples [4, 26] intoreinforcement learning settings; observational experiments in Section 2.2),• Shortest path problem (deterministic) in Fig.
1(a):
At state 1 , an agent
transits to either state1 or 0 with reward 0 or b , respectively.
Assume the value function for state 0 is V (0)
= 0 .
TheBellman’s optimality equation for state 1 is V (1) =
max { V (1) , b } , where any V (1) ≥ b is afeasible solution.
If initialize V 0 (1) ≥ b , a resulting policy is that an agent at state 1 always transitsback to state 1 ; otherwise, drives to terminal state
(always returns back to itself with reward 0 ).
• Blackmailer’s problem (stochastic) in Fig.
1(b):
At state 1 , a profit maximizing blackmailerdemands a cash amount a ∈ (0 , 1] ; a victim transits to state 1 with probability a or state 0 withprobability 1 − a , respectively.
At state 0 , a victim always refuses to yield, i.e., V (0) = 0 .
TheBellman’s optimality equation for state 1 is V (1) = max a { a +(1 − a ) V (1) } , where any V (1) ≥ 1is a feasible solution.
If initialize V 0 (1) > 1 , the blackmailer’s policy is demanding a → 0 to keepthe victim at state 1 ; otherwise, demanding a = 1 that drives the victim to terminal state 0 .
[Figure]
• Optimal stopping problem (terminating policies) in Fig.
1(c):
In a space R 2 with terminalstate of point 0 , an agent at point x ̸ = 0 moves to either point 0 with negative reward − c orpoint αx with reward −|| x || , respectively, where α ∈ (0 , 1) .
The Bellman’s optimality equationis V ( x ) = max {− c, −|| x || + V ( αx ) } and the optimal policy is to continue inside the sphere ofradius (1 − α ) c and to stop outside.
If add a cone region C within which an agent always receivesa reward − c , a second policy is jumping to point 0 at any point in region C .
The instability problem has been partially addressed, such as ensemble methods [2, 7], regularizationapproaches [33, 8], and baseline-correction approaches [28, 36].
In particular, Generalized Advantage
Estimation (GAE) [28] is a widely used one that significantly reduces the variance of the advantagefunction.
However, they did NOT fix the foundational issue of Bellman’s optimality equation in Fig.1, thus the objective function inherently fails to search for a consistent policy.
For practical usage, weexpect a DRL algorithm stably converges to a certain policy independent of initialization and noises.
As a fix of the problem, we make a novel analogy between an MDP and a quantum K-spin Ising model[22, 11], and reformulate a reward-maximization RL task into an energy-minimization problem,namely, finding the lowest-energy configuration of a quantum spin system.
We hypothesize thata physically stationary policy would have the lowest energy .
Different from our quantum K-spinperspective, several recent papers utilized the (classical) Hamiltonian equation to endow RL agentsthe capability of inductive biases.
For example, [17, 35] used Hamiltonian mechanics to train anagent that learns and respects conservation laws ; [37] applied a Hamiltonian Monte Carlo (HMC)simulator to approximate the posterior action probability; and [24] proposed an unbiased estimatorfor the stochastic Hamiltonian gradient methods for min-max optimization problems.
In this paper, we propose a quantum-inspired K-spin Hamiltonian regularization term (H-term) thathelps policy gradient algorithms stably converge to a physically stationary policy.
Our contributionsare as follows: 1) we derive a novel regularizer, H-term , by a novel reformulation of the RL’s objectiveas a K-spin Hamiltonian equation that measures the energy of a policy; 2) we propose a genericactor-critic algorithm that utilizes the H-term to regularize the policy/actor network and provide
Hamiltonian policy gradient calculations on both deterministic and stochastic cases; and 3) on sixchallenging MuJoCo tasks over 20 runs, we show that the proposed algorithm reduces the variance ofcumulative rewards by 65 . 2% ∼ 85 .
6% compared with those of existing algorithms.
Bellman’s Optimality Equation and Multiple Fixed Points
We describe Bellman’s optimaility equation and provide observational experiments to verify theexistence of multiple policies.
2.1 Bellman’s Optimality Equation
A reinforcement learning (RL) [31] agent interacts with an unknown environment and learns anoptimal policy that maximizes the cumulative reward.
Mathematically, the environment can beformulated as a Markov Decision Process (MDP) with the five-tuple ⟨S , A , P , R, γ ⟩ .
Here S and
A denote the state and action spaces; P : S ×
A → ∆( S ) denotes a transition probability function,where ∆ is a probability simplex; R : S ×
A × S → R denotes a reward function; and γ ∈ (0 , 1]denotes a discount factor.
The objective is to find an optimal policy π ∗ : S → ∆( A ) that maximizes(discounted) expected reward.
Consider a discrete, finite, discounted MDP with infinite horizon, one can define the Q-value functionof a state-action pair ( s, a ) under policy π as follows
[Equation]
where R ( S k , A k , S k +1 ) denotes the immediate reward when taking action A k at state S k and
arrivingat state S k +1 , capital letters denote random variables and lowercase letters denote values.
Theconventional objective function J ( θ ) of reinforcement learning [31] takes the following form
[Equation]
where τ is a trajectory, i.e., τ = ( S 0 , A 0 , · · · ) , and P ( τ | π ) =
d 0 ( s 0 ) · (cid:81) Tk =
P ( S k +1 | s k
[Equation]
a k ) π ( a k | s k ) .
The Bellman equation [31] converts (1) into a recursive form as follows
[Equation]
which expresses the expected reward as a summation of immediate reward R ( s, a ) and discounted future rewards, and the immediate reward R ( s, a ) is defined as R ( s, a )
= (cid:80) s ′ ∈S P ( s ′ | s, a ) R ( s, a, s ′ ) .
The Bellman’s optimality equation [31] is
[Equation]
The optimal policy π ∗ is given by a greedy strategy such that π ∗ =
arg max π Q π ( s, a ) .
2.2 Observational Experiments for Multiple Polices and Physically Stationary Policy
As we pointed out in Fig.
1, theoretically there exists multiple policies.
Here, we perform observational experiments on four challenging MuJoCo tasks [34], namely, Humanoid, Hopper, HalfCheetah,and Ant (details given in Appx.
B.1), which are typical examples of the locomotion control of a robot.
We render the obtained policies over multiple runs and then identify physically stationary ones.
Weobserve various types of moving strategies, as shown in Fig.
2, which verifies that multiple policiesare very common.
For example, the Humanoid agent learns either jumping with a single leg orrunning with two legs, as shown in Fig.
2 (top-left); another interesting example is HalfCheetah, inwhich an agent can run normally or in a flipped manner, as shown in Fig.
2(bottom-left).
Amongthe obtained policies, one can easily identify the physically stationary polices that control the robotmoving forward with a stable gait (defined as gait that does not lead to fall). [Table] [Table] [Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table]  Reinforcement Learning as Quantum K-spin Hamiltonian Equation First, we make a novel analogy between the MDP and a quantum K-spin Ising model, and reformulatethe objective function (2) into a quantum K-spin Hamiltonian equation.
Then, we show that this
Hamiltonian equation helps fix the issue of multiple fixed points in Fig.
3.1 Motivation through Analogy with Quantum K-spin Ising Model
Given the multiple policies in Section 2.2, it is critical to employ a proper criteria to search for aphysically stationary policy.
In Table 1, we make a novel analogy between an MDP and a quantum
K-spin Ising model [22, 11].
The Hamiltonian equation for a quantum K-spin Ising model [22] [11] measures the energy of aparticular configuration of a quantum K-spin system and takes the following form
[Equation]
5) where N is the number of spins in the k -th configuration, σ j k = ± 1 are spin variables, and L j 0 ...
j k isan energy density function for k nearest spins’ configuration ( σ j 0 , . . .
, σ j k ) .
Analogy in Table 1 .
Starting from an analogy between a state-action pair µ k = ( S k , A k ) anda spin j k , we can map an optimal policy π ∗ ( µ k ) ∈ { 0 , 1 } in (4) to a single-qubit spin operatorσ j k ∈ {− 1 , 1 } via π ∗ ( µ k ) ←→ ( 1 µ k − σ µ k ) / 2 , where π θ ( µ k ) denotes the probability of takingaction A k at state S k , following policy π θ .
The energy density function L j 0 ...
j k can be defined as thediscounted cumulative reward on a path ( µ 0 , · · · , µ k − 1 ) of length k , i.e.,
[Equation]
obtained via Monte Carlo simulation)
[Equation]
6) where d 0 ( S 0 ) denotes the distribution of initial state S 0 .
Analogy to the quantum K-spin Ising model,we can derive a functional of policy H ( π µ 0 , ..., π µ K − 1 ) in the context of MDP, which measures theenergy of an RL policy.
In this case, a physically stationary policy satisfies the stationary conditionδH ( π µ 0 , · · · , π µ K − 1 ) /δπ µ = 0 .
3.2 Reformulation into Quantum K-spin Hamiltonian Equation
We formally reformulate (2) into a K -spin Hamiltonian equation
[Equation]
[Equation]
where K → ∞ , the expectation is taken over S 0 ∼ d 0 ( · ) , A 0 ∼ π θ ( S 0 , · ) , and the density function
L µ 0 ,...,µ k is given in (6).
Physical interpretation : Analogy to a quantum K-spin system, H ( θ ) in (7) measures the energy ofpolicy π .
We hypothesize that the energy of a policy is a favorable criteria, since a stationary policywith minimum energy: 1).
achieves a relative high reward independent of the initialization ; and 2).
isrobust to interference/noise in the inference stage .
K-step truncation in practice .
Minimizing (7) is NP-hard [10].
Since γ ∈ (0 , 1) , γ K monotonicallydecreased with look-ahead steps K , therefore, we truncate (7) to finite K terms.
One can showthat these K terms in (7) is a geometric sequence with a truncation error ratio 1 − γ K .
Assuming
− γ K ≤ 1 − ϵ , where ϵ > 0 is small, thus we have the look-ahead steps satisfies K ≥ log γ ϵ .
3.3 Revisiting Examples in Fig.
We elaborate how adding the energy measured by (7) onto each state can help drive to the terminal state (a stationary policy), which fixes the foundational issue of multiple fixed points in Fig.
1 where γ =
We have H = {+
+}
for the terminal state 0 .
• (a) Shortest path problem (deterministic) : H (1) = − (cid:80) ∞ k =1 b = −∞ .
At state 1 , the Bellman’soptimality equation becomes V (1) =
max { V (1) + λH (1) , b } .
Independent of the initial valueV 0 (1) , an agent obtains a policy that always transits back to terminal state 0 .
• (b) Blackmailer’s problem (stochastic) : H (1) =
−∞ .
The Bellman’s optimality equation becomes V (1) = max a { a +(1 − a )( V (1)+ λH (1)) } for state 1 .
For any V 0 (1) < ∞ , the optimal
policy becomes a = 1 that drives to the terminal state 0 .
• (c) Optimal stopping problem (terminating policies) : any policy that takes infinite steps willhave H ( x ) =
−∞ , since at each step number k , there are always trajectories that jump to point 0with reward − c ; and a direct jumping policy will have H ( x ) = − c .
Therefore, adding H ( x ) toeach point x ̸ = 0 will lead to a policy of jumping back to point 0 .
Actor-Critic Algorithm with Quantum K-spin Hamiltonian Regularization
We propose a generic actor-critic algorithm with a H-term, derive two Hamiltonian’s policy gradienttheorems for both deterministic and stochastic cases, and present the Monte Carlo gradient estimation.
4.1 Stationary Actor-Critic Algorithm with H-term
Actor-critic algorithms in reinforcement learning perform a bilevel optimization, namely alternatingbetween approximating a value function and optimizing a policy.
In practice, a critic network withparameter ϕ approximates the Q -value function, and an actor network with parameter θ approximatesthe policy π , details given in Appx.
E.
However, since the critic’s update is governed by the Bellman’soptimality equation, actor-critic algorithms suffer the multiple fixed points problem.
Motivated by Section 3.3, we propose a novel H-term for both deterministic and stochastic actor-criticalgorithms.
Similar to the entropy term in [19], the proposed H-term is an add-on term to regularizethe actor network and help it converge to a stationary policy.
Specifically, the objective functions ofactor and critic networks become:
[Equation]
where a target Q-value is y ( S k , A k )
=
R ( S k , A k ) + γQ ϕ
( S k +1 , A k +1 ) , and λ > 0 is a temperatureparameter.
As an interpretation, the second term − λH ( θ ) in the maximization objective function ofactor network aims to find a minimum energy configuration for the MDP problem, namely, a policyπ that will add a minimum amount of energy to each state’s value function (as in Section 3.3).
New algorithm .
In Alg. 1, an agent interacts with an environment and alternatively updates its actornetwork and critic network.
The algorithm has M episodes and each episode consists of a (Monte
Carlo) simulation process and a learning process (gradient estimation) as follows:
• During the (Monte Carlo) simulation process (lines 5 - 10 of Alg. 1), an agent takes action a t according to a policy π θ ( ·| s t ) , t = 0 , · · · , T − 1 , generating a trajectory of T steps/transitions.
Then, these T transitions are stored into a replay buffer D 1 , while the full trajectory τ =( s 0 , a 0 , r 0 , s 1 , · · · , s T − 1 , a T − 1 , r T − 1 , s T ) is stored in replay buffer D 2 .•
During the learning process ( G ≥ 1 updates in one episode) (lines 11 - 16 of Alg. 1), a minibatch of B transitions { ( s i , a i , r i , s i +1 ) } Bi =1 and a mini-batch of B ′ trajectories (of length K ){ τ j = ( s j 0 , a j 0 , r j 0 , s j 1 , · · · , s jK − 1 , a jK − 1 , r jK − 1 , s jK ) } B ′ j =1 are sampled from D 1 and D 2 , respectively. The critic network is updated by a conventional method, e.g., minimizing the mean squared error(MSE) between an estimated Q-value and a target value. The actor is updated by a Monte Carlogradient estimator over B transitions and B ′ trajectories.
1: Input : learning rate α , temperature λ , look-ahead step K , and parameters M, T, G, B, B ′ 2: Initialize actor network π and critic network Q with parameters θ , ϕ , and replay buffers D 1 , D3: for episode = 1 , · · · , M do 4:
Initialize state s5: for t = 0 , · · · , T − 1 do 6:
Select action a t ∼ π θ ( ·| s t ) 7:
Execute action a t , receive reward r t , and observe new state s t + 8:
Store a transition ( s t , a t , r t , s t +1 ) in D9: end 10:
Store a trajectory τ of length T in D11: for g = 1 , · · · , G do 12:
Randomly sample a mini-batch of B transitions { ( s i , a i , r i , s i +1 ) }
Bi
=1 from D13:
Randomly sample a mini-batch of B ′ trajectories (of length K ) { τ j } B ′ j =1 from D14:
Update critic network using a conventional method 15:
Update actor network as θ ← θ + α (cid:16) ∇ θ (cid:98)
J ( θ ) − λ ∇ θ
(cid:98) H ( θ ) (cid:17) .
16: end 17: end
Two new hyperparameters .
We introduce two hyperparameters: a temperature λ > 0 that is arelative weight of the H-term, and a look-ahead step K ≤ T that defines the horizon of the H-term.
Implementation of replay buffer D 2 .
After a full trajectory τ of length T is generated, it ispartitioned into T − K +1 trajectories of length K .
We rank them according to the cumulative rewardand store the top portion, say 80% , into a new replay buffer D 2 (line 10 of Alg. 1).
We randomlysample a mini-batch of B ′ trajectories from D 2 (line 13 of Alg. 1) to compute the H-term.
4.2 Hamiltonian Policy Gradient and Monte Carlo-based Gradient Estimator
We provide the policy gradient of the quantum K-spin Hamiltonian equation in (7), which are variantsof the well-known policy gradient theorem [31].
We provide detailed derivations in Appx.
D.
Stochastic version .
The Hamiltonian stochastic gradient of (7) w.r.t.
parameter θ is
[Equation]
Deterministic version .
Let η θ ( · ) :
S → A denote a deterministic policy, while we use (cid:101) π θ,δ ( µ ) torepresent that a Gaussian noise (a.k.a, an exploration noise) with standard deviation δ > 0 is added inthe exploration process.
The Hamiltonian deterministic gradient of (7) w.r.t.
parameter θ is
[Equation]
The quantum K-spin Hamiltonian equation in (7) is a reformulation of (2).
We verify the gradientcalculation by showing that: when K → ∞ , the Hamiltonian stochastic and deterministic policygradient ∇ θ H ( θ ) and ∇ θ H ′ ( θ ) are equal to the stochastic policy gradient ∇ θ J ( θ ) in [32] anddeterministic policy gradient ∇ θ J ′ ( θ ) in [30], respectively.
Note that the gradient ∇ θ H ( θ ) in (9) and ∇ θ H ′ ( θ ) in (10) w.r.t.
a distributional parameter θ takes anexpectation form.
Thus, a Monte Carlo gradient estimator is practically useful.
We obtain the Monte
Carlo gradient estimator of ∇ θ H ( θ ) , illustrated in Fig.
3 (right), as follows
[Equation]
As a contrast, we provide the Monte Carlo gradient estimator of REINFORCE’s [32] policy gradient,as illustrated in Fig.
3 (left), as follows
[Equation]
where G
[Equation]
An interesting observation is that both gradient calculations follow a similar pattern as shown in Fig.3.
REINFORCE’s policy gradient [32] in Fig.
3 (left) employs an estimate of future rewards, while
Hamiltonian policy gradient in Fig.
3 (right) uses trajectories in replay buffer D 2 .
Computational complexity : we measure the computation complexity by the times of computing one∇ θ log π θ ( µ ) .
Assume N = B and N ′ =
B ′ , since most DRL algorithms use a mini-batch stochasticgradient decent methods.
REINFORCE’s
[32] policy gradient in (12) takes O ( BT ) computations,while Alg. 1 adds O ( B ′ K ( K + 1) / 2) computations in each gradient update step, thus a totalcomplexity of O ( BT + B ′ K ( K + 1) / 2) .
Performance Evaluation
We evaluate the proposed H-term from four aspects: 1) increasing cumulative reward, 2) reducingvariance, 3) driving to physically stationary policy, and 4) the impact of trajectory length K .
Allexperiments were executed on an NVIDIA DGX-2 server [9].
The server contains 8 A100 GPUs,320 GB GPU memory, and 128 CPU cores running at 2.25 GHz.
5.1 Experimental Settings
Environments (tasks) .
We consider six challenging MuJoCo tasks [34] as in Section 2.2.
The agentlearns to control the locomotion of a robot and aims to move forward as quickly as possible.
Thesetasks have high-dimensional continuous state and action spaces, in which there exists multiple locallyoptimal polices as revealed in Section 2.2.
Compared algorithms .
To evaluate deterministic and stochastic algorithms, we choose Deep
Deterministic Policy Gradient (DDPG) [23] and Proximal Policy Optimization (PPO) [29].
Since the
H-term is compatible with existing variance reduction techniques, we implement the PPO algorithmwith GAE [28].
For a fair comparison, we keep the hyperparameters (listed in Appx.
G) the sameand make sure that the obtained results reproduce existing benchmark tests [12].
Performance metrics .
We employ two performance metrics, the cumulative rewards and variance,while in Section 5.4, we further consider different policies and report the number of convergence.
Werun each experiment with 20 random seeds and in each run we test 100 episodes.
5.2 H-term Increases Cumulative Reward
Experience replay is crucial in improving performance in terms of cumulative reward.
The proposed
H-term in (11) can be viewed as a novel experience replay technique for an actor network.
Here, weadd a compared algorithm, DDPG with Prioritized Experience Replay [27] (DDPG+PER), where
PER prioritizes experience by the TD error to update a critic network.
In Fig.
4, both DDPG+PER and DDPG+H achieve a substantial improvement of cumulative reward.
Inparticular, DDPG+H achieves the highest cumulative rewards in all six tasks, which are comparable to
PPO’s performance in Fig.
It is worthwhile to discuss the advantage of DDPG+H over DDPG+PER.
DDPG+PER utilizes a prioritized replay strategy to obtain a more accurate critic network, however,it is updated via the Bellman equation with the trouble of multiple fixed points.
In contrast, the
H-term in DDPG+H is performed on the actor network.
Our results indicate that an experience replaytechnique on actor network may be much more powerful. [Table] [Figure] t i v
[Figure] r [Figure] t
i
[Figure]
u
[Figure]
u l a t i v e r
[Figure]
w a r
[Figure]
s [Figure] 5.3 H-term Reduces Variance The PPO algorithm with GAE is regarded as the state-of-the-art algorithm in MuJoCo environments.
However, it still has a very high variance (the shaded area) after the policies have converged, asshown in Fig.
We observe that, at the end of training, the PPO algorithm has a variance of 969 . 2 ,1563 . 4 , 2513 . 5 , 905 . 3 , 60 . 7 , 1290 .
1 in the six tasks, respectively.
Such a high variance is mainly dueto the fact that the agent may converge to a random one of multiple policies.
In Fig.
4, the shaded areas of PPO+H ( K = 16 ) are dramatically smaller, i.e., a variance of 228 . 4 ,225 . 4 , 683 . 7 , 184 . 2 , 31 . 6 , and 296 . 8 , respectively.
The variance has been reduced by 65 . 2% ∼ 85 .
6% ,which verifies the effectiveness of the proposed H-term.
In Fig.
4, we also observe that the H-termcan help the DDPG algorithm reduce variances, namely, the variances of DDPG+H are much smallerthan those of vanilla DDPG and DDPG+PER.
Therefore, we may conclude that the H-term guidesthe agent to search for a stationary policy among multiple feasible ones.
More experimental results are given in Appx.
G due to the space limit, including the cases of K = 8and K
= 24 , and the curves of the H-term value during the training process.
One may verify that thestationary policies have relative lower H-values.
5.4 H-term Drives to Physically Stationary Policy
A key question needs to be answered: is H-term really guiding the agent converge to a physicallystationary policy?
Similar to Section 2.2, we perform observational experiments on the six MuJoCotasks and measure the number of convergence to the different policies over 20 runs.
As shown in
[Table] the PPO+H ( K = 16 ) converges to the stationary policywith 20 , 20 , 16 , 20 , 20 , and 16 times, respectively.
From the empirical observation, we find that the
PPO gets stuck in locally optimal policies, failing to find a consistent one.
As expected, PPO+H canconverge to the stationary policy with a substantially higher ratio, which verifies the effectiveness ofthe proposed H-term in finding a physically stationary policy.
5.5 Impact of Trajectory Length K
We investigate the impact of trajectory length K that defines the horizon of the H-term.
Based on (11),we foresee that a large K means a more accurate estimation of ∇ θ H ( θ ) but at a price of computations.
Here, we evaluate the PPO+H with K = 8 , 16 and set the size of replay buffer D 2 to 1 , 000 .
In Table2, we observe that the cumulative reward increases and the variance decreases as K increases from 8to 16 .
However, for the case K = 24 , both metrics get worse due to the out-of-memory issue.
For
K = 24 , we reduce the replay buffer size to 800 to meet the memory limit.
The smaller replay buffersize hurts the diversity of the trajectories and may lead to a small performance drop.
Conclusions
In this paper, we have addressed a foundational issue of existing deep reinforcement learningalgorithms that Bellman’s optimality equation has multiple fixed points .
This issue leads to theinstability of DRL algorithms, puts a challenge on their reliability and reproducibility, and thuslimits the wider adoption in real-world tasks.
As a fix of the problem, we propose a novel H-term, aphysically inspired regularizer, by making a novel analogy between a MDP and a quantum K-spin
Ising model.
Experimentally, we show that the H-term helps DRL algorithms find a stationarypolicy that has the lowest energy in a system and reduces the variance of cumulative rewards by65 . 2% ∼ 85 .
6% compared with those of existing algorithms.
For future works, we will explore the potential of directly training a policy network using (7) as in
Appx.
H, quantum simulator [21] and quantum reinforcement learning [5][15].
It is interesting toapply Monte Carlo estimator for unbiased policy gradient calculations.
We would like to show thatthe proposed H-term can help distributional RL algorithms [3] find a stationary policy, since thedistributional Bellman optimality operator is not a contraction and thus there is also no unique policy.
Broader Impact Statement
In the field of DRL research, this paper points out the issue of multiple fixed points and may attractmore attention from various perspectives.
In terms of applications, this paper moves toward thereliable and reproducible research by improving the stability of existing DRL algorithms.
On the otherhand, the obtained stationary policy may have broad practical impact in many real-world applicationareas, including but not limited to robotics, transportation, and finance.
From an interdisciplinary perspective, our approach lies at the intersection of (quantum) Hamiltonianmechanics and DRL, espcially the explicit analogy between an MDP and a quantum K-spin Isingmodel.
We hope that our study will attract more attention to bring together the strengths of bothapproaches and yield new insights in both fields.
In terms of broader societal impact of this work, we do not see any foreseeable strongly negativeimpacts.
However, this work essentially makes a tradeoff between the computational resource andstability, which may lead to future works with higher computational cost.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately
reflect the
paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] More computational cost.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] May lead to future works with higher computational cost.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] In the appendix.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We reported the variance over 20 random seeds as shadedarea in the figures, and also list it in a table.(d)
Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[-If initialize V 0 (1) = c > 1 , the blackmailer’s policy is demanding a → 0 at the k -th step to keep thevictim stay at state 1 , for any k ≤ K 0 = −⌊ log γ c ⌋ ; and taking a = 1 to transit to terminal state 
-]
[-atthe k -th step, for any k ≥ K 
-]
[-+ 1 ; otherwise initialize V 
-]
[-c ≤ 1 , the result policy is demandingthe maximum a = 1 that drives the victim to a refusal state 
-]
[-Therefore, the policy always drives to terminal state 
-]
[-In equality ( f ), we reorganize (cid:80) K − 1 k =0 (cid:80) ki =0 into (cid:80) K − 1 i =
-]
[-In equality ( h ), we normalize (cid:80) S S 
-]
[-i =
-]
[-• In equality ( i ), we remove the constant (cid:80) S S (cid:80) S S 
-]
[-i =
-]
[-PPO+H ( K = 24 ) running    flipping    diving   
-]
[N/A]