DeepTOP: Deep Threshold-Optimal Policy for MDPs and RMABs
Abstract
We consider the problem of learning the optimal threshold policy for control problems.
Threshold policies make control decisions by evaluating whether an element of the system state exceeds a certain threshold, whose value is determined by other elements of the system state.
By leveraging the monotone property of threshold policies, we prove that their policy gradients have a surprisingly simple expression.
We use this simple expression to build an o ﬀ -policy actor-critic algorithm for learning the optimal threshold policy.
Simulation results show that our policy signiﬁcantly outperforms other reinforcement learning algorithms due to its ability to exploit the monotone property.
In addition, we show that the Whittle index, a powerful tool for restless multi-armed bandit problems, is equivalent to the optimal threshold policy for an alternative problem.
This observation leads to a simple algorithm that ﬁnds the Whittle index by learning the optimal threshold policy in the alternative problem.
Simulation results show that our algorithm learns the Whittle index much faster than several recent studies that learn the Whittle index through indirect means.
Introduction
This paper considers a class of control policies, called threshold policies , that naturally arise in many practical problems.
For example, a smart home server may only turn on the air conditioner when the room temperature exceeds a certain threshold and a central bank may only raise the interest rate when inﬂation exceeds a certain threshold.
For such problems, ﬁnding the optimal control policies can be reduced to ﬁnding the appropriate thresholds given other factors of the system, such as the number of people in the room in the smart home server scenario or the unemployment rate and the current interest rate in the central bank scenario.
An important feature of threshold policies is that their actions are monotone.
For example, if a smart home server would turn on the air conditioner at a certain temperature, then, all other factors being equal, the server would also turn on the air conditioner when the temperature is even higher.
By leveraging this monotone property, an algorithm aiming to learn the optimal threshold can potentially be much more e ﬃ cient than generic reinforcement learning algorithms seeking to learn the optimal action at di ﬀ erent points of temperature separately.
In order to design an e ﬃ cient algorithm for learning the optimal threshold policy, we ﬁrst formally deﬁne a class of Markov decision processes (MDPs) that admit threshold policies and its objective function.
The optimal threshold policy is then the one that maximizes the objective function.
However, the objective function involves an integral over a continuous range, which makes it infeasible to directly apply standard tools, such as backward-propagation in neural networks, to perform gradient updates.
Surprisingly, we show that, by leveraging the monotone property of threshold policies, the gradient of the objective function has a very simple expression.
Built upon this expression, we propose
Deep Threshold-Optimal Policy (DeepTOP), a model-free actor-critic deep reinforcement learning algorithm that ﬁnds the optimal threshold policies.
We evaluate the performance of DeepTOP by considering three practical problems, an electric vehicle (EV) charging problem that determines whether to charge an EV in the face of unknown ﬂuctuations of electricity price, an inventory management problem that determines whether to order for goods in the face of unknown seasonal demands, and a make-to-stock problem for servicing jobs with di ﬀ erent sizes.
For all problems,
DeepTOP signiﬁcantly outperforms other state-of-the-art deep reinforcement learning algorithms due to its ability to exploit the monotone property.
We also study the notoriously hard restless multi-armed bandit (RMAB) problem.
We show that the
Whittle index policy, a powerful tool for RMABs, can be viewed as an optimal threshold policy for an alternative problem.
Based on this observation, we deﬁne an objective function for the alternative problem, of which the Whittle index is the maximizer.
We again show that the gradient of the objective function has a simple expression.
This simple expression allows us to extend DeepTOP for the learning of the Whittle index.
We compare this DeepTOP extension to three recently proposed algorithms that seek to learn the optimal index policies through other indirect properties.
Simulation results show that the DeepTOP extension learns much faster because it directly ﬁnds the optimal threshold policy.
The rest of the paper is organized as follows.
Section 2 deﬁnes the MDP setting and threshold policies.
We present the DeepTOP algorithm for MDP in Section 3.
We then discuss how the Whittle index policy for RMABs can be viewed as a threshold policy in Section 4 and develop a DeepTOP extension for learning it in Section 5.
We show DeepTOP’s performance results for MDPs and RMABs in
Section 6, and give related works in Section 7 before concluding.
Threshold Policies for MDPs
Consider an agent controlling a stochastic environment E described as an MDP E =
( S , A , R , P , γ ), with state space S ,
binary action space A :
= { 0 , 1 } , reward function R : S ×
A → Ω , transition dynamics P : S ×
A × S → (cid:146) , and discount factor γ ∈ [0 , 1), where (cid:146) is the set of real numbers and Ω is the set of random variables.
At each timestep t , the agent picks an action a t ∈ A for the current state s t .
The state s t ∈ S =
(cid:146) × V has two components: a scalar state λ t ∈ (cid:146) , and a vector state v t ∈ V , where V is a discrete set of vectors.
We assume the environment state is fully observable.[-
-]
Given the state-action pair ( s t , a t ), the MDP generates a reward r t following the unknown random variable R ( s t , a t ), and a random next state s t + 1 = ( λ t + 1 , v t + 1 ) following the unknown distribution P .
We use ¯ r ( λ, v , a ) :
=
E [ R (( λ, v ) , a )] to denote the unknown expected one-step reward that can be obtained for the state-action pair ( λ, v , a ).
A threshold policy is one that deﬁnes a threshold function µ : V → (cid:146) mapping each vector state to a real number.
The policy then deterministically picks a t = 1 ( µ ( v t ) > λ t ), where 1 ( · ) is the indicator function.
There are many applications where it is natural to consider threshold policies and we discuss some of them below.
Example 1.
Consider the problem of charging electric vehicles (EV).
When an EV arrives at a charging station, it speciﬁes its demands for charge and a deadline upon which it will leave the station.
The electricity price changes over time following some random process.
The goal of the operator is to fulﬁll the EV’s requirement with minimum cost.
In this problem, we can model the system by letting the scalar state λ t be the current electricity price and the vector state v t be the remaining charge and time to deadline of the EV.
For this problem, it is natural to consider a threshold policy that deﬁnes a threshold µ ( v t ) as the highest price the operator is willing to pay to charge the
EV under vector state v t .
The operator only charges the vehicle, i.e., chooses a t
= 1 , if λ t < µ ( v t ) .
Example 2.
Consider the problem of warehouse management.
A warehouse stores goods waiting to be sold.
When the number of stored goods exceeds the demand, then there is a holding cost for each unsold good.
On the other hand, if the number of stored goods is insu ﬃ cient to fulﬁll the demand, then there is a cost of lost sales.
The goal of the manager is to decide when to place orders so as to minimize the total cost.
In this problem we can let the scalar state λ t be the current inventory and let the vector state v t be the vector of all factors, such as upcoming holidays, that can inﬂuence future demands.
It is natural to consider a threshold policy where the manager only places a new order if the current inventory λ t falls below a threshold µ ( v t ) based on the current vector state v t .
Example 3.
Consider a smart home server that controls the air conditioner.
Let λ t be − ( current temperature ) and v t be the time of the day and the number of people in the house.
The server should turn on the air conditioner only if the temperature exceeds some threshold determined by v t , or, equivalently, λ t < µ ( v t ) .
Given a threshold policy with threshold function µ ( · ), we can deﬁne the corresponding action-value function by Q µ ( λ, v , a ).
Let ρ µ ( λ (cid:48) , v (cid:48) , λ, v ) be the discounted state distribution when the initial state is ( λ, v ) under the threshold policy to a visited state ( λ (cid:48) , v (cid:48) ).
When the initial state is ( λ, v ), the expected discounted reward under the policy is
[Equation]
Let M be a su ﬃ ciently large constant such that λ t ∈ [ − M , + M ] for all t .
Our goal is to learn the optimal threshold function µ φ ( v ) parametrized by a vector φ that maximizes the objective function
[Equation]
Deep Threshold Optimal Policy for MDPs
In this section, we present a deep threshold optimal policy (DeepTOP) for MDPs that ﬁnds the optimal φ for maximizing K ( µ φ ).
3.1 Threshold Policy Gradient Theorem for MDPs
In order to design DeepTOP, we ﬁrst study the gradient ∇ φ K ( µ φ ).
At ﬁrst glance, computing ∇ φ K ( µ φ ) looks intractable since it involves an integral over λ ∈ [ − M , + M ].
However, we establish the following threshold policy gradient theorem that shows the surprising result that ∇ φ K ( µ φ ) has a simple expression.
Theorem 1.
Given the parameter vector φ , let ¯ ρ ( λ, v ) be the discounted state distribution when the initial state is chosen uniformly at random under the threshold policy. If all vector states v ∈ V have distinct values of µ φ ( v ) , then,
[Equation]
Proof.
Let ¯ ρ t ( λ, v ) be the distribution that the state at time t is ( λ, v ) when the initial state is chosen uniformly at random.
Clearly, we have ¯ ρ ( λ, v ) = (cid:80) ∞ t = 1 γ t − 1 ¯ ρ t ( λ, v ). Given φ , we number all states in V such that µ φ ( v 1 ) > µ φ ( v 2 ) > . . . .
Let M
= + M , M n = µ φ ( v n ), for all 1 ≤ n ≤ |V| , and M |V| + 1 = − M .
Also, let V n be the subset of states { v | µ φ ( v ) > M n } = { v 1 , v 2 , . .
. , v n − 1 } .
Now, consider the interval ( M n + 1 , M n ) for some n .
Notice that, for all λ ∈ ( M n + 1 , M n ), 1 ( µ φ ( v ) > λ ) = 1 if and only if v ∈ V n + 1 .
In other words, for any vector state v , the threshold policy would take the same action under all λ ∈ ( M n + 1 , M n ) [Equation] and we use π n + 1 ( v ) to denote this action.
We then have
[Equation]
where the summation-integration swap in the ﬁrst equation follows the Fubini-Tonelli theorem and the last step follows the Leibniz integral rule. We simplify the ﬁrst two terms in the last step by
[Equation]
Next, we expand the last term in (4)
[Equation]
Note that Q µ φ
[Equation]
λ [Equation]
v
[Equation]
a
[Equation]
= ¯ r
[Equation]
λ
[Equation]
v
[Equation]
a
[Equation]
, where p ( ·|· ) is the transition probability. Hence, ∇ φ Q
[Equation]
Using the same techniques in (4) and (5)
[Equation]
we have
[Equation]
λ [Equation] In the above equation, expanding the last term in time establishes (3).
[Equation] 3.2 DeepTOP Algorithm Design for MDPs
Motivated by Theorem 1, we now present DeepTOP-MDP, a model-free, actor-critic Deep RL algorithm.
DeepTOP-MDP maintains an actor network with parameters φ that learns a threshold function µ φ ( v ), and a critic network with parameters θ that learns an action-value function Q θ ( λ, v , a ).
DeepTOP-MDP also maintains a target critic network with parameters θ (cid:48) that is updated slower than the critic parameters θ .
The purpose of the target critic network is to improve the learning stability as demonstrated in [8, 19].
The objective of the critic network is to ﬁnd θ that minimizes the loss function
[Equation]
(6) where ( s t , a t , r t , s t + 1 ) is sampled under some policy with s t =
( λ t , v t ).
The objective of the actor network is to ﬁnd φ that maximizes
[Equation]
λ, v , 1 ( µ φ ( v ) > λ ) (cid:17) d λ .
In each timestep t , the environment E provides a state s t to the agent.
We set an exploration parameter (cid:15) t ∈ [0 , 1) that takes a random action with probability (cid:15) t .
Otherwise, DeepTOP-MDP calculates µ φ ( v t ) based on v t , and chooses a t = 1 ( µ φ ( v t ) > λ t ).
E generates a reward r t and a next state s t + 1 .
A replay memory denoted by M then stores the transition { s t , a t , r t , s t + 1 } .
After ﬁlling the memory with at least B transitions, DeepTOP-MDP updates the parameters φ, θ, θ (cid:48) in every timestep using a sampled minibatch of size B of transitions { s t k , a t k , r t k , s t k + 1 } , for 1 ≤ k ≤ B .
The critic network uses the sampled transitions to calculate the estimated gradient of L ( θ ):
[Equation]
Similarly, the actor network uses the sampled transitions and Equation (3) to calculate the estimated gradient:
[Equation]
Algorithm
Randomly select initial actor network parameters φ and critic network parameters θ .
Set target critic network parameters θ (cid:48) ← θ , and initialize replay memory M .
for timestep t = 1 , 2 , 3 , . . .
do
Receive state s t =
( λ t , v t ) from environment E .
Select action a t = 1 ( µ φ ( v t ) > λ t ) with probability 1 − (cid:15) t .
Otherwise, select action a t randomly.
Execute action a t , and observe reward r t and next state s t + 1 from E .
Store transition { s t , a t , r t , s t + 1 } into M .
Sample a minibatch of B transitions { s t k , a t k , r t k , s t k + 1 } , for 1 ≤ k ≤ B from M .
Update critic network parameters θ using the estimated gradient from Equation (7).
Update actor network parameters φ using the estimated gradient from Equation (8).
Soft update target critic parameters θ (cid:48) : θ (cid:48) ← τθ
+ (1 − τ ) θ (cid:48) .
end for Both the critic network and the actor network then take a gradient update step.
Finally, we soft update the target critic’s parameters θ (cid:48) using θ (cid:48) ← τθ + (1 − τ ) θ (cid:48) , with τ < 1.
The complete pseudocode is given in Algorithm 1.
Whittle Index Policy for RMABs
In this section, we demonstrate how the Whittle index policy [32], a powerful tool for solving the notoriously intractable Restless Multi-Armed Bandit (RMAB) problem, can be represented with a set of threshold functions.
We ﬁrst describe the RMAB control problem, and then deﬁne the Whittle index function.
An RMAB problem consists of N arms.
The environment of an arm i , denoted as E i , is an MDP with a discrete state space s i , t ∈ S i , and a binary action space a i , t ∈
A : = { 0 , 1 } , where a i , t = 1 means that arm i is activated, and a i , t = 0 means that arm
i is left passive at time t .
Given the state-action pair ( s i , t , a i , t ), E i generates a random reward r i , t and a random next state s i , t + 1 following some unknown probability distributions based on ( s i , t , a i , t ).
Here we also use ¯ r i ( s i , a i ) to denote the unknown expected one-step reward that can be obtained for the state-action pair ( s i , a i ).
A control policy over all arms takes the states ( s 1 , t , s 2 , t , . . .
, s N , t ) as input, and activates V out of
N arms in every timestep.
Solving for the optimal control policy for RMABs was proven to be intractable [21], since the agent must optimize over an input state space exponential in N .
To circumvent the dimensionality challenge, the Whittle index policy assigns real values to an arm’s states using a Whittle index function for each arm W i : S i → (cid:146) .
Based on the assigned Whittle indices (cid:0) W 1 ( s 1 , t ) , W 2 ( s 2 , t ) , . . .
, W N ( s N , t ) (cid:1) , the Whittle index policy activates the V highest-valued arms out of N arms in timestep t , and picks the passive action for the remaining arms.
4.1 The Whittle Index Function as The Optimal Threshold Function
To deﬁne the Whittle index and relate it to threshold functions, let us ﬁrst consider an alternative control problem of a single arm
i as environment E i with activation cost λ .
In this problem, the agent follows a control policy that determines whether the arm is activated or not based on its current state s i , t .
If the policy activates the arm, then the agent must pay an activation cost of λ .
Hence, the agent’s net reward at timestep t is deﬁned as r i , t − λ a i , t .
We now consider applying threshold policies for this alternative control problem.
A threshold policy deﬁnes a threshold function µ i : S i → (cid:146) that maps each state to a real value.
It then activates the arm if and only if µ i ( s i , t ) > λ , i.e., a i , t = 1 ( µ i ( s i ) > λ ).
The value of µ i ( s i , t ) can therefore be viewed as the largest activation cost that the agent is willing to pay to activate the arm under state s i , t .
To characterize the performance of a threshold policy with a threshold function µ i ( · ), we let ρ µ i ,λ ( s [-
-](cid:48)
i , s i ) be the discounted state distribution, which is the average discounted number of visits of state s (cid:48)
i when the initial state is s i under the threshold policy and λ .
When the initial state is s i , the expected discounted net reward under the threshold policy is
[Equation]
The performance of the threshold policy under a given λ is deﬁned as J i ,λ ( µ i ) :
= (cid:80) s i ∈S i Q
i ,λ (cid:0) s i , 1 ( µ i ( s i ) > λ ) (cid:1) .
The Whittle index of this arm is deﬁned as the function µ i ( · ) whose corresponding threshold policy maximizes J i ,λ ( µ i ) for all λ :
Deﬁnition 1.
(Whittle Index)
If there exists a function µ i : S i → (cid:146) such that choosing 1 ( µ i ( s i ) > λ ) maximizes J i ,λ ( µ i ) for all λ ∈ ( −∞ , + ∞ ) , then we say that µ i ( s i ) is the Whittle index W
i ( s i ) 1 .
We note that, for some arms, there does not exist any function µ i ( s i ) that satisﬁes the condition in
Deﬁnition 1.
For such arms, the Whittle index does not exist.
We say that an arm is indexable if it has a well-deﬁned Whittle index function.
Deﬁnition 1 shows that ﬁnding the Whittle index is equivalent to ﬁnding the optimal µ i ( · ) that maximizes J i ,λ ( µ i ) for all λ ∈ ( −∞ , + ∞ ).
Parameterizing a threshold function µ φ i i ( · ) by parameters φ i and letting M be a su ﬃ ciently large number such that µ φ i i ( s i ) [-
-]∈
( − M , + M ) for all s i and φ i , we aim to ﬁnd the optimal φ i for maximizing the objective function
[Equation]
Deep Threshold Optimal Policy for RMABs
To design a DeepTOP variant for RMABs, we ﬁrst give the gradient of the objective function.
Theorem 2.
Given the parameter vector φ i , let ¯ ρ λ ( s i ) be the discounted state distribution when the initial state is chosen uniformly at random and the activation cost is λ . If all states s i ∈ S i have distinct values of µ φ
i i ( s i ) , then,
[Equation] [Equation]
Proof.
The proof is similar to that of Theorem 1.
For completeness, we provide it in Appendix A. (cid:3)
We note that Theorem 2 does not require the arm to be indexable.
Whether an arm is indexable or not, using Theorem 2 along with a gradient ascent algorithm will ﬁnd a locally-optimal φ i that maximizes
K i ( µ φ i i ).
When the arm is indexable, the resulting threshold function µ φ
i i is the Whittle index function.
Using the gradient result from Equation (11), we present the algorithm DeepTOP-RMAB for ﬁnding the optimal parametrized threshold functions µ φ i i for arms i = 1 , 2 , . . .
, N .
The training method is similar to the MDP version, except for two important di ﬀ erences.
First, the training of each arm is done independently from others.
Second, the value of λ is an artiﬁcial value that only exists in the alternative problem but not in the original RMAB problem.
Similar to DeepTOP-MDP, we maintain three network parameters for each arm i : actor φ i , critic θ i , and target-critic θ (cid:48) i .
The critic network parametrizes the action-value function, and is optimized by minimizing the loss function L i ( θ i ) : = [Equation] (cid:90)
[Equation]
with ( s i , t , a i , t , r i , t , s i , t + 1 ) sampled under some policy.
In each timestep t , each arm environment E
i provides its current state s i , t to the agent.
For each arm i = 1 , 2 , . . .
, N , DeepTOP-RMAB calculates the state value µ φ i
i ( s i , t ) with the arm’s respective actor network parameters φ i .
Given an exploration parameter (cid:15) t ∈ [0 , 1), DeepTOP-RMAB activates the V arms with the largest µ φ i i ( s i , t ) with probability 1 − (cid:15) t , and activates V randomly selected arms with probability (cid:15) t .
Based on the executed actions, each arm provides a reward r i , t and the next state s i , t + 1 .
An arm’s transition { s i , t , a i , t , r i , t , s i , t + 1 } is then stored in the arm’s memory denoted by M i .
After ﬁlling each arm’s memory with at least B transitions, DeepTOP-RMAB updates φ i , θ i , and [-✓ 
-]θ (cid:48)
i in every timestep.
For each arm i , DeepTOP-RMAB ﬁrst samples a minibatch of size B of transitions { s i , t k , a i , t k , r i , t k , s i , t k + 1 } , for 1 ≤ k ≤ B from the memory M i .
It then randomly samples B values[-
-] [ λ i , 1 , λ i , 2 , . . .
, λ i , B ] from the range [ − M , + M ].
Using the sampled transitions and λ values, it estimates the gradient of L i ( θ i ) as
[Equation]
Using the sampled transitions and Equation (11), it estimates the gradient of K i ( µ φ i i ) as
[Equation]
A gradient update step is taken after calculating the actor and critic networks’ gradients.
Finally,
DeepTOP-RMAB soft updates the target critic parameters [-✓ 
-]θ (cid:48)
i using θ (cid:48) i ←
τθ i + (1 − τ ) θ (cid:48)
i , with τ < 1.
The complete DeepTOP-RMAB pseudocode is given in Appendix B.
Simulations
We have implemented and tested both DeepTOP-MDP and DeepTOP-RMAB in a variety of settings.
The training procedure of the two DeepTOP algorithms are similar to that of the DDPG [19] algorithm except for the expression of gradients.
We implemented the DeepTOP algorithms by modifying an open-source implementation of DDPG [12].
All source code can be found in the repository https://github.com/khalednakhleh/deeptop .
6.1 Simulations for MDPs
We evaluate three MDPs, namely, the electric vehicle charging problem, the inventory management problem, and the make-to-stock problem.
EV charging problem.
This problem is based on Yu, Xu, and Tong [34].
It considers a charging station serving EVs.
When an EV arrives at the station, it speciﬁes the amount of charges it needs and a deadline upon which it will leave the station.
The electricity price changes over time and we model it by an Ornstein-Uhlenbeck process [30].
In each timestep, the station decides whether to charge the EV or not.
If it decides to charge the EV, then it provides one unit charge to the EV.
The station then obtains a unit reward and pays the current electricity price.
If the station fails to fully charge the EV by the deadline of the EV, then the station su ﬀ ers from a penalty that is a convex function of the remaining needed charge.
A new EV arrives at the station when the previous EV leaves.
We model this problem by letting the scalar state be the current electricity price and the vector state be the remaining needed charge and time-to-deadline of the current EV.
A threshold policy is one that calculates a threshold based on the EV’s remaining needed charge and time-to-deadline, and then decides to charge the EV if and only if the current electricity price is below the threshold.
Inventory management problem.
We construct an inventory management problem by jointly incorporating a variety of practical challenges, including seasonal ﬂuctuations in demands and lead times in orders, in the literature [28, 15, 10, 27].
We consider a warehouse holding goods.
In each timestep, there is a random amount of demand whose mean depends on the time of the year.
The warehouse can fulﬁll the demand as long as it has su ﬃ cient inventory, and it makes a proﬁt for each unit of sold goods.
At the end of the timestep, the warehouse incurs a unit holding cost for each unit of unsold goods.
The warehouse manager needs to decide whether to order more goods.
When it places an order for goods, there is a lead time of one time step, that is, the goods ordered at timestep t are only available for sale at timestep t + 1.
We model this problem by letting the scalar state be the current inventory and the vector state be the time of the year.
A threshold policy calculates a threshold based on the time of the year and decides to place an order for goods if the current inventory is below the threshold.
Make-to-stock production problem.
This problem is considered in [26].
It studies a system that produces m items with W demand classes and bu ﬀ er size S .
Accepting a class v order leads to a reward R v , as long as there is still room in the bu ﬀ er for the order.
The classes of demands are ordered such that R 1 > R 2 > . . . .
In this problem, the scalar state is the number of accepted but unﬁnished orders and the vector state is the class of the next arriving order.
More details about the three MDPs can be found in Appendix C.
Evaluated policies.
We compare DeepTOP-MDP against DDPG [19] and TD3 [8], two state-ofthe-art o ﬀ -policy and model free deep RL algorithms.
We use open-source implementations of these two algorithms for [12, 7].
We use the same hyper-parameters, including the neural network architecture, learning rates, etc., for all three algorithms.
We also evaluate the Structure-Aware
Learning for Multiple Thresholds algorithm (SALMUT) [26], a reinforcement learning algorithm that ﬁnds the optimal threshold policy.
SALMUT requires the vector states to be pre-sorted by their threshold values.
Hence, SALMUT can only be applied to the make-to-stock production problem.
Details about the training parameters can be found in Appendix D. For the EV charging problem, Yu,
Xu, and Tong [34] has found the optimal threshold policy.
We call the optimal threshold policy the
Deadline Index policy and compare DeepTOP-MDP against it.
Simulations results.
Simulation results of the three MDPs are shown in Figure 1.
The results are the average of 20 independent runs.
Before starting a run, we ﬁll an agent’s memory withtransitions by randomly selecting actions.
We plot the average reward obtained from the previous 100 timesteps, and average them over 20 runs.
In addition, we provide the standard deviation bounds from the average reward.
It can be observed that DeepTOP signiﬁcantly outperforms DDPG, TD3, and SALMUT.
Although the training procedure of DeepTOP is similar to that of DDPG, DeepTOP is able to achieve much faster learning by leveraging the monotone property.
Without leveraging the monotone property, DDPG and
TD3 need to learn the optimal policy for each scalar state independently, and therefore have much worse performance.
DeepTOP performs better than SALMUT because DeepTOP directly employs the threshold policy gradient.
SALMUT in contrast approximates threshold policies through randomized policies since it can only handle continuous and di ﬀ erentiable functions.
We believe this might be the reason why DeepTOP outperforms SALMUT. We also note that DeepTOP performs virtually the same as the Deadline Index policy for the EV charging problem in about 2000 timesteps, suggesting that DeepTOP indeed ﬁnds the optimal threshold policy quickly.
We also evaluate DeepTOP for di ﬀ erent neural network architectures in Appendix E, and show that DeepTOP performs the best in all settings.
6.2 Simulations for RMABs
[Figure]
[Figure] and q i . We evaluate two RMABs, namely, the onedimensional bandits from [17] and the recovering bandits from [20]. One-dimensional bandits.
We consider an extension of the RMAB problem evaluated in Killian et al.
Killian et al.
[17] considers the case when each arm is a two-state Markov process.
We extend it so that each arm is a Markov process withstates, numbered as 0 , 1 , . . .
, 99 , as shown in Figure 2 where state 99 is the optimal state.
The reward of an arm depends on the distance between its current state and state 99.
Suppose the current state of arm i is s i , t , then it generates a reward r i , t = 1 − ( s i , t − 99 99 ) 2 .
If the arm is activated, then it changes to state s i , t + 1 = min { s i , t + 1 , 99 } with probability p i .
If the arm is not activated, then it changes to state s i , t + 1 = max { s i , t − 1 , 0 } with probability q i .
In the simulations, we pick the probabilities p i to be evenly spaced depending on the number of arms N from the interval [0 . 2 , 0 . 8]. We set the [Figure] g
[Figure] a r d [Figure] probabilities q i = p i .
We consider that there are N arms and that the agent needs to activate V arms in each timestep.
We evaluate three settings of ( N , V ) =
(10 , 3) , (20 , 5) , and (30 , 6). Recovering bandits.
First introduced in [25], we consider the case that studies the varying behavior of consumers over time.
A consumer’s interest in a particular product falls if the consumer clicks on its advertisement link.
However their interest in the product would recover with time.
The recovering bandit is modelled as an RMAB with each arm being the advertisement link.
The reward of playing an arm is given by a function f (cid:0) min( z , z max ) (cid:1) , with z being the time since the arm was last played. In our experiments, we consider arms with di ﬀ erent reward functions, with the arm’s state being the value min { z , z max } and z max = 100. We also evaluate recovering bandits on three settings of ( N , V ) = (10 , 3) , (20 , 5) , and (30 , 6).
More details can be found in Appendix F. Evaluated policies. We compare DeepTOP-RMAB against three recent studies that aim to learn index policies for RMABs, namely, Lagrange policy Q learning (LPQL) [17], Whittle index based Q learning (WIBQL)
[1], and neural Whittle index network (NeurWIN) [20].
LPQL consists of three steps: First, it learns a Q function for each arm independently.
Second, it uses the Q functions of all arms to determine a common Lagrangian.
Third, it uses the Lagrangian to calculate the index of each arm.
WIBQL is a two-timescale algorithm that learns the Whittle indices of indexable arms by updating Q values on the fast timescale, and index values on the slower timescale.
NeurWIN is an o ﬀ -line training algorithm based on REINFORCE that requires a simulator to learn the Whittle index.
Both LPQL and WIBQL are tabular learning methods which may perform poorly compared to deep RL algorithms when the size of the state space is large.
Hence, we also design deep RL equivalent algorithms that approximate their Q functions using neural networks.
We refer to the Deep
RL extensions as neural LPQL and neural WIBQL.
In all experiments, neural LPQL, neural WIBQL, and NeurWIN use the same hyper-parameters as DeepTOP-RMAB.
For the one-dimensional bandits, it can be shown that the Whittle index is in the range of [ − 1 , 1], and hence we set M = 1. For the recovering bandits, we set M = 10.
Simulation results.
Simulation results are shown in Figures 3 and 4.
It can be observed that DeepTOP achieves the optimal average rewards in all cases.
The reason that neural LPQL performs worse than DeepTOP may lie in its reliance on a common Lagrangian.
Since the common Lagrangian is calculated based on the Q functions of all arms, an inaccuracy in one arm’s Q function can result in an inaccurate Lagrangian, which, in turn, leads to inaccuracy in the index values of all arms.
Prior work [17] has already shown that WIBQL performs worse than LPQL.
Hence, it is not surprising that neural WIBQL performs worse than both neural LPQL and DeepTOP.
NeurWIN performs worse than DeepTOP because it is based on REINFORCE and therefore can only apply updates at the end of each minibatch of episodes.
We also evaluate DeepTOP for di ﬀ erent neural network architectures and the results are shown in Appendix G for the one-dimensional bandits and Appendix H for the recovering bandits. [Figure] v e r a g
e
R e w a r d
[Figure]
(a) N = 10 .
V = 3 .
(b) N = 20 .
[Figure]
Related Work
Threshold policies have been analysed for many decision-making problems formed as MDPs.
[11] examined the residential energy storage under price ﬂuctuations problem, and proved the existence [Figure] a [Figure]
Neural
[Figure] Neural WIBQL Tabular [Figure]
Tabular
[Figure] of optimal threshold policies for minimizing the cost. [5] proved that MDPs with a convex and piecewise linear cost functions admit an optimal threshold policy. [24] shows the existence of an optimal threshold policy for energy arbitrage given degrading battery capacity, with [2] using the
REINFORCE algorithm [33] to learn a trading policy with price thresholds for intraday electricity markets.
[14] considered mean ﬁeld games in a multi-agent MDP setting, and characterized individual agent strategy with a threshold policy when the mean game admits a threshold policy.
More recently, [31] studies ﬁnding a job assigning threshold policy for data centers with heterogeneous servers and job classes, and gave conditions for the existence of optimal threshold policies.
[35] proposed a distributed threshold-based control policy for graph traversal by assigning a state threshold that determines if the agent stays in or leaves a state.
For minimizing the age of information in energy-harvesting sensors, [4] used the ﬁnite-di ﬀ erence policy gradient [23] to learn a possibly sub-optimal threshold policy in the average cost setting.
[13] proposed an RL-based threshold policy for semi-MDPs in controlling micro-climate for buildings with simulations proving e ﬃ cacy on a single-zone building.
[29] used the Deep Q-network RL algorithm for selecting alert thresholds in anti-fraud systems with simulations showing performance improvements over static threshold policies.
[26] described the SALMUT RL algorithm for exploiting the ordered multi-threshold structure of the optimal policy with SALMUT implementations in [16] for computing node’s overload protection.
In contrast to these works, DeepTOP-MDP is applicable to any MDP that admits threshold policies.
In learning the Whittle index policy for RMABs, [6] proposed a Q-learning heuristic called the Q
Whittle Index Controller (QWIC) which may not ﬁnd the Whittle indices even when the training converges.
[20] describes a Deep RL algorithm called NeurWIN for learning the Whittle index of a restless arm independently of other arms.
However, NeurWIN requires a simulator to train the neural networks.
Some recent studies, such as [1, 3, 17], proposed various online learning algorithms that can ﬁnd Whittle index when the algorithms converge.
These algorithms rely on some indirect property of the Whittle index which explains why they converge slower than DeepTOP.
Conclusion and Future Work
In this paper, we presented DeepTOP: a Deep RL actor-critic algorithm that learns the optimal threshold function for MDPs that admit a threshold policy and for RMAB problems.
We ﬁrst developed the threshold policy gradient theorem, where we proved that a threshold function has a simple to compute gradient.
Based on the gradient expressions, we design the DeepTOP-MDP and DeepTOP-RMAB algorithm variants and compare them against state-of-the-art learning algorithms.
In both the MDP and RMAB settings, experiment results showed that DeepTOP exceeds the performance of baselines in all considered problems. A promising future direction is to extend DeepTOP to threshold policies with multiple actions.
For example, the Federal Reserve needs to decide not only whether to raise interest rate, but also the amount of rate hike.
This material is based upon work supported in part by NSF under Award Number ECCS-2127721, in part by the U.S. Army Research Laboratory and the U.S. Army Research O ﬃ ce under Grant Number
W911NF-22-1-0151, and in part by O ﬃ ce of Naval Research under Contract N00014-21-1-2385.
Portions of this research were conducted with the advanced computing resources provided by Texas
A&M High Performance Research Computing.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] see Section 6 for simulations’ results.
(b) Did you describe the limitations of your work?
[Yes] described in Section 2.
Our algorithm is only applicable to MDPs that admit a threshold policy.
(c) Did you discuss any potential negative societal impacts of your work?
[N / A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] see Sectionand Section 4 for problem deﬁnitions and assumptions.
(b) Did you include complete proofs of all theoretical results?
[Yes] see Section 3 and Appendix A for proofs.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] code and run instructions are available in the supplemental material.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[Yes] see Appendix D. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] we averaged the results over 20 runs and plotted the standard deviation from the average.
See Section 6 for results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] see Appendix D. 4.
If you are using existing assets (e.g., code, data, models) or curating / releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] we cite and use open-source code for the baselines: DDPG, TD3, LPQL, and WIBQL.
(b) Did you mention the license of the assets?
[Yes] licenses are mentioned in Appendix D. (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] our code is included with the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using / curating?
[N / A] (e) Did you discuss whether the data you are using / curating contains personally identiﬁable information or o ﬀ ensive content?
[N / A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N / A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N / A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N / A]
Appendices For DeepTOP: Deep Threshold-Optimal Policy for MDPs and RMABs A Threshold Optimal Policy Gradient Theorem Proof for RMABs Proof. Let ¯ ρ λ t ( s i ) be the distribution that the state at time t is s i when the initial state is chosen uniformly at random.
{+We have ¯ ρ λ t ( s i ) = (cid:80) ∞ t = 1 γ t − 1 ¯ ρ t ,λ ( s i ). Given φ i , we number all states in S i such that µ φ i i ( s 1 i ) > µ φ i i ( s 2 i ) > . . . . Let M 
+}
= + M , M n = µ φ i i
( s ni ), for all 1 ≤ n ≤ |S i | , and M |S i | + 1 = − M . Also, let S ni be the subset of states { s i | µ φ i i ( s i ) > M n } = { s 1 i , s 2 i , . . . , s n − 1 i } . Now, consider the interval ( M n + 1 , M n ) for some n .
{+For all λ ∈ ( M n + 1 , M n ), 1 ( µ φ i i ( s i ) > λ ) = 
 if and only if s i ∈ S n + 1 i . In other words, the threshold policy takes the same action under all λ ∈ ( M n + 1 , M n ), and we use π n + 1 ( s i ) to denote this policy.+}
We then have ∇ φ i K i ( µ φ i i )
= ∇ φ i (cid:90) λ
= [Equation] M λ = [Equation]
M (cid:88) s i ∈S
i Q i , [Equation] ( s
[Equation]
( µ
[Equation]
i
[Equation]
( s
[Equation]
[Equation]
d
[Equation] = (cid:88) s i ∈S i ∇ φ i (cid:90) λ
=+ M λ = − M Q
i , [Equation] [Equation] i ,
[Equation]
[Equation]
φ i
[Equation]
[Equation]
[Equation]
d λ = (cid:88) s i ∈S i |S i | (cid:88) n =∇ φ i (cid:90) λ = M n λ = M n
[Equation]
1 Q
i
[Equation]
where the summation-integration swap in the ﬁrst equation follows the Fubini-Tonelli theorem and the last step follows the Leibniz integral rule.
We simplify the ﬁrst two terms in the last step by
[Equation]
Next, we expand the last term in (15)
[Equation]
Note that Q i ,λ ( s i , a i ) =
¯ r
( s i , a i ) + γ (cid:82) λ (cid:48) =
+ M λ (cid:48)
= − M (cid:80) s (cid:48)
i p ( s (cid:48)
i | s i , a i ) Q i ,λ (cid:48)
( s (cid:48)
i , 1 ( µ φ i i ( s (cid:48) i ) > λ (cid:48) ))
d λ (cid:48) , where p ( ·|· ) is the transition probability.
Hence, ∇ φ i Q i ,λ ( s i , 1 ( µ φ i i ( s i ) > λ ))
= ∇ φ i γ (cid:82) λ (cid:48)
=
+ M λ (cid:48)
= − M (cid:80) s (cid:48)
i p ( s (cid:48)
i | s i , a i ) Q i ,λ (cid:48)
( s (cid:48)
i , 1 ( µ φ i i ( s (cid:48) i ) > λ (cid:48) ))
d λ (cid:48) .
Using the same techniques in (15) and (16), we have (cid:88) s i ∈S
i |S i | (cid:88) n =(cid:90) λ =
M n λ =
M n + 1 ∇ φ i Q
i ,λ ( s i , π n + 1 ( s i ))
d λ = (cid:88) s i ∈S i (cid:90) λ
=+ M λ = − M ∇ φ i Q
i ,λ ( s i , 1 ( µ φ i i ( s i ) > λ ))
d λ = γ (cid:88) s i ∈S i (cid:90) λ
=+ M λ = − M (cid:16) ∇ φ i (cid:90) λ (cid:48) =
+ M λ (cid:48)
= − M (cid:88) s (cid:48)
i ∈S i p
( s (cid:48)
i | s i , 1 ( µ φ i i ( s i )
[Equation]
In the above equation, expanding the last term in time establishes (11).
(cid:3)
B DeepTOP-RMAB Algorithm Pseudocode
Algorithm for arm
i = 1 , 2 , . .
. , N do
Randomly select initial parameters for the actor network φ i and critic network θ i .
Set target critic network θ (cid:48) i ← θ i , and initialize replay memory M i . end for for timestep t = 1 , 2 , 3 , . . . do for arm i = 1 , 2 , . .
, N do Receive state s i , t from arm environment E i , and calculate the state value µ φ i i ( s i , t ). end for
With probability 1 − (cid:15) t , activate the V largest-valued arms and keep the remaining arms passive.
Otherwise, randomly activate V arms with the remaining arms left passive.
for arm i = 1 , 2 , . . . , N do Observe reward r i , t and next state s i , t + 1 . Store transition { s i , t , a i , t , r i , t , s i , t + 1 } in memory M i .
Sample a minibatch of B transitions { s i , t k , a i , t k , r i , t k , s i , t k + 1 } , for 1 ≤ k ≤ B from memory M i .
Randomly select B values [ λ i , 1 , λ i , 2 , . .
, λ i , B ], for 1 ≤ k ≤ B from the range [ − M , + M ].
Update arm i ’s critic network using the estimated gradient in Equation (13).
Update arm i ’s actor network using the estimated gradient in Equation (14).
Soft update target critic θ (cid:48)
i network parameters: θ (cid:48)
i ←
τθ i + (1 − τ ) θ (cid:48)
i .
end for end for
C MDP Problems’ Description
EV charging.
The vector state v t =
( C t , D t ) consists of the charging requirement C t , and the time remaining until the vehicle departs the station D t at time t .
In the simulations, we upper-bound the state elements with C ≤ 8 and D ≤ 12.
The scalar state λ t is sampled from an Ornstein-Uhlenbeck process with noise parameter 0 .
15, noise mean 0 . 0, and noise standard deviation 0 .
If the agent chooses to charge the vehicle by selecting action a t = 1, the agent then obtains a reward of 1 − λ , and the MDP transitions to the next state v t
+ 1 =
( C t − 1 , D t − 1).
{+Otherwise for a t = 0, the reward is zero and the MDP transitions to next state v t + 
+}
=
( C t , D t − 1).
If the charging spot is empty at the next timestep (i.e. v t + 1 = (0 , 0)), the environment randomly picks the charge requirement C t + 1 and time until deadline D t + 1 of the next EV vehicle.
For the vehicle occupying the charging station, if it’s charge requirement is not met by the deadline, the agent incurs a penalty of F ( C t )
= 0 .
2( C t ) 2 that is subtracted from reward r t .
The net reward is then r t − F ( C t ).
Inventory management.
The warehouse can store a maximum of 1000 items, and is able to purchase new stock in bulks of 500 items.
The selling price of a single item is set to 20.
The vector state v t is the current market shopping season at time t .
The scalar state λ t is the current warehouse inventory count at time t .
We set 10 di ﬀ erent shopping seasons indexed by b that model the customers’ current demand rate.
The corresponding demand rates for the seasons are 10 di ﬀ erent
Poisson distributions with parameters sin( b π/ 10) · 300 for 0 ≤ b ≤ 9.
If the agent orders items (i.e. a t = 1), it receives a reward equal to the total items’ selling price minus the minimum of remaining inventory count and current demand rate.
{+The next state v t + 1 is then the next market season index v t + 
+}
=
b
+ 1 mod 10.
Otherwise for a t = 0, the agent holds o ﬀ on buying new items, and incurs a holding cost from the remaining unsold items.
The next state is v t + 1 = b +mod 10.
Make-to-stock production.
The environment models a queueing system with m servers serving at rate 1 /µ and a ﬁnite bu ﬀ er with size S .
There are W customer classes each with Poisson mean arrival rate β .
The state at timestep t is ( λ t , v t ), with the scalar state λ t ∈ { 0 , 1 , . . .
, m + S } and the vector state v t ∈ { 1 , 2 , . . .
, W } .
If the agent picks the passive action a t = 0, then the reward is equal to the holding
[Table]
cost h ( λ t ) = − 0 .
1( λ t ) 2 .
For action a t = 1, the agent receives total net reward of R v − h ( λ t ) if the scalar state λ t
= m + S .
Otherwise, the reward is the holding cost − h ( λ t ).
In the simulations, we set the number of servers m = 50, bu ﬀ er size S = 50, number of customer classes W = 50, µ = 4, and arrival rate β = 1 for all customer classes.
The reward R v is chosen to be evenly spaced between 200 and 10 depending on the number of customer classes W .
D Experiments’ Details
For all Deep RL algorithms, we used PyTorch [22] to implement them, with Adam [18] as the optimizer.
We used 10 − 4 as the learning rate for the actor networks, and a learning rate of 10 − 3 for the critic networks.
We also set the initial learning rate of action-value function to 0 .
1 for tabular LPQL and tabular WIBQL.
Tabular WIBQL initial learning rate for updating indices is 0 .
The warmup period has 1000 timesteps used for ﬁlling the memory M with transitions from random actions.
We use a constant (cid:15) t = 0 .
through all timesteps.
A discount factor of γ = 0 .
99 was selected.
All neural network layers were initialized using PyTorch’s default method.
We update parameters using a minibatch size of 64 transitions, with policy updates made at every timestep after the warmup period ends.
The neural networks used for results in Section 6 have two hidden layers with sizes [128 , 128], with the input layer dimension depending on the state size.
Output layer has a dimension of one.
The used code for TD3, tabular LPQL, and tabular WIBQL are licensed under the MIT license.
The
DDPG code we used is licensed under the Apache license 2.0.
All Deep RL algorithms were trained using a computing cluster with 9632 computing cores distributed over 320 nodes.
All algorithms were trained using CPU cores.
E Additional MDP Simulation Results Using Di ﬀ erent Neural Network Architectures
We provide results here for the considered MDP problems for di ﬀ erent neural network architectures.
All other hyperparameters were kept the same as described in Appendix D.
[Figure]
[Figure]
[Figure]
F Recovering Bandits’ Description
The state s i , t is the waiting time since the arm i was last activated, with the maximum waiting time z max set to 100.
If the agent chooses to activate the arm, the arm’s state is reset to 1.
The arm’s reward is provided by the recovering function f
( s i , t ), where if the arm is activated, the reward is the function value at s i , t .
Otherwise a reward of zero is given if the arm is left passive.
The recovering reward function is generated from
[Equation]
[Equation]
=
[Equation]
[Equation]
We use the same Θ =
[ θ 0 , θ 1 ] hyperparameters for setting the arms’ reward classes as used in [20] and provide them in Table 1.
G Additional One-Dimensional Bandits’ Simulation Results Using Di ﬀ erent Neural Network Architectures
We provide results here for the considered RMAB problem for di ﬀ erent neural network architectures.
All other hyperparameters were kept the same as described in Appendix D.
[Figure]
A v e r a g
e R e w a r d
[Figure]
TabularWIBQL
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
v e r a g
e
R e w a r d
[Figure]
[Figure]
r a g e R e w a r d
[Figure]
H Additional Recovering Bandits’ Simulation Results Using Di ﬀ erent Neural Network Architectures
[Figure]
[Figure]
[Figure]