DeepTOP: Deep Threshold-Optimal Policy for MDPs and RMABs
Anonymous Author(s)
A  liation
Address email
Abstract
We consider the problem of learning the optimal threshold policy for controlproblems.
Threshold policies make control decisions by evaluating whether anelement of the system state exceeds a certain threshold, whose value is determinedby other elements of the system state.
By leveraging the monotone propertyof threshold policies, we prove that their policy gradients have a surprisinglysimple expression.
We use this simple expression to build an o ↵ -policy actor-criticalgorithm for learning the optimal threshold policy.
Simulation results show thatour policy signiﬁcantly outperforms other reinforcement learning algorithms dueto its ability to exploit the monotone property.
In addition, we show that the Whittle index, a powerful tool for restless multi-armedbandit problems, is equivalent to the optimal threshold policy for an alternativeproblem.
This observation leads to a simple algorithm that ﬁnds the Whittle indexby learning the optimal threshold policy in the alternative problem.
Simulationresults show that our algorithm learns the Whittle index much faster than severalrecent studies that learn the Whittle index through indirect means.
Introduction
This paper considers a class of control policies, called threshold policies , that naturally arise in manypractical problems.
For example, a smart home server may only turn on the air conditioner whenthe room temperature exceeds a certain threshold, and a central bank may only raise the interest ratewhen inﬂation exceeds a certain threshold.
For such problems, ﬁnding the optimal control policiescan be reduced to ﬁnding the appropriate thresholds given other factors of the system, such as thenumber of people in the room in the smart home server scenario or the unemployment rate and thecurrent interest rate in the central bank scenario.
An important feature of threshold policies is that their actions are monotone.
For example, if a smarthome server would turn on the air conditioner at a certain temperature, then, all other factors beingequal, the server would also turn on the air conditioner when the temperature is even higher.
Byleveraging this monotone property, an algorithm aiming to learn the optimal threshold can potentiallybe much more e  cient than generic reinforcement learning algorithms seeking to learn the optimalaction at di ↵ erent points of temperature separately.
In order to design an e  cient algorithm for learning the optimal threshold policy, we ﬁrst formallydeﬁne a class of Markov decision processes (MDPs) that admit threshold policies and its objectivefunction.
The optimal threshold policy is then the one that maximizes the objective function.
However,the objective function involves an integral over a continuous range, which makes it infeasible todirectly apply standard tools, such as backward-propagation in neural networks, to perform gradientupdates.
Submitted to 36th not distribute.
Surprisingly, we show that, by leveraging the monotone property of threshold policies, the gradientof the objective function has a very simple expression.
Built upon this expression, we propose
Deep Threshold-Optimal Policy (DeepTOP), a model-free actor-critic deep reinforcement learningalgorithm that ﬁnds the optimal threshold policies.
We evaluate the performance of DeepTOP byconsidering three practical problems, an electric vehicle (EV) charging problem that determineswhether to charge an EV in the face of unknown ﬂuctuations of electricity price, an inventorymanagement problem that determines whether to order for goods in the face of unknown seasonaldemands, and a make-to-stock problem for servicing jobs with di ↵ erent sizes.
For all problems,
DeepTOP signiﬁcantly outperforms other state-of-the-art deep reinforcement learning algorithms dueto its ability to exploit the monotone property.
We also study the notoriously hard restless multi-armed bandit (RMAB) problem.
We show that the
Whittle index policy, a powerful tool for RMABs, can be viewed as an optimal threshold policy foran alternative problem.
Based on this observation, we deﬁne an objective function for the alternativeproblem, of which the Whittle index is the maximizer.
We again show that the gradient of theobjective function has a simple expression.
This simple expression allows us to extend DeepTOP forthe learning of the Whittle index.
We compare this DeepTOP extension to three recently proposedalgorithms that seek to learn the optimal index policies through other indirect properties.
Simulationresults show that the DeepTOP extension learns much faster because it directly ﬁnds the optimalthreshold policy.
The rest of the paper is organized as follows.
Section 2 deﬁnes the MDP setting and threshold policies.
We present the DeepTOP algorithm for MDP in Section 3.
We then discuss how the Whittle indexpolicy for RMABs can be viewed as a threshold policy in Section 4 and develop a DeepTOP extensionfor learning it in Section 5.
We show DeepTOP’s performance results for MDPs and RMABs in
Section 6, and give related works in Section 7 before concluding.
Threshold Policies for MDPs
Consider an agent controlling a stochastic environment E described as an MDP E =
( S , A , R , P ,  ), 61 with state space
S , binary action space
A :
= { 0 , 1 } , reward function R : S ⇥ A !
⌦ , transitiondynamics P : S ⇥
A ⇥ S !
í , and discount factor  2 [0 , 1), where í is the set of real numbers and⌦ is the set of random variables.
At each timestep t , the agent picks an action a t 2 A for the currentstate s t .
The state s t 2 S
=
í ⇥ V has two components: a scalar state  t 2 í , and a vector state 65 v t 2 V , where V is a discrete set of vectors.
We assume the environment state is fully observable.
Given the state-action pair ( s t , a t ), the MDP generates a reward r t following the unknown randomvariable R ( s t , a t ), and a random next state s t + 1 =
(  t + 1 , v t + 1 ) following the unknown distribution
P .
We use ¯ r (  , v , a ) :
=
E [ R ((  , v ) , a )] to denote the unknown expected one-step reward that can beobtained for the state-action pair (  , v , a ).
A threshold policy is one that deﬁnes a threshold function µ : V !
í mapping each vector state to areal number.
The policy then deterministically picks a t = 1 ( µ ( v t ) >  t ), where 1 ( · ) is the indicatorfunction.
There are many applications where it is natural to consider threshold policies and we discusssome of them below.
Example 1.
Consider the problem of charging electric vehicles (EV).
When an EV arrives at acharging station, it speciﬁes its demands for charge and a deadline upon which it will leave thestation.
The electricity price changes over time following some random process.
The goal of theoperator is to fulﬁll the EV’s requirement with minimum cost.
In this problem, we can model thesystem by letting the scalar state  t be the current electricity price and the vector state v t be theremaining charge and time to deadline of the EV.
For this problem, it is natural to consider a thresholdpolicy that deﬁnes a threshold µ ( v t ) as the highest price the operator is willing to pay to charge the
EV under vector state v t .
The operator only charges the vehicle, i.e., chooses a t
= 1 , if  t < µ ( v t ) .
Example 2.
Consider the problem of warehouse management.
A warehouse stores goods waiting tobe sold.
When the number of stored goods exceeds the demand, then there is a holding cost for eachunsold good.
On the other hand, if the number of stored goods is insu  cient to fulﬁll the demand,then there is a cost of lost sales.
The goal of the manager is to decide when to place orders so as tominimize the total cost.
In this problem we can let the scalar state  t be the current inventory and letthe vector state v t be the vector of all factors, such as upcoming holidays, that can inﬂuence futuredemands.
It is natural to consider a threshold policy where the manager only places a new order ifthe current inventory  t falls below a threshold µ ( v t ) based on the current vector state v t .
Example 3.
Consider a smart home server that controls the air conditioner.
Let  t be  ( currenttemperature ) and v t be the time of the day and the number of people in the house.
The server shouldturn on the air conditioner only if the temperature exceeds some threshold determined by v t , or,equivalently,  t < µ ( v t ) .
Given a threshold policy with threshold function µ ( · ), we can deﬁne the corresponding action-valuefunction by Q µ (  , v , a ).
Let ⇢ µ (  0 , v 0 ,  , v ) be the discounted state distribution when the initial state is(  , v ) under the threshold policy to a visited state (  0 , v 0 ).
When the initial state is (  , v ), the expecteddiscounted reward under the policy is
[Equation]
Let M be a su  ciently large constant such that  t 2 [  M , + M ] for all t .
Our goal is to learn theoptimal threshold function µ  ( v ) parametrized by a vector  that maximizes the objective function
[Equation]
Deep Threshold Optimal Policy for MDPs
In this section, we present a deep threshold optimal policy (DeepTOP) for MDPs that ﬁnds theoptimal  for maximizing K ( µ  ).
3.1 Threshold Policy Gradient Theorem for MDPs
In order to design DeepTOP, we ﬁrst study the gradient r  K ( µ  ).
At ﬁrst glance, computing r  K ( µ  )looks intractable since it involves an integral over  2 [  M , + M ].
However, we establish thefollowing threshold policy gradient theorem that shows the surprising result that r  K ( µ  ) has asimple expression.
Theorem 1.
Given the parameter vector  , if all vector states v 2 V have distinct values of µ  ( v ) ,then,
[Equation]
Proof.
Given  , we number all states in V such that µ  ( v 1 ) > µ  ( v 2 ) > . . . .
Let M
= + M ,
M n = µ  ( v n ), for all 1  n  |V| , and M |V| + 1 =  M .
Also, let V n be the subset of states { v | µ  ( v ) > 112 M n } = { v 1 , v 2 , . .
. , v n  1 } .
Now, consider the interval ( M n + 1 , M n ) for some n .
Notice that, for all  2 ( M n + 1 , M n ), 1 ( µ  ( v ) > )
= 1 if and only if v 2 V n + 1 .
In other words, for any vector state v , the threshold policy would takethe same action under all  2 ( M n + 1 , M n ), and we use ⇡ n + 1 ( v ) to denote this action.
We then have
[Equation]
|
X n
[Equation] [Equation]
[Equation]
(Leibniz integral rule) =
X
[Equation]
V |V|
[Equation]
=
[Equation]
⇣
[Equation]
Randomly select initial actor network parameters  and critic network parameters ✓ .
Set target critic network parameters ✓✓ , and initialize replay memory M .
for timestep t = 0 , 1 , 2 , . . .
do
Receive state s t =
(  t , v t ) from environment E .
Select action a t = 1 ( µ  ( v t ) >  t ) with probability 1  ✏ .
Otherwise, select action a t randomly.
Execute action a t , and observe reward r t and next state s t + 1 from E .
Store transition { s t , a t , r t , s t + 1 } into M .
Sample a minibatch of B transitions { s t k , a t k , r t k , s t k + 1 } , for 1  k  B from M .
Update critic network parameters ✓ using the estimated gradient from Equation (5).
Update actor network parameters  using the estimated gradient from Equation (6).
Soft update target critic parameters ✓ 0 : ✓⌧✓ + (1  ⌧ ) ✓ 0 .
end for
Since Q µ  ⇣  , v , 1 ( µ  ( v ) >  ) ⌘ is discounted for all vector states, the summation converges to a ﬁnitevalue.
The summation-integration swap in the ﬁrst equation follows from the Fubini-Tonelli theorem.
The last equation follows because Q µ
[Equation]
)  for allv , v n .
⇤
3.2 DeepTOP Algorithm Design for MDPs
Motivated by Theorem 1, we now present DeepTOP-MDP, a model-free, actor-critic Deep RLalgorithm.
DeepTOP-MDP maintains an actor network with parameters  that learns a thresholdfunction µ  ( v ), and a critic network with parameters ✓ that learns an action-value function Q ✓ (  , v , a ).
DeepTOP-MDP also maintains a target critic network with parameters ✓ 0 that is updated slower thanthe critic parameters ✓ .
The purpose of the target critic network is to improve the learning stabilityas demonstrated in [8, 18].
The objective of the critic network is to ﬁnd ✓ that minimizes the lossfunction
[Equation]
where ( s t , a t , r t , s t + 1 ) is sampled under some policy with s t =
(  t , v t ).
The objective of the actornetwork is to ﬁnd  that maximizes
[Equation]
In each timestep t , the environment E provides a state s t to the agent.
We set an exploration parameter 131 ✏ 2 [0 , 1) that takes a random action with probability ✏ .
Otherwise, DeepTOP-MDP calculates µ  ( v t )based on v t , and chooses a t
= 1 ( µ  ( v t ) >  t ).
E generates a reward r t and a next state s t + 1 .
A replaymemory denoted by M then stores the transition { s t , a t , r t , s t + 1 } .
After ﬁlling the memory with at 134 least B transitions, DeepTOP-MDP updates the parameters  , ✓ , ✓ 0 in every timestep using a sampledminibatch of size B of transitions { s t k , a t k , r t k , s t k + 1 } , for 1  k  B .
The critic network uses the 136 sampled transitions to calculate the estimated gradient of
L ( ✓ ):
[Equation]
Similarly, the actor network uses the sampled transitions and Equation (3) to calculate the estimatedgradient:
[Equation]
Both the critic network and the actor network then take a gradient update step.
Finally, we soft updatethe target critic’s parameters ✓ 0 using ✓⌧✓ + (1  ⌧ ) ✓ 0 , with ⌧ < 1.
The complete pseudocode isgiven in Algorithm 1.
Whittle Index Policy for RMABs
In this section, we demonstrate how the Whittle index policy [30], a powerful tool for solving thenotoriously intractable Restless Multi-Armed Bandit (RMAB) problem, can be represented with a set of threshold functions.
We ﬁrst describe the RMAB control problem, and then deﬁne the Whittleindex function.
An RMAB problem consists of N arms.
The environment of an arm i , denoted as E i , is an MDP witha discrete state space s i , t 2 S i , and a binary action space a i , t 2 A :
= { 0 , 1 } , where a i , t = 1 means that 149 arm i is activated, and a i , t = 0 means that arm
i is left passive at time t .
Given the state-action pair( s i , t , a i , t ), E i generates a random reward r i , t and a random next state s i , t + 1 following some unknown 151 probability distributions based on ( s i , t , a i , t ).
Here we also use ¯ r i ( s i , a i ) to denote the unknown expectedone-step reward that can be obtained for the state-action pair ( s i , a i ).
A control policy over all arms takes the states ( s 1 , t , s 2 , t , . . .
, s N , t ) as input, and activates V out of
N arms in every timestep.
Solving for the optimal control policy for RMABs was proven to beintractable [20], since the agent must optimize over an input state space exponential in N .
Tocircumvent the dimensionality challenge, the Whittle index policy assigns real values to an arm’sstates using a Whittle index function for each arm W i : S i !
í .
Based on the assigned Whittleindices
W 1 ( s 1 , t ) , W 2 ( s 2 , t ) , . . .
, W N ( s N , t )  , the Whittle index policy activates the V highest-valued armsout of N arms in timestep t , and picks the passive action for the remaining arms.
4.1 The Whittle Index Function as The Optimal Threshold Function
To deﬁne the Whittle index and relate it to threshold functions, let us ﬁrst consider an alternativecontrol problem of a single arm
i as environment E i with activation cost  .
In this problem, the agentfollows a control policy that determines whether the arm is activated or not based on its current states i , t .
If the policy activates the arm, then the agent must pay an activation cost of  .
Hence, the agent’snet reward at timestep t is deﬁned as r i , t   a i , t .
We now consider applying threshold policies for this alternative control problem.
A threshold policydeﬁnes a threshold function µ i : S i !
í that maps each state to a real value.
It then activates the armif and only if µ i ( s i , t ) >  , i.e., a i , t = 1 ( µ i ( s i ) >  ).
The value of µ i ( s i , t ) can therefore be viewed as thelargest activation cost that the agent is willing to pay to activate the arm under state s i , t .
To characterize the performance of a threshold policy with a threshold function µ i ( · ), we let ⇢ µ i ,  ( s
i , s i )be the discounted state distribution, which is the average discounted number of visits of state s 0 iwhen
the initial state is s i under the threshold policy and  .
When the initial state is s i , the expecteddiscounted net reward under the threshold policy is
[Equation]
The performance of the threshold policy under a given  is deﬁned as J i ,  ( µ i ) :
=
P s i 2 S
i Q i ,
[Equation]
s i , 1 ( µ i ( s i ) >  )  .
The Whittle index of this arm is deﬁned as the function µ i ( · ) whosecorresponding threshold policy maximizes J i ,  ( µ i ) for all  :
Deﬁnition 1.
(Whittle Index)
If there exists a function µ i : S i !
í such that choosing 1 ( µ i ( s i ) >  )maximizes J i ,  ( µ i ) for all  2 ( 1 , + 1 ) , then we say that µ i ( s i ) is the Whittle index W
i ( s i ) 1 .
We note that, for some arms, there does not exist any function µ i ( s i ) that satisﬁes the condition in
Deﬁnition 1.
For such arms, the Whittle index does not exist.
We say that an arm is indexable if ithas a well-deﬁned Whittle index function.
Deﬁnition 1 shows that ﬁnding the Whittle index is equivalent to ﬁnding the optimal µ i ( · ) thatmaximizes J i ,  ( µ i ) for all  2 ( 1 , + 1 ).
Parameterizing a threshold function µ  i i ( · ) by parameters  iand letting M be a su  ciently large number such that µ  i i ( s i )
(  M , + M ) for all s i and  i , we aim toﬁnd the optimal  i for maximizing the objective function
[Equation]
Deep Threshold Optimal Policy for RMABs
To design a DeepTOP variant for RMABs, we ﬁrst give the gradient of the objective function.
Theorem 2.
Given the parameter vector  i , if all states s i 2 S i have distinct values of µ  i
i ( s i ) , then,
[Equation]
Proof.
The proof is similar to that of Theorem 1.
For completeness, we provide it in Appendix A. ⇤
We note that Theorem 2 does not require the arm to be indexable.
Whether an arm is indexable or not,using Theorem 2 along with a gradient ascent algorithm will ﬁnd a locally-optimal  i that maximizes
K i ( µ  i i ).
When the arm is indexable, the resulting threshold function µ
i i is the Whittle index function.
Using the gradient result from Equation (9), we present the algorithm DeepTOP-RMAB for ﬁndingthe optimal parametrized threshold functions µ  i i for arms i = 1 , 2 , . . .
, N .
The training method issimilar to the MDP version, except for two important di ↵ erences.
First, the training of each arm isdone independently from others.
Second, the value of  is an artiﬁcial value that only exists in thealternative problem but not in the original RMAB problem.
Similar to DeepTOP-MDP, we maintain three network parameters for each arm i : actor  i , critic ✓ i ,and target-critic ✓ 0 i .
The critic network parametrizes the action-value function, and is optimized byminimizing the loss function
[Equation]
with ( s i , t , a i , t , r i , t , s i , t + 1 ) sampled under some policy.
In each timestep t , each arm environment E
i provides its current state s i , t to the agent.
For each armi = 1 , 2 , . . .
, N , DeepTOP-RMAB calculates the state value µ  i
i ( s i , t ) with the arm’s respective actornetwork parameters  i .
Given an exploration parameter ✏ 2 [0 , 1), DeepTOP-RMAB activates the Varms with the largest µ  i i ( s i , t ) with probability 1  ✏ , and activates V randomly selected arms withprobability ✏ .
Based on the executed actions, each arm provides a reward r i , t and the next state s i , t + 1 .
An arm’s transition { s i , t , a i , t , r i , t , s i , t + 1 } is then stored in the arm’s memory denoted by M i .
After ﬁlling each arm’s memory with at least B transitions, DeepTOP-RMAB updates  i , ✓ i , and ✓
i inevery timestep.
For each arm i , DeepTOP-RMAB ﬁrst samples a minibatch of size B of transitions{ s i , t k , a i , t k , r i , t k , s i , t k + 1 } , for 1  k  B from the memory M i .
It then randomly samples B values
[  i , 1 ,  i , 2 , . . .
,  i , B ] from the range [  M , + M ].
Using the sampled transitions and  values, it estimatesthe gradient of L i ( ✓ i ) as
[Equation]
Using the sampled transitions and Equation (9), it estimates the gradient of K i ( µ  i i ) as
[Equation]
[Equation]
A gradient update step is taken after calculating the actor and critic networks’ gradients.
Finally,
DeepTOP-RMAB soft updates the target critic parameters ✓
i using ✓ 0 i ⌧✓
[Equation]
+
[Equation]
i , with ⌧ < 1.
The complete DeepTOP-RMAB pseudocode is given in Appendix B.
Simulations
We have implemented and tested both DeepTOP-MDP and DeepTOP-RMAB in a variety of settings.
The training procedure of the two DeepTOP algorithms are similar to that of the DDPG [18] algorithm except for the expression of gradients.
We implemented the DeepTOP algorithms by modifying anopen-source implementation of DDPG [12].
All source code can be found in the repository [redactedfor submission version].
6.1 Simulations for MDPs
We evaluate three MDPs, namely, the electric vehicle charging problem, the inventory managementproblem, and the make-to-stock problem.
EV charging problem.
This problem is based on Yu, Xu, and Tong [32].
It considers a chargingstation serving EVs.
When an EV arrives at the station, it speciﬁes the amount of charges it needs anda deadline upon which it will leave the station.
The electricity price changes over time and we modelit by an Ornstein-Uhlenbeck process [28].
In each timestep, the station decides whether to chargethe EV or not.
If it decides to charge the EV, then it provides one unit charge to the EV.
The stationthen obtains a unit reward and pays the current electricity price.
If the station fails to fully charge the
EV by the deadline of the EV, then the station su ↵ ers from a penalty that is a convex function of theremaining needed charge.
A new EV arrives at the station when the previous EV leaves.
We model this problem by letting the scalar state be the current electricity price and the vector statebe the remaining needed charge and time-to-deadline of the current EV.
A threshold policy is one thatcalculates a threshold based on the EV’s remaining needed charge and time-to-deadline, and thendecides to charge the EV if and only if the current electricity price is below the threshold.
Inventory management problem.
We construct an inventory management problem by jointlyincorporating a variety of practical challenges, including seasonal ﬂuctuations in demands and leadtimes in orders, in the literature [26, 14, 10, 25].
We consider a warehouse holding goods.
In eachtimestep, there is a random amount of demand whose mean depends on the time of the year.
Thewarehouse can fulﬁll the demand as long as it has su  cient inventory, and it makes a proﬁt for eachunit of sold goods.
At the end of the timestep, the warehouse incurs a unit holding cost for each unitof unsold goods.
The warehouse manager needs to decide whether to order more goods.
When itplaces an order for goods, there is a lead time of one time step, that is, the goods ordered at timestep tare only available for sale at timestep t + 1.
We model this problem by letting the scalar state be the current inventory and the vector state be thetime of the year.
A threshold policy calculates a threshold based on the time of the year and decidesto place an order for goods if the current inventory is below the threshold.
Make-to-stock production problem.
This problem is considered in [24].
It studies a system thatproduces m items with W demand classes and bu ↵ er size S .
Accepting a class v order leads to areward R v , as long as there is still room in the bu ↵ er for the order.
The classes of demands are orderedsuch that R 1 > R 2 > . . . .
In this problem, the scalar state is the number of accepted but unﬁnishedorders and the vector state is the class of the next arriving order.
More details about the three MDPs can be found in Appendix C.
[Figure]
Evaluated policies.
We compare DeepTOP-MDP against DDPG [18] and TD3 [8], two state-ofthe-art o ↵ -policy and model free deep RL algorithms.
We use open-source implementations ofthese two algorithms for [12, 7].
We use the same hyper-parameters, including the neural networkarchitecture, learning rates, etc., for all three algorithms.
We also evaluate the Structure-Aware
Learning for Multiple Thresholds algorithm (SALMUT) [24], a reinforcement learning algorithmthat ﬁnds the optimal threshold policy.
SALMUT requires the vector states to be pre-sorted by theirthreshold values.
Hence, SALMUT can only be applied to the make-to-stock production problem.
Details about the training parameters can be found in Appendix D. For the EV charging problem, Yu,
Xu, and Tong [32] has found the optimal threshold policy.
We call the optimal threshold policy the
Deadline Index policy and compare DeepTOP-MDP against it.
Simulations results.
Simulation results of the three MDPs are shown in Figure 1.
The results arethe average of 20 independent runs.
Before starting a run, we ﬁll an agent’s memory with 1000transitions by randomly selecting actions.
We plot the average reward obtained from the previous100 timesteps, and average them over 20 runs.
In addition, we provide the standard deviation boundsfrom the average reward.
It can be observed that DeepTOP signiﬁcantly outperforms DDPG, TD3, and SALMUT.
Althoughthe training procedure of DeepTOP is similar to that of DDPG, DeepTOP is able to achieve muchfaster learning by leveraging the monotone property.
Without leveraging the monotone property,
DDPG and TD3 need to learn the optimal policy for each scalar state independently, and thereforehave much worse performance.
DeepTOP performs better than SALMUT because DeepTOP directlyemploys the threshold policy gradient.
We also note that DeepTOP performs virtually the same asthe Deadline Index policy for the EV charging problem in about 2000 timesteps, suggesting that
DeepTOP indeed ﬁnds the optimal threshold policy quickly.
We also evaluate DeepTOP for di ↵ erentneural network architectures in Appendix E, and show that DeepTOP performs the best in all settings.
6.2 Simulations for RMABs
Description of the RMAB.
We consider an extension of the RMAB problem evaluated in Killianet al.
Killian et al.
[16] considers the case when each arm is a two-state Markov process.
Weextend it so that each arm is a Markov process with 100 states, numbered as 0 , 1 , . .
99 , as shown in
Figure 2 where state 99 is the optimal state.
[Figure] [Figure]
The reward of an arm depends on the distance between its current state and state99.
Suppose the current state of arm i is s i , t ,then it generates a reward r i , t = 1  ( s i , t  99 99 ) 2 .
If the arm is activated, then it changes tostate
s i , t + 1 = min { s i , t + 1 , 99 } with probabil- 291 ity p i .
If the arm is not activated, then itchanges to state s i , t + 1 = max { s i , t  1 , 0 } with 293 probability q i .
In the simulations, we pickthe probabilities p i to be evenly spaced depending on the number of arms N from theinterval [0 . 2 , 0 . 8].
We set the probabilitiesq i = p i .
We consider that there are N arms and that the agent needs to activate V arms in eachtimestep.
We evaluate three settings of ( N , V ) =
(10 , 3) , (20 , 5) , and (30 , 6).
Evaluated policies.
We compare DeepTOP-RMAB against three recent studies that aim to learnindex policies for RMABs, namely, Lagrange policy Q learning (LPQL) [16], Whittle index based Qlearning (WIBQL)
[1], and neural Whittle index network (NeurWIN) [19].
LPQL consists of threesteps: First, it learns a Q function for each arm independently.
Second, it uses the Q functions ofall arms to determine a common Lagrangian.
Third, it uses the Lagrangian to calculate the index ofeach arm.
WIBQL is a two-timescale algorithm that learns the Whittle indices of indexable armsby updating Q values on the fast timescale, and index values on the slower timescale.
NeurWIN isan o ↵ -line training algorithm based on REINFORCE that requires a simulator to learn the Whittleindex.
Both LPQL and WIBQL are tabular learning methods which may perform poorly comparedto deep RL algorithms when the size of the state space is large.
Hence, we also design deep RLequivalent algorithms that approximate their Q functions using neural networks.
We refer to the Deep
RL extensions as neural LPQL and neural WIBQL.
In all experiments, neural LPQL, neural WIBQL,and NEurWIN use the same hyper-parameters as DeepTOP-RMAB.
It can be shown that the Whittleindex in this setting is in the range of [  1 , 1], and hence we set M = 1.
Simulation results.
Simulation results are shown in Figure 3.
It can be observed that DeepTOPachieves the optimal average rewards in all cases.
The reason that neural LPQL performs worsethan DeepTOP may lie in its reliance on a common Lagrangian.
Since the common Lagrangian iscalculated based on the Q functions of all arms, an inaccuracy in one arm’s Q function can result inan inaccurate Lagrangian, which, in turn, leads to inaccuracy in the index values of all arms.
Priorwork [16] has already shown that WIBQL performs worse than LPQL.
Hence, it is not surprising thatneural WIBQL performs worse than both neural LPQL and DeepTOP.
NeurWIN performs worsethan DeepTOP because it is based on REINFORCE and therefore can only apply updates at the endof each episode.
We also evaluate DeepTOP for di ↵ erent neural network architectures.
Results areshown in Appendix F, which show that DeepTOP performs the best in all settings.
[Figure] [Figure]
Related Work
Threshold policies have been analysed for many decision-making problems formed as MDPs.
[11]examined the residential energy storage under price ﬂuctuations problem, and proved the existenceof optimal threshold policies for minimizing the cost.
[5] proved that MDPs with a convex andpiecewise linear cost functions admit an optimal threshold policy.
[23] shows the existence of anoptimal threshold policy for energy arbitrage given degrading battery capacity, with [2] using the
REINFORCE algorithm [31] to learn a trading policy with price thresholds for intraday electricitymarkets.
[13] considered mean ﬁeld games in a multi-agent MDP setting, and characterized individualagent strategy with a threshold policy when the mean game admits a threshold policy.
More recently, [29] studies ﬁnding a job assigning threshold policy for data centers with heterogeneousservers and job classes, and gave conditions for the existence of optimal threshold policies.
[33]proposed a distributed threshold-based control policy for graph traversal by assigning a state thresholdthat determines if the agent stays in or leaves a state.
For minimizing the age of information inenergy-harvesting sensors, [4] used the ﬁnite-di ↵ erence policy gradient [22] to learn a possibly suboptimal threshold policy in the average cost setting.
[27] used the Deep Q-network RL algorithm forselecting alert thresholds in anti-fraud systems with simulations showing performance improvementsover static threshold policies.
[24] described the SALMUT RL algorithm for exploiting the orderedmulti-threshold structure of the optimal policy with SALMUT implementations in [15] for computingnode’s overload protection.
In contrast to these works, DeepTOP-MDP is applicable to any MDP thatadmits threshold policies.
In learning the Whittle index policy for RMABs, [6] proposed a Q-learning heuristic called the Q
Whittle Index Controller (QWIC) which may not ﬁnd the Whittle indices even when the trainingconverges.
[19] describes a Deep RL algorithm called NeurWIN for learning the Whittle index ofa restless arm independently of other arms.
However, NeurWIN requires a simulator to train theneural networks.
Some recent studies, such as [1, 3, 16], proposed various online learning algorithmsthat can ﬁnd Whittle index when the algorithms converge.
These algorithms rely on some indirectproperty of the Whittle index which explains why they converge slower than DeepTOP.
Conclusion
In this paper, we presented DeepTOP: a Deep RL actor-critic algorithm that learns the optimal threshold function for MDPs that admit a threshold policy and for RMAB problems.
We ﬁrst developed thethreshold policy gradient theorem, where we proved that a threshold function has a simple to computegradient.
Based on the gradient expressions, we design the DeepTOP-MDP and DeepTOP-RMABalgorithm variants and compare them against state-of-the-art learning algorithms.
In both the MDPand RMAB settings, experiment results showed that DeepTOP exceeds the performance of baselinesin all considered problems.
[Table]
[Figure] [Figure]
(cid:4) (cid:5) (cid:6)(cid:6) (cid:7) (cid:8) (cid:9) (cid:10) (cid:11) (cid:12)
[Figure]
(cid:7)(cid:4)(cid:4)(cid:4) (cid:10)(cid:4)(cid:4)(cid:4) (cid:13)(cid:4)(cid:4)(cid:4)
[Figure]
[Figure]
[Table] [Figure] [Table] [Figure]
setting over 20 independent runs.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope? [Yes] see Section 6 for simulations’ results.
(b) Did you describe the limitations of your work?
[Yes] described in Section 2.
Our
algorithm is only applicable to MDPs that admit a threshold policy.
(c) Did you discuss any potential negative societal impacts of your work?
[N / A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] see Section
and Section 4 for problem deﬁnitions and assumptions.
(b) Did you include complete proofs of all theoretical results?
[Yes] see Section 3 and
Appendix A for proofs.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] code and run
instructions are available in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] see Appendix D.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] we averaged the results over 20 runs and plotted thestandard deviation from the average.
See Section 6 for results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] see Appendix D.
If you are using existing assets (e.g., code, data, models) or curating / releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] we cite and useopen-source code for the baselines:
DDPG, TD3, LPQL, and WIBQL.(b) Did you mention the license of the assets?
[Yes] licenses are mentioned in AppendixD.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
our code is included with the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using / curating?
[N / A]
(e) Did you discuss whether the data you are using / curating contains personally identiﬁable
information or o ↵ ensive content?
[N / A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable
N
[Table]
you
any
participant
, with links to Institutional Review
[Table]
Board
approvals,
[Table]
you
the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation
N
A
[Table]