T IMING IS E VERYTHING : L EARNING TO A CT S ELECTIVELY WITH C OSTLY A CTIONS AND C ONSTRAINTS
Anonymous authors
Paper under double-blind review
A BSTRACT
Many real-world settings involve costs for performing actions; transaction costs in financial systems and fuel costs being common examples.
In these settings, performing actions at each time step quickly accumulates costs leading to vastly suboptimal outcomes.
Additionally, repeatedly acting produces wear and tear and ultimately, damage.
Determining when to act is crucial for achieving successful outcomes and yet, the challenge of efficiently learning to behave optimally when actions incur minimally bounded costs remains unresolved.
In this paper, we introduce a reinforcement learning (RL) framework named L earnable I mpulse C ontrol
R einforcement A lgorithm (LICRA), for learning to optimally select both when to act and which actions to take when actions incur costs.
At the core of LICRA is a nested structure that combines RL and a form of policy known as impulse control which learns to maximise objectives when actions incur costs.
We prove that LICRA, which seamlessly adopts any RL method, converges to policies that optimally select when to perform actions and their optimal magnitudes.
We then augment LICRA to handle problems in which the agent can perform at most k < ∞ actions and more generally, faces a budget constraint.
We show LICRA learns the optimal value function and ensures budget constraints are satisfied almost surely.
We demonstrate empirically LICRA’s superior performance against benchmark
RL methods in OpenAI gym’s Lunar Lander and in Highway environments and a variant of the Merton portfolio problem within finance.
I NTRODUCTION
There are many settings in which agents incur costs each time they perform an action.
Transaction costs in financial settings (Mguni, 2018a), fuel expenditure (Zhao et al., 2018), toxicity as a side effect of controlling bacteria (Sootla et al., 2016) and physical damage produced by repeated action that produces wear and tear are just a few among many examples (Grandt Jr, 2011).
In these settings, performing actions at each time step is vastly suboptimal since acting in this way results in prohibitively high costs and undermines the service life of machinery.
Minimising wear and tear is an essential attribute to safeguard against failures that can result in catastrophic losses (Grandt Jr, 2011).
Reinforcement learning (RL) is a framework that enables autonomous agents to learn complex behaviours from interactions with the environment (Sutton & Barto, 2018).
Within the standard RL paradigm, determining optimal actions involves making a selection among many (possibly infinite) actions; a procedure that must be performed at each time-step as the agent decides on an action.
In unknown settings, the agent cannot immediately exploit any topological structure of the action set.
Consequently, learning not to take an action i.e performing a zero or null action , involves expensive optimisation procedures over the entire action set.
Since this must be done at each state, this process is vastly inefficient for learning optimal policies when the agent incurs costs for acting.
In this paper, we tackle this problem by developing an RL framework for finding both an optimal criterion to determine whether or not to execute actions as well as learning optimal actions.
A key component of our framework is a novel combination of RL with a form of policy known as impulse control (Mguni, 2018d;a).
This enables the agent to determine the appropriate points to perform an action as well as the optimal action itself.
Despite its fundamental importance as a tool for tackling decision problems with costly actions (Korn, 1999; Mitchell et al., 2014), presently, the use of impulse control within learning contexts (and unknown environments) is unaddressed.
We present an RL impulse control framework called LICRA, which, to our knowledge, is the first learning framework for impulse control.
To enable learning optimal impulse control policies in unknown environments, we devise a framework that consists of separate RL components for learning when to act and, how to act, optimally.
The resulting framework is a structured two-part learning process which differs from current RL protocols.
In LICRA, at each time step, the agent firstly makes a decision whether to act or not leading to a binary decision space { 0 , 1 } .
The second decision part determines the best action to take.
This generates a subdivision of the state space into two regions; one in which the agent performs actions and another in which it does not act at all.
We show the set of states where the agent ought to act is characterised using an easy-to-evaluate criterion on the value function and, LICRA quickly learns this set within which it then learns which actions are optimal.
We then establish theory that ensures convergence of a Q-learning variant of LICRA to the optimal policy for such settings.
To do this, we give a series of results namely:
i)
We establish a dynamic programming principle (DPP) for impulse control and show that the optimal value function can be obtained as a limit of a value iterative procedure (Theorem 1) which lays the foundation for an RL approach to impulse control.
ii)
We extend result i) to a new variant of Q-learning which enables the impulse control problem to be solved using our RL method (Theorem 2).
Thereafter, we extend the result to (linear) function approximators enabling the value function to be parameterised (Theorem 3).
iii)
We characterise the optimal conditions for performing an action which we reveal to be a simple ‘obstacle condition’ involving the agent’s value function (Proposition 1).
Using this, the agent can quickly determine whether or not it should act and if so, then learn what the optimal action is.
iv)
In Sec. 6, we extend LICRA to include budgetary constraints so that each action draws from a fixed budget which the agent must stay within.
Analogous to the development of i), we establish another DPP from which we derive a Q-learning variant for tackling impulse control with budgetary
constraints (Theorem 4).
A particular case of a budget constraint is when the number of actions the agent can take over the horizon is capped.
Lastly, we perform a set of experiments to validate our theory within the Highway driving simulator and OpenAI’s
LunarLander (Brockman et al., 2016).
As we demonstrate in our experiments, LICRA learns to compute the optimal problems in which the agent faces costs for acting in an efficient way which outperforms leading RL baselines.
Second, as demonstrated in Sec. 6, LICRA handles settings in which the agent has a cap the total number of actions it is allowed to execute and more generally, generic budgetary constraints.
LICRA is able to accommodate any RL algorithm unlike various RL methods designed to handle budgetary constraints.
R ELATED W ORK
In continuous-time optimal control theory (Øksendal, 2003), problems in which the agent faces a cost for each action are tackled with a form of policy known as impulse control (Mguni, 2018d;a;
Al-Fagih, 2015).
In impulse control frameworks, the dynamics of the system are modified through a sequence of discrete actions or bursts chosen at times that the agent chooses to apply the control policy.
This distinguishes impulse control models from classical decision methods in which an agent takes actions at each time step while being tasked with the decision of only which action to take.
Impulse control models represent appropriate modelling frameworks for financial environments with transaction costs, liquidity risks and economic environments in which players face fixed adjustment costs (e.g. menu costs )
(Korn, 1999; Mguni, 2018b; Mundaca & Øksendal, 1998).
The current setting is intimately related to the optimal stopping problem which widely occurs in finance, economics and computer science (Mguni, 2019; Tsitsiklis & Van Roy, 1999).
In the optimal stopping problem, the task is to determine a criterion that determines when to arrest the system and receive a terminal reward.
In this case, standard RL methods are unsuitable since they require an expensive sweep (through the set of states) to determine the optimal point to arrest the system.
The current problem can be viewed as an augmented problem of optimal stopping since the agent must now determine both a sequence of points to perform an action or intervene and their optimal magnitudes — only acting when the cost of action is justified (Øksendal & Sulem, 2019).
Adapting
RL to tackle optimal stopping problems has been widely studied (Tsitsiklis & Van Roy, 1999; Becker et al., 2018; Chen et al., 2020) and applied to a variety of real-world settings within finance (Fathan & Delage, 2021) and network operating systems (Alcaraz et al., 2020).
Our work serves as a natural extension of RL approaches to optimal stopping to the case in which the agent must decide at which points to take a series of actions.
As with optimal stopping, standard RL methods cannot efficiently tackle this problem since determining whether to perform a 0 action requires a costly sweep through the action space at every state (Tsitsiklis & Van Roy, 1999).
In (Pang et al., 2021) the authors introduce “sparse action” with a similar motivation as impulse control.
However, the authors treat only the discrete action space case.
Moreover, (Pang et al., 2021) do not provide a general theoretical framework of dealing with “sparse actions” but develop purely algorithmic solutions.
Additionally, unlike the approach taken in (Pang et al., 2021), the problem setting we consider is one in which the agent faces a cost for each action - this produces a need for the agent to be selective about where it performs actions (but does not necessarily constrain the magnitude or choice of those actions).
Our framework bears resemblance to hierarchical RL (HRL) (Barto & Mahadevan, 2003).
Policies in this hierarchy have usually different objectives.
For example in sub-goal learning (Noelle, 2019), a top-level policy decides the goal of a low-level policy.
In this case, the top-level policy gives control to the low-level policy for a number of time steps and/or until the sub-goal is achieved.
In contrast,
LICRA’s top-level policy must make a decision at every time step.
One can note further similarities with a specific HRL framework — RL with options (Precup et al., 1998; Klissarov & Precup, 2021).
There the agent chooses an option and then selects an action (or a sequence of actions) based on the current state and the chosen option.
Therefore, there are two policies that the agent learns.
Options and HRL offer abstraction frameworks with higher levels of internal structure than most of the standard RL algorithms.
This can significantly speed up learning in many situations.
For example, in a setting with sparse rewards, a subgoal discovery algorithm can guide the agent to an optimal policy more quickly.
Our impulse control framework offers an even higher level of structure than options and HRL in order to determine when to act (Jinnai et al., 2019).
This is achieved by exploiting the specificity of the impulse control setting which includes a cost of acting while employing a structure that enables isolating null actions from the action space.
P RELIMINARIES
In RL, an agent sequentially selects actions to maximise its expected returns.
The underlying problem is typically formalised as an MDP ⟨S , A , P, R, γ ⟩ where S ⊂ R p is the set of states, A ⊂ R k is the (continuous) set of actions, P : S ×
A × S → [0 , 1] is a transition probability function describing the system’s dynamics, R : S ×
A → R is the reward function measuring the agent’s performance and the factor γ ∈ [0 , 1) specifies the degree to which the agent’s rewards are discounted over time (Sutton & Barto, 2018).
At time t ∈ 0 , 1 , . . .
, the system is in state s t ∈ S and the agent must choose an action a t ∈
A which transitions the system to a new state s t +1 ∼ P ( ·| s t , a t ) and produces a reward R ( s t , a t ) .
A policy π : S ×
A → [0 , 1] is a probability distribution over state-action pairs where π ( a | s ) represents the probability of selecting action
a ∈ A in state s ∈ S .
The goal of an
RL agent is to find a policy ˆ π ∈ Π that maximises its expected returns given by the value function: v π ( s )
=
E [ (cid:80) ∞ t =0 γ t R ( s t , a t ) | a t ∼ π ( ·| s t ) , s
= s ] where Π is the agent’s policy set.
The action value function is given by Q ( s, a ) =
E [ (cid:80) ∞ t =0 R ( s t , a t ) | a 0 = a, s
= s ] .
We consider a setting in which the agent faces at least some minimal cost for each action it performs.
With this, the agent’s task is to maximise:
[Equation]
where for any state s ∈ S and any action a ∈ A , the functions R and C are given by R ( s, a )
=
R ( s, a ) 1 a ∈A / { 0 } + R ( s, 0)(1 − 1 a ∈A / { 0 } ) where 1 a ∈A / { 0 } is the indicator function which is 1 when a ∈ A / { 0 } and 0 otherwise and C ( s, a ) :
=
c ( s, a ) 1 a ∈A / { 0 } where c : S ×
A → R is a strictly positive (cost) function that introduces a cost each time the agent performs an action.
Examples of the cost function is a quasi-linear function of the form c ( s t , a t ) = κ
+ f ( a t ) where f : A → R > 0 and κ is a positive real-valued constant.
Since acting at each time step would incur prohibitively high costs, the agent must be selective when to perform an action.
Therefore, in this setting, the agent’s problem is augmented to learning both an optimal policy for its actions and, learning at which states to apply its action policy.
Examples of problem settings of this kind include financial portfolio problems with transaction costs (Davis & Norman, 1990) where each investment incurs a fixed minimal cost and robotics where actions produce mechanical stress leading to fatigue (Grandt Jr, 2011).
T HE LICRA F RAMEWORK
In RL, the agent’s problem involves learning to act at every state including those in which actions do not significantly impact on its total return.
While we can add a zero action to the action set A and apply standard methods, we argue that this may not be the best solution in many situations.
We argue the optimal policy has the following form:
[Equation]
which implies that we simplify policy learning by determining the set S I first — the set where we actually need to learn the policy.
We now introduce a learning method for producing impulse controls which enables the agent to learn to select states to perform actions.
Therefore, now the agent is tasked with learning to act at states that are most important for maximising its total return given the presence of the cost for each action.
To do this effectively, at each state the agent first makes a binary decision to decide to perform an action.
Therefore, LICRA consists of two core components: first, an RL process g : S × { 0 , 1 } → [0 , 1] and a second RL process π : S ×
A → [0 , 1] .
The role of g is to determine whether or not an action is to be performed by the policy π at a given state s .
In LICRA, the policy π first proposes an action a ∈ A which is observed by the policy g .
If activated, the policy π determines the action to be selected.
Therefore, g prevents actions under π for which the change in expected return does not exceed the costs incurred for taking such actions.
While we now consider two policies π , g , the cardinality of the action space does not change.
Crucially, the agent must compute optimal actions at only the subset of states chosen by g .
By isolating the decision of whether to act or not, the LICRA framework may also increase the learning speed in this setting.
T , mini-epochs e 2: Initialise: Policy network (acting) π , Policy network (switching) g ,
Critic network (acting ) V π ,Critic network (switching ) V g 3:
Given reward objective function, R , initialise Rollout Buffers B π , B g (use Replay Buffer for
SAC) 4: for N episodes do 5:
Reset state s 0 , Reset Rollout Buffers B π , B g 6: for t = 0 , 1 , . . .
do 7:
Sample a t ∼ π ( ·| s t ) 8:
Sample g t ∼ g ( ·| s t ) 9: if g t = 1 then 10:
Apply a t so s t +1 ∼ P ( ·| a t , s t ) , 11:
Receive rewards r t =
R ( s t , a t ) − c ( s t , a t ) 12:
Store ( s t , a t , s t +1 , r t ) in B π 13: else 14:
Apply the null action so s t +1 ∼ P ( ·| 0 , s t ) , 15:
Receive rewards r t =
R ( s t , 0) .
16: end if 17:
Store ( s t , g t , s t +1 , r t ) in B g 18: end for 19: //
Learn the individual policies 20:
Update policy π and critic V π networks using B π 21:
Update policy g and critic V g networks using B g 22: end for
To motivate this, first note the decision space for determining whether to execute an action is S × { 0 , 1 } i.e at each state it makes a binary decision (c.f. Prop. 1).
Since each action incurs a cost of at least c > 0 , acting at some states outside of S I may produce an appreciable reduction in expected return i.e Q ( s, a ≡ 0 ) ≫ Q ( s, a ′ ) for any a ′ ∈ A / { 0 } . Consequently, LICRA can quickly learn to take the null or action i.e. find the optimal binary policy since all action-policy proposals of to the binary policy would lead to a lower payoff.
This feature is critical in settings with continuous action spaces for which isolating the 0 action generally requires an expensive optimisation.
Moreover, this can result in the agent rapidly learning which states to focus on to learn which actions to perform.
Additionally, if the set S / S I ,is large enough, then again it can be more efficient to learn g first and only then learn π (we show this empirically in Sec. 11.2).
LICRA consists of two independent procedures: a learning process for the policy π and simultaneously, a learning process for the impulse policy g which determines at which states to perform an action.
In Sec. 5, we prove the convergence properties of LICRA.
In our implementation, we used proximal policy optimisation (PPO)
(Schulman et al., 2017) for the policy π and for the impulse policy g , whose action set consists of two actions (intervene or do not intervene) we used a soft actor critic (SAC) process (Haarnoja et al., 2018).
LICRA is a plug & play framework which enables these RL components to be replaced with any RL algorithm of choice.
Above is the pseudocode for
LICRA, we provide full details of the code in Sec.
9 of the Appendix.
C ONVERGENCE AND O PTIMALITY OF LICRA WITH Q-L EARNING
A key aspect of our framework is the presence of two RL processes that make decisions in a sequential order.
In order to determine when to act the policy g must learn the states to allow the policy π to perform an action which the policy π must learn to select optimal actions whenever it is allowed to execute an action.
In this section, we prove that a the LICRA converges to an optimal solution of the system.
Our proof is instantiated in a Q-learning variant of LICRA which serves as the natural basis for other extensions such as actor-critic methods.
We then extend the result to allow for (linear) function approximators.
We provide a result that shows the optimal intervention times are characterised by an ‘obstacle condition’ which can be evaluated online therefore allowing the g .
Given a function Q : S × A → R , ∀ π, π ′ ∈ Π and ∀ g , ∀ s τ k ∈ S , we define the intervention operator
M by M π [ Q π ′ , g ( s τ k , a )] :=
R ( s τ k , a τ k ) − c
( s τ k , a τ k )+ γ (cid:80) s ′ ∈S P ( s ′ ; a τ k , s ) Q π ′ , g ( s ′ , a τ k ) | a [Equation] k ∼ π ( ·| s τ k ) .
The interpretation of M π ′ [ Q π, g ] is the following: suppose that at time τ k the system is at a state s τ k and the agent performs an immediate action a τ k ∼ π ′ ( ·| s τ k ) .
A cost of c ( s τ k , a τ k ) is then incurred by the agent and the system transitions to s ′ ∼ P ( · ; a τ k , s τ k ) whereafter the agent executes the policy ( π, g ) .
Therefore M π ′ Q π, g is the expected future stream of rewards after an immediate action minus the cost of action.
This object plays a crucial role in the LICRA framework which as we later discuss, exploits the cost structure of the problem to determine when the agent should take an action.
Denote by M [ Q π, g ] the intervention operator acting on Q π, g when the immediate action is chosen according to an epsilon-greedy policy.
Given any v π, g : S → R , the Bellman operator T is:
[Equation]
The Bellman operator captures the nested sequential structure of the LICRA method.
In particular, the structure in (3) consists of an inner structure that consists of two terms: the first term M [ Q π, g ] is the estimated expected return given the current best estimate of the optimal action is executed and the policy π is executed thereafter.
The second term R ( s, 0) + γ (cid:80) s ′ ∈S P ( s ′ , 0 , s ) v π, g ( s ′ ) isolates the estimated expected return following no action and when the policy π is executed thereafter.
Lastly, the outer structure is an optimisation which compares the two quantities and selects the maximum.
Note that when A is a continuous (dense) set, this reduces the burden on an optimisation (possibly over a function approximator) to extract the point 0 in the action space when it is optimal not to act.
Our first result proves T is a contraction operator in particular, the following bound holds:
Lemma
The Bellman operator T is a contraction, that is the following bound holds:
[Equation]
where v, v ′ are elements of a finite normed vector space.
We can now state our first main result:
Theorem 1 Given any v π, g : S ×
A → R , the optimal value function is given by lim k →∞ T k v π, g = max [Equation] = [Equation]
⋆ where ( π ⋆ , g ⋆ ) is the optimal policy pair.
The result of Theorem 1 enables the solution to the agent’s impulse control problem to be determined using a value iteration procedure.
Moreover, Theorem 1 enables a Q-learning approach (Bertsekas, 2012) for finding the solution to the agent’s problem.
Theorem 2 Consider the following Q-learning variant:
[Equation]
then Q t converges to Q ⋆ with probability 1 , where s t , s t +1 ∈ S and a t ∈ A .
We now extend the result to (linear) function approximators:
Theorem 3 Given a set of linearly independent basis functions Φ = { ϕ 1 , . . . , ϕ p } with ϕ k ∈ L 2 , ∀ k .
LICRA converges to a limit point r ⋆ ∈ R p which is the unique solution to Π F (Φ r ⋆ ) =
Φ r ⋆ where
[Equation]
The theorem establishes the convergence of the LICRA Q-learning variant to a stable point with the use of linear function approximators.
The second statement bounds the proximity of the convergence point by the smallest approximation error that can be achieved given the choice of basis functions.
Having constructed a procedure to find the optimal agent’s optimal value function, we now seek to determine the conditions when an intervention should be performed.
Let us denote by { τ k } k ≥
the points at which the agent decides to act or intervention times , so for example if the agent chooses to perform an action at state s 6 and again at state s 8 , then τ 1 = 6 and τ 2 = 8 .
The following result characterises the optimal intervention policy g and the optimal times { τ k } k ≥ 0 .
Proposition 1 The policy g is given by: g ( s t ) =
H ( M π
[ Q π, g ] − Q π, g )( s t , a t ) , ∀ s t ∈ S , where
Q π, g is the solution in Theorem 1, M is the intervention operator and H is the Heaviside function, moreover the intervention times are τ k = inf { τ > τ k −
|M π
[ Q π, g ]
= Q π, g } .
Prop. 1 characterises the (categorical) distribution g .
Moreover, given the function Q , the times { τ k } can be determined by evaluating if M [ Q ] = Q holds.
A key aspect of Prop. 1 is that it exploits the cost structure of the problem to determine when the agent should perform an intervention.
In particular, the equality M [ Q ] =
Q implies that performing an action is optimal.
B UDGET A UGMENTED LICRA VIA S TATE A UGMENTATION
We now tackle the problem of RL with a budget.
To do this, we combine the above impulse control technology with state augmentation technique proposed in (Sootla et al., 2022)
The mathematical formulation of the problem is now given by the following for any s ∈ S :
[Equation]
where n ∈ N is a fixed value that represents the maximum number of allowed interventions and (cid:80) k ≥ 1 δ tτ k is equal to one if an impulse was applied at time t and zero if it was not.
In order to avoid dealing with a constrained MDP, we propose to introduce a new variable z t tracking the remaining number of impulses: z t = n − (cid:80) t − 1 i =0 (cid:80) k ≥ 1 δ iτ k .
We treat z t as another state and augment the state-space resulting in the transition (cid:101) P :
[Equation]
To avoid violations, we reshape the reward as follows: (cid:101) R ( s t , z t [Equation] =
[Equation]
where ∆ > 0 is a large enough hyper-parameter ensuring there are no safety violations.
To summarise, we aim to solve the following budgeted impulse control (BIC) problem v π , g ( s, z ) =
E (cid:104)(cid:80) ∞ t =0 γ t (cid:101) R ( s t , z t , a t ) | a t ∼ π ( ·| s t , z t ) (cid:105) , where the policy now depends on the variable z t .
Note (cid:101) P in Equation 6 is a Markov process and, the rewards (cid:101) R are bounded given the boundedness of R .
Therefore, we can apply directly the results for impulse control.
We denote the augmented MDP by (cid:102)
M = ⟨S ×
Z , A , (cid:101) P , (cid:101) R, γ ⟩ , where Z is the space of the augmented state.
We have the following:
Theorem 4 Consider the MDP (cid:102) M for the BIC problem, then:
a)
The Bellman equation holds, i.e.
∃ ˜ v ∗ , π , g : S × Z
→ R such that ˜ v ∗ , π , g ( s, z ) = max a ∈A (cid:16) (cid:101) R ( s, z, a ) + γ E s ′ ,z ′ ∼P
[˜ v ∗ , π , g ( s ′ , z ′ )] (cid:17) , where the optimal policy for (cid:102)
M has the form π ∗ ( ·| s, z ) ; b) Given a (cid:101) v : S ×
Z
→ R , the stable point solution for (cid:102) M is given by lim k →∞ ˜ T k (cid:101) v π ,g = max ˆ π ∈ Π , ˆ g (cid:101) v ˆ π , ˆ g = (cid:101) v π ∗ , g ∗ , where ( π ∗ , g ∗ ) is an optimal policy and ˜ T is the Bellman operator of (cid:102) M .
The result has several important implications.
The first is that we can use a modified version of LICRA to obtain the solution of the problem while guaranteeing convergence (under standard assumptions).
Secondly, our state augmentation procedure admits a Markovian representation of the optimal policy.
E XPERIMENTS
We will now study empirically the performance of the LICRA framework.
In experiments, we use different instances of LICRA, one where both policies are trained using PPO update (referred to as LICRA_PPO ) and one where the policy deciding whether to act is trained using SAC and the other policy trained with PPO (referred to as LICRA_SAC ).
We have benchmarked both of these algorithms together with common baselines on environments, where it would be natural to introduce the concept of the cost associated with actions.
We performed a series of ablation studies (see
Appendix) which examine i)
LICRA’s ability to handle different cost functions including the case when c ( s, a ) ≡ 0 . ii) the benefits of LICRA in settings with a varying intervention regions.
iii) the performance of the LICRA Q-learning variant in a discrete state space environment.
In the following experiments, all algorithms apply actions that are drawn from an action set which contains a 0 action.
When this action is executed it produces costs and exerts no influence on the transition dynamics.
Merton’s Portfolio Problem with Transaction Costs.
To exemplify the problem in an applied setting, we choose a well-known problem in finance known as the Merton Investment Problem with transaction costs (Davis & Norman, 1990).
In this simple environment, the agent can decide to move its wealth between a risky asset and a risk-free asset.
The agent receives a reward only at the final step, equal to the utility of the portfolio with a risk aversion factor equal to 0 .
If the final wealth of risky asset is s T and final wealth of risk-free asset is c T , then the agent will receive a reward of u ( x )
= 2 √ s T + c T .
The wealth evolves according to the following SDE:
[Equation]
[Figure]
where W t is the current wealth (the state variable), dB t is an increment of Brownian motion and p t is the proportion of wealth invested in the risky asset.
We set the risk-free return r = 0 .
01 , risky asset return µ = 0 .
05 and volatility σ = 1 .
We discretise the action space so that at each step the agent has three actions available: move 10 % of risky asset wealth to the risk-free asset, move 10 % of risk-free asset wealth to the risky asset or do nothing.
Each time the agent moves the assets, it incurs a cost of 1 i.e. a transaction fee.
The agent can act after a time interval of 0 .
01 seconds and the episode ends after 75 steps.
The results of training are shown in Fig.
1 which clearly demonstrates that LICRA_PPO finds the optimal policy faster than standard PPO.
Driving Environment Fuel Rationing.
We studied an autonomous driving scenario where fuelefficient driving is a priority.
One of the main components of fuel-efficient driving is controlled usage of acceleration and braking, in the sense that 1) the amount of acceleration and braking should be limited 2) if accelerations should be performed slowly and gently.
We believe this is a problem where
LICRA should thrive as the impulse control agent can learn to restrict the amount of acceleration and braking in the presence of other cars and choose when to allow the car to decelerate naturally.
We used the highway-env (Leurent, 2018) environment on a highway task (see Fig (2. a)) where the green vehicle is our controlled vehicle and the goal is to avoid crashing into other vehicles whilst driving at a reasonable speed.
We add a cost function into the reward term dependent on the continuous acceleration action, C ( a t )
=
K + a 2 t , where K > 0 is a fixed constant cost of taking any action, and a t ∈ [ − 1 , 1] , with larger values of acceleration or braking being penalised more.
The results are presented in Fig.
(2.b).
Notably, LICRA is able to massively outperform the baselines, especially our safety specific baselines which struggle to deal with the cost function associated with the environment.
We believe one reason for the success of LICRA is that it is far easier for it to utilise the null action of zero acceleration/braking than the other algorithms, whilst all the algorithms have a guaranteed cost at every time step whilst not gaining a sizeable reward to counter the cost.
[Figure]
The scores were
Lunar Lander Environment.
We tested the ability of LICRA to perform in environment that simulate real-world physical dynamics.
We tested LICRA’s performance the Lunar Lander environment in
OpenAI gym (Brockman et al., 2016) which we adjusted to incorporate minimal bounded costs in the reward definition.
In this environment, the agent is required to maintain both a good posture mid-air and reach the landing pad as quickly as possible.
The reward function is given by:
Reward (
[Equation]
− 0 .
[Equation]
FuelSpent
[Equation] [Equation] [Equation] +
[Equation] hasLanded ( s [Equation] ) where d t is the distance to the landing pad, v t is the velocity of the agent, and ω t is the angular velocity of the agent at time t .
1 X is the indicator function of taking actions, which is 1 when the statement
X is true and 0 when X is false.
Considering the limited fuel budget, we assume that we have a fixed cost for each action taken by the agent here, and doing nothing brings no cost.
Then, to describe the goal of the game, we define the function of the final status by hasLanded(), which is 0 when not landing; 1 when the agent has landed softly on the landing pad; and − 1 when the lander runs out of fuel or loses contact with the pad on landing.
The reward function rewards the agent for reducing its distance to the landing pad, decreasing its speed to land smoothly and keeping the angular speed at a minimum to prevent rolling.
Additionally, it penalises the agent for running out of fuel and deters the agent from taking off again after landing.
By introducing a reward function with minimally bounded costs, our goal was to test if LICRA can exploit the optimal policy.
In Fig.
3, we observe that the LICRA agent outperforms all the baselines, both in terms of sample efficiency and average test return (total rewards at each timestep).
We also observe that LICRA enables more stable training than PPO, PPO-Lagrangian and CPO.
Ablation Study 1.
Prioritisation of Most Important Actions.
We next tested LICRA’s ability to prioritise where it performs actions when the necessity to act varies significantly between states.
To test this, we modified the Drive Environment to now consist of a single lane, a start state and a goal state start (at the end) where there is a reward.
With no acceleration, the vehicle decreases velocity.
To reach the goal, the agent must apply an acceleration a t ∈ [ − 1 , 1] .
Each acceleration a t incurs a cost C ( a t ) as defined above.
At zones k = 1 , 2 , 3 of the lane, if the vehicle is travelling below a velocity v min , it is penalised by a strictly negative cost c k where c 1 < c 2 < c 3 .
As shown in Fig.
4, when the intervention cost increases i.e. when K → ∞ , LICRA successfully prioritises the highest penalty zones to avoid incurring large costs.
[Figure]
V i
o l a t i o n
V i
o l a t i o n
V i
o l a t i o n
[Figure]
C ONCLUSION
We presented a novel method to tackle the problem of learning how to select when to act in addition to learning which actions to execute.
Our framework, which is a general tool for tackling problems of this kind seamlessly adopts RL algorithms enabling them to efficiently tackle problems in which the agent must be selective about when it executes actions.
This is of fundamental importance in practical settings where performing many actions over the horizon can lead to costs and undermine the service life of machinery.
We demonstrated that our solution, LICRA which at its core has a sequential decision structure that first decides whether or not an action ought to be taken under the action policy can solve tasks where the agent faces costs with extreme efficiency as compared to leading reinforcement learning methods.
In some tasks, we showed that LICRA is able to solve problems that are unsolvable using current reinforcement learning machinery.
We envisage that this framework can serve as the basis extensions to different settings including adversarial training for solving a variety of problems within RL.