Timing is Everything: Learning to Act Selectively with Costly Actions and Budgetary Constraints
Anonymous Author(s)
AffiliationAddress
Abstract
Many real-world settings involve costs for performing actions; transaction costsin financial systems and fuel costs being common examples.
In these settings,performing actions at each time step quickly accumulates costs leading to vastlysuboptimal outcomes.
Additionally, repeatedly acting produces wear and tear andultimately, damage.
Determining when to act is crucial for achieving successfuloutcomes and yet, the challenge of efficiently learning to behave optimally whenactions incur minimally bounded costs remains unresolved.
In this paper, we introduce a reinforcement learning (RL) framework named L earnable I mpulse C ontrol
R einforcement A lgorithm (LICRA), for learning to optimally select both whento act and which actions to take when actions incur costs.
At the core of LICRAis a nested structure that combines RL and a form of policy known as impulsecontrol which learns to maximise objectives when actions incur costs.
We provethat LICRA, which seamlessly adopts any RL method, converges to policies thatoptimally select when to perform actions and their optimal magnitudes.
We thenaugment LICRA to handle problems in which the agent can perform at most k < ∞actions and more generally, faces a budget constraint.
We show LICRA learns theoptimal value function and ensures budget constraints are satisfied almost surely.
We demonstrate empirically LICRA’s superior performance against benchmark
RL methods in OpenAI gym’s Lunar Lander and in Highway environments and avariant of the Merton portfolio problem within finance.
Introduction
There are many settings in which agents incur costs each time they perform an action.
Transactioncosts in financial settings [17], fuel expenditure [28], toxicity as a side effect of controlling bacteria [25] and physical damage produced by repeated action that produces wear and tear are just afew among many examples [11].
In these settings, performing actions at each time step is vastlysuboptimal since acting in this way results in prohibitively high costs and undermines the service lifeof machinery.
Minimising wear and tear is an essential attribute to safeguard against failures that canresult catastrophic losses [11].
Reinforcement learning (RL) is a framework that enables autonomous agents to learn complexbehaviours from interactions with the environment [26].
Within the standard RL paradigm, determining optimal actions involves making a selection from among many (possibly infinite) actions; aprocedure that must be performed at each time-step as the agent decides on an action.
In unknownsettings, the agent cannot immediately exploit any topological structure of the action set (if anyexists).
Consequently, learning not to take an action i.e performing a zero or null action , involvesexpensive optimisation procedures over the entire action set.
Since this must be done at each state,this process is vastly inefficient for learning optimal policies when the agent incurs costs for acting
In this paper, we tackle this problem by developing an RL framework for finding both an optimalcriterion to determine whether or not to execute actions as well as learning optimal actions.
A keycomponent of our framework is a novel combination of RL with a form of policy known as impulsecontrol [19, 17].
This enables the agent to determine the appropriate points to perform an action aswell as the optimal action itself.
Despite its fundamental importance as a tool for tackling decisionproblems with costly actions, presently, the use of impulse control within learning contexts (andunknown environments) is unaddressed.
We present an RL impulse control framework called LICRA, which, to our knowledge, is the firstlearning framework for impulse control.
To enable learning optimal impulse control policies inunknown environments, we devise a framework that consists of separate RL components for learningwhen to act and how to act optimally.
The resulting framework is a structured two-part learningprocess which differs from current RL protocols.
In LICRA, at each time step, the agent firstly makesa decision whether to act or not leading to a binary decision space { 0 , 1 } (we later show that thisis determined by evaluating an easy-to-evaluate criterion which has the value function as its input).
The second decision part determines the best action to take.
This generates a subdivision of the statespace into two regions; one in which the agent performs actions and another in which it does not actat all.
This is extremely useful since the agent quickly determines the set of states to not take actionswhile performing actions only at the subset of states where actions are to be executed.
We then establish theory that ensures convergence of LICRA to an optimal policy for such settings.
To do this, we give a series a results namely:
i)
We establish a dynamic programming principle (DPP) for impulse control and show that the optimalvalue function can be obtained as a limit of a value iterative procedure (Theorem 1) which lays the
foundation for an RL approach to impulse control.
ii)
We extend result i) to a new variant of Q learning which enables the impulse control problem to be
solved using our RL method (Theorem 2).iii)
We characterise the optimal conditions for performing an action which we reveal to be a simple‘obstacle condition’ involving the agent’s value function (Prop. 1).
Using this, the agent can quickly
determine whether
or
not it should act and if so, then learn what the optimal action is.
iv)
We then extend the result i) to (linear) function approximators enabling the value function to beparameterised (Theorem 3). iv)
In Sec. 6, we extend LICRA to include budgetary constraints so that each action draws from afixed budget which the agent must stay within.
Analogous to the development of i), we establishanother DPP from which we derive a Q-learning variant for tackling impulse control with budgetaryconstraints (Theorem 4).
A particular case of a budget constraint is when the number of actions theagent can take over the horizon is capped.
Lastly, we perform a set of experiments to validate our theory within the Highway driving simulatorand OpenAI’s
LunarLander [7].
LICRA confers a series of advantages.
As we demonstrate in our experiments, LICRA learns tocompute the optimal problems in which the agent faces costs for acting in an efficient way whichoutperforms leading RL baselines.
Second, as demonstrated in Sec.
6 LICRA handles settings inwhich the agent has a cap the total number of actions it is allowed to execute and more generally,generic budgetary constraints.
LICRA is able to accommodate any RL base algorithm (see Sec.
11)unlike various RL methods designed to handle budgetary constraints.
Related Work
In continuous-time optimal control theory [21], problems in which the agent faces a cost for eachaction are tackled with a form a policy known as impulse control [19, 17, 2].
In impulse controlframeworks, the dynamics of the system are modified through a sequence of discrete actions or burstschosen at times that the agent chooses to apply the control policy.
This distinguishes impulse controlmodels from classical decision methods in which an agent takes actions at each time step while beingtasked with the decision of only which action to take.
Impulse control models represent appropriatemodelling frameworks for financial environments with transaction costs, liquidity risks and economicenvironments in which players face fixed adjustment costs (e.g. menu costs ) [14, 18].
The current setting is intimately related to the optimal stopping problem which widely occurs infinance, economics and computer science [20, 27].
In the optimal stopping problem, the task is determine a criterion that determines when to arrest the system and receive a terminal reward.
In thiscase, standard RL methods are unsuitable since they require an expensive sweep (through the setof states) to determine the optimal point to arrest the system.
The current problem can be viewedas an augmented problem of optimal stopping since the agent must now determine both a sequenceof points to perform an action or intervene and their optimal magnitudes — only acting when thecost of action is justified [22].
Adapting RL to tackle optimal stopping problems has been widelystudied [27, 4, 8] and applied to a variety of real-world settings within finance [10], radiation therapy[1] and network operating systems [3].
Our work serves as a natural extenstion to RL approaches tooptimal stopping to the case in which the agent must decide at which points to take many actions.
Aswith optimal stopping, standard RL methods cannot efficiently tackle this problem since determiningwhether to perform a 0 action requires a costly sweep through the action space at every state [27].
Preliminaries
Reinforcement Learning (RL).
In RL, an agent sequentially selects actions to maximise its expectedreturns.
The underlying problem is typically formalised as an MDP ⟨S , A , P, R, γ ⟩ where S ⊂ R pis the set of states, A ⊂ R k is the set of actions, P : S ×
A × S → [0 , 1] is a transition probabilityfunction describing the system’s dynamics, R : S ×
A → R is the reward function measuring theagent’s performance and the factor γ ∈ [0 , 1) specifies the degree to which the agent’s rewards arediscounted over time [26].
At time t ∈ 0 , 1 , . . . , the system is in state s t ∈ S and the agent mustchoose an action a t ∈
A which transitions the system to a new state s t +1 ∼ P ( ·| s t , a t ) and producesa reward R ( s t , a t ) .
A policy π : S ×
A → [0 , 1] is a probability distribution over state-action pairswhere
π ( a | s ) represents the probability of selecting action a ∈ A in state s ∈ S .
The goal of an
RL agent is to find a policy ˆ π ∈ Π that maximises its expected returns given by the value function:v π ( s )
=
E [ (cid:80) ∞ t =0 γ t R ( s t , a t ) | a t ∼ π ( ·| s t [-) , s 
 = s ]-])] where Π is the agent’s policy set.
[-E [ (cid:80) ∞ t =0 R ( s t , a t ) | a 0 = a, s 
-]
We consider a setting in which the agent faces at least some minimal cost for each action it performs.
With this, the agent’s task is to maximise:
[Equation]
where for any state s ∈ S and any action a ∈ A , the functions R and C are given by R ( s, a )
=
R ( s, a ) 1 a ∈A + R ( s, 0)(1 − 1 a ∈A ) where 1 a ∈A is the indicator function which is 1 when a ∈ A and 0otherwise and C ( s, a ) :
= c ( s, a ) 1 a ∈A where c : S ×
A → R is a minimally bounded (cost) function 1that introduces a cost each time the agent performs an action.
Examples of the cost function is aquasi-linear function of the form c ( s t , a t ) = κ
+ f ( a t ) where f : A → R > 0 and κ is a positive realvalued constant.
Since acting at each time step would incur prohibitively high costs, the agent mustbe selective when to perform an action.
Therefore, in this setting, the agent’s problem is augmented tolearning both an optimal policy for its actions and, learning at which states to apply its action policy.
Example: Merton Portfolio Problem with Transaction Costs [9].
An investor performs a series ofcostly portfolio adjustments by buying and selling amounts of different assets within their portfolio.
Each investment incurs a fixed minimal cost (also known as transaction costs ) which is deductedfrom the investor’s available cash-flow.
The investor’s aim is to maximise their total wealth (the valueof the sum of their assets) at some time horizon by adjusting their portfolio of investments.
Problemsof this kind, portfolio investment problems are of fundamental importance within finance [16].
Example 2.
An autonomous vehicle must perform a series of actions to perform a task.
Each actiondraws from its fuel budget.
In order to complete its task successfully, during the task, the vehiclemust ensure it maintains an available supply.
The LICRA Framework
In RL, the agent’s problem involves learning to act at every state including those in which actions donot significant impact on its total return.
While we can add a zero action to the action set A and applystandard methods, we argue that this may not be the best solution in many situations.
We argue the optimal policy has the following form:
[Equation]
which implies that we simplify policy learning by determining the set S I first — the set where weactually need to learn the policy.
We now introduce a learning method for producing impulse controls.
This enables the agent to learnto select states to perform actions.
Therefore, now agent is tasked with learning to act at states thatare most important for maximising its total return given the presence of the cost for each action.
Nowat each state the agent first makes a binary decision to decide to perform an action.
Our framework, LICRA consists of two core components: firstly a RL process g : S ×{ 0 , 1 } → [0 , 1]and a second RL process π : S ×
A → [0 , 1] .
The role of g is to determine whether or not an actionis to be performed by the policy π at a given state s .
If activated, the policy π determines the actionto be selected.
Prior to decisions being made, the policy π communicates to g the action it wouldtake.
An important feature of our LICRA is the sequential decision process .
In LICRA, the policyπ first proposes an action a ∈ A which is observed by the policy g .
Therefore, the role of g is toprevent actions for which the change in expected future rewards does not exceed the costs incurredfor taking such actions.
By isolating a decision policy over whether an action should be taken or not,the impulse controls mechanism results in a framework in which the problem facing the agent has amarkedly reduced decision space (in comparison to a standard RL method).
Crucially, the agent mustcompute optimal actions at only a subset of states which are chosen by the policy g .
Below is thepseudocode for LICRA, we provide full details of the code in Sec.
10 of the Appendix.
: Initialise:
Given reward B ) 4: for N episodes do 5: state s 0 , Reset Rollout Buffers B π , B g 6: for t = 0 , 1 , . . . do 7:
Sample a t ∼ π ( ·| s t ) 8:
Sample g t ∼ g ( ·| s t ) 9: if g t = 0 then 10:
Apply a t so s t +1 ∼ P ( ·| a t , s t ) , 11:
Receive rewards r t =
R ( s t , a t ) 12:
Store ( s t , a t , s t +1 , r t ) in B π 13: else 14:
Apply the null action so s t +1 ∼ P ( ·| 0 , s t ) , 15:
Receive rewards r t =
R ( s t , 0) .
16: end if 17:
Store ( s t , g t , s t +1 , r t ) in B g 18: end for 19: //
Learn the individual policies 20:
Update (using SAC or PPO) policy π and critic V π networks using B π 21:
Update (using SAC or PPO) policy g and critic V g networks using B g 22: end for
While we consider now two policies π , g , the cardinality of the action space does not change.
In thediscrete case the cardinality is still |A| + 1 , where |A| is cardinality of the action space for policy π .
Although action space cardinality does not change there are still benefits of using impulse controlmechanism.
This mechanism forces the agent to first determine the set of states to perform actionsonly then determine the optimal actions at these states.
An important fact to note is that the decisionspace for the determining whether or not to execute an action is S × { 0 , 1 } i.e at each state it makes abinary decision.
Consequently, the learning process for aspect is much quicker than a policy whichmust optimise over a decision space which is |S||A| (choosing an action from its action space at everystate).
This results in the agent rapidly learning which states to focus on to learn which actions to perform.
In the case of π with a continuous action space again the impulse control mechanism doesnot change the cardinality of the action space.
However, if the set S / S I , where the optimal policychooses 0 , is large enough, then again it can be more efficient to learn g first and only then learn π .
In Sec. 5, we show the convergence properties of LICRA.
LICRA consists of two independentprocedures: the agent learns its action policy π and simultaneously learns its impulse policy g whichdetermines at which states to perform an action.
In our implementation, we used proximal policyoptimisation (PPO) [23] for the policy π . For the impulse policy g , we used a soft actor critic(SAC) process [12] whose action set consists of two actions: intervene or do not intervene. As wedemonstrate in Sec. 11, LICRA is a plug & play framework which enables these RL components tobe replaced with any RL algorithm of choice.
Convergence and Optimality of LICRA
A key aspect of our framework is the presence of two RL processes that make decisions in a sequentialorder.
In order to determine when to act the policy g must learn the states to allow the policy π toperform an action which the policy π must learn to select optimal actions whenever it is allowed toexecute an action.
In this section, we prove that LICRA converges to an optimal solution of the system.
Central to
LICRA is a Q-learning type method which is adapted to handle RL settings in which the agent mustalso learn when to act.
We then extend the result to allow for (linear) function approximators.
Weprovide a result that shows the optimal intervention times are characterised by an ‘obstacle condition’which can be evaluated online therefore allowing the g .
Given a function Q π, g : S × A → R , ∀ π ∈ Π and g , ∀ s τ k ∈ S , we definethe intervention operator M π, g Q π, g by M π, g Q π, g ( s τ k , a τ k ) :
= R ( s τ k , a τ k ) − c ( s τ k , a τ k ) +γ (cid:80)
[Equation]
∈S
[Equation]
(cid:12)(cid:12)(cid:12) a τ k ∼ π , where τ k is an intervention time.
The interpretation of M is the following: suppose that the agent is using the policy π and at time τ kthe system is at a state s τ k and the agent performs an action a τ k ∼ π ( ·| s τ k ) .
A cost of c ( s τ k , a τ k )is then incurred by the agent and the system transitions to s ′ ∼ P ( · ; a τ k , s ) .
Lastly, recall v π, g isthe agent value function under the policy pair ( π, g ) .
Therefore, the quantity M v π, g measures theexpected future stream of rewards after an immediate intervention minus the cost of intervention.
This object plays a crucial role in the LICRA framework which as we later discuss, exploits the coststructure of the problem to determine when the agent should perform an intervention.
For any state s t ∈ S , we define the Bellman operator T by:
[Equation]
The Bellman operator captures the nested sequential structure of the LICRA algorithm.
In particular,the structure in (3) consists of an inner structure which consists of two terms: the first term is theexpected future return given an action is taken at the current state under the policy π .
The second termis the expected future return given no action is taken at the current state.
Lastly, the outer structure isan optimisation which compares the expected return of the two possibilities and selects the maximum.
Our first result proves T is a contraction operator in particular, the following bound holds:
Lemma [-
-]1.
The Bellman operator T is a contraction, that is the following bound holds:
[Equation]
where v, v ′ are elements of a finite normed vector space.
We can now state our first main result:
Theorem 1. Given any v π, g : S ×
A → R , the optimal value function is given by lim k →∞ T k v π, g =max ˆ [Equation] [Equation] ˆ
[Equation] [Equation] g = v π ⋆ ,g ⋆ where ( π ⋆ , g ⋆ ) is the optimal policy pair. The result of Theorem 1 enables the solution to the agent’s impulse control problem to be determinedusing a value iteration procedure.
Moreover, Theorem 1 enables a Q-learning approach [6] for findingthe solution to the agent’s problem.
Theorem 2. Consider the following Q learning variant: Q t
[Equation]
[Equation]
a
[Equation]
then Q t ( s ) converges to Q ⋆ with probability 1 , where s t , s t +1 ∈ S and a t ∈ A .
We now extend the result to (linear) function approximators:
Theorem 3. Given a set of linearly independent basis functions Φ = { ϕ 1 , . . . , ϕ p } with ϕ k ∈ L 2 , ∀ k .
LICRA converges to a limit point r ⋆ ∈ R p which is the unique solution to Π F (Φ r ⋆ ) =
Φ r ⋆ where
F Λ := ˆ R + γP max {M Λ , Λ } .
Moreover, r ⋆ satisfies: ∥ Φ r ⋆ − Q ⋆ ∥ ≤ (1 − γ 2 ) − 1 / 2 ∥
Π Q ⋆ − Q ⋆ ∥ .
The theorem establishes the convergence of LICRA to a stable point with the use of linear functionapproximators.
The second statement bounds the proximity of the convergence point by the smallestapproximation error that can be achieved given the choice of basis functions.
Having constructed a procedure to find the optimal agent’s optimal value function, we now seek todetermine the conditions when an intervention should be performed.
Let us denote by { τ k } k ≥ 0 thepoints at which the agent decides to act or intervention times , so for example if the agent choosesto perform an action at state s 6 and again at state s 8 , then τ 1 = 6 and τ 2 = 8 .
The following resultcharacterises the optimal intervention policy g and the optimal times { τ k } k ≥ 0 .
Proposition 1. The policy g is given by: g ( s [Equation]
= [Equation] ) , ∀ ( s t ) ∈ S×{ 0 , 1 } ,where Q π, g is the solution in Theorem 1, M is the intervention operator and H is the Heavisidefunction, moreover the intervention times are τ k = inf { τ > τ k − 1 |M π, g Q π, g = Q π, g } .
Prop. 1 characterises the (categorical) distribution g .
Moreover, given the function Q , the times { τ k }can be determined by evaluating if M Q = Q holds.
A key aspect of Prop. 1 is that it exploits the cost structure of the problem to determine when theagent should perform an intervention.
In particular, the equality M Q = Q implies that performingan action and incurring a cost for doing so is optimal.
Budget Augmented LICRA via State Augmentation
We now tackle the problem of RL with a budget.
To do this, we combine the above impulse controltechnology with state augmentation technique proposed in [24] The mathematical formulation of theproblem is now given by the following for any s ∈ S :
[Equation]
where n ∈ N is a fixed value that represents the maximum number of allowed interventions.
In orderto avoid dealing with a constrained MDP, we propose to introduce a new variable z t tracking theremaining number of impulses: z t = n − (cid:80) t − 1 i =0 (cid:80) k ≥ 1 δ iτ k .
We treat z t as another state and augmentthe state-space resulting in the transition (cid:101) P :
[Equation]
where (cid:80) k ≥ 1 δ tτ k is equal to one if there an impulse was applied at time t and zero if it was not. Inorder to avoid violations, we reshape the reward as follows:
[Equation]
where ∆ > 0 is a large enough hyper-parameter ensuring that there are no safety violations.
Tosummarise we aim to solve the following problem:
[Equation]
where the policy now depends on the variable z t .
Note that (cid:101) P in Equation 5 is a Markov processand, the rewards (cid:101) R are bounded, as long as the rewards R are bounded.
Therefore, we can applydirectly the results for impulse control to this case as well.
We denote the augmented MDP by(cid:102)
M = ⟨S ×
Z , A , (cid:101) P , (cid:101) R, γ ⟩ , where Z is the space of the augmented state.
We have the following.
Theorem 4. Consider the MDP (cid:102) M for the problem 6, then:
a)
The Bellman equation holds, i.e. there exists a function ˜ v ∗ , π , g s.th.
[Equation]
[Equation] [Equation]
, where the optimal policy for (cid:102)
M has the formπ ∗ ( ·| s, z ) ;
b) Given a (cid:101) v : S×Z → R , the stable point solution for (cid:102)
M is a given by lim k →∞ ˜ T k (cid:101) [Equation] π [Equation]
g = max ˆ π ∈ Π ,
[Equation]
g (cid:101)
[Equation]
ˆ π [Equation] g =(cid:101) v ∗ , π , g ∗ , where ( π ∗ , g ∗ ) is an optimal policy of (cid:102) M and ˜ T is the Bellman operator of (cid:102) M .
The result has several important implications.
The first is that we can use a modified version of LICRAto obtain the solution of the problem while guaranteeing convergence (under standard assumptions).
Secondly, our state augmentation procedure admits a Markovian representation of the optimal policy.
Experiments
We will now study empirically the performance of the LICRA framework.
In experiments, we usedifferent instances of LICRA, one where both policies are trained using PPO update (referred toas LICRA_PPO ) and one where the policy deciding whether to act is trained using SAC and theother policy trained with PPO (referred to as LICRA_SAC ).
We have benchmarked both of thesealgorithms together with common baselines on environments, where it would be natural to introducethe concept of the cost associated with actions.
Further experiment details are in the Appendix.
Merton’s Portfolio Problem with Transaction Costs.
Merton Investment Problem in which the investor faces transaction costs [9] is a well-known problemwithin finance.
In our environment, the agent can decide to move its wealth between a risky asset anda risk-free asset.
The agent receives a reward only at the final step, equal to the utility of the portfoliowith a risk aversion factor equal to 0 .
If the final wealth of risky asset is s T and final wealth ofrisk-free asset is c T , then the agent will receive a reward of u ( x ) = 2 √ s T + c T .
The wealth evolvesaccording to the following SDE:
[Equation]
where W t is the current wealth, dB t is an increment of Brownian motion and p t is the proportionof wealth invested in the risky asset. In our experiments we set the risk-free return r = 0 . 01 , riskyasset return µ = 0 .
05 and volatility σ = 1 .
We discretise the action space so that at each step the
1.0 Training Steps (mil) 1e 1.
A v e r a g e E p
i s o d e R e t u r n
LICRA PPO
SAC CPO
PPO-Lagrangan TRPO-Lagrangian
[Figure]
(b) agent has three actions available: move 10 % of risky asset wealth to the risk-free asset, move 10 % ofrisk-free asset wealth to the risky asset or do nothing.
Each time the agent moves the assets, it incursa cost of 1 , which can be thought of as a transaction fee.
The agent can act after a time interval of0 .
01 seconds and the episode ends after 75 steps.
The results of training are shown in Figure 1 andthey clearly demonstrate that LICRA finds a better policy than standard PPO and learns much fasterthan SAC.
Also comparing the variance among different seeds, we can see that LICRA is a muchmore stable algorithm than the other two.
Driving Environment Fuel Rationing.
We studied an autonomous driving scenario where fuel-efficient driving is a priority.
One of the maincomponents of fuel-efficient driving is controlled usage of acceleration and braking, in the sense that1) one wants to limit the amount of acceleration and braking altogether and 2) if ones need to speedup or slow down it should be done slowly and gently.
Intuitively, we believe this is a problem where
LICRA should thrive as the impulse control agent can learn to restrict the amount of accelerationand braking dependent on current velocity and the presence of other cars in favour of, for example,allowing the car to decelerate naturally.
We utilise the highway-env [15] environment on a simplehighway task (see Fig (2. a)) where the green vehicle is our controlled vehicle) where the goal isto avoid crashing into other vehicles whilst driving at a reasonable speed.
We add a cost functioninto the reward term dependent on the continuous acceleration action, C ( a t )
=
K + a 2 t , where Kis a fixed constant cost of taking any action, and a t ∈ [ − 1 , 1] , with larger values of acceleration orbraking being penalised more.
In Fig. (2.b) we demonstrate our results.
Notably, LICRA is able to massively outperform thebaselines, especially our safety specific baselines which struggle to deal with the cost functionassociated with the environment.
We believe one reason for the success of LICRA is that it is fareasier for it to utilise the null action of zero acceleration/braking than the other algorithms, whilstall the algorithms have a guaranteed cost at every time step whilst not gaining a sizeable reward tocounter the cost.
Lunar Lander Environment.
We have used the Lunar Lander environment with OpenAI gym and manually added minimal bounded costs in the reward definition.
The agent needs to maintain both a good posture mid-air and reach the landing pad as quickly as possible.
Therefore, the reward function is given by:
[Equation]
where d t is the distance to the landing pad, v t is the velocity of the agent, and ω t is the angularvelocity of the agent at time t .
1 X is the indicator function of taking actions, which is 1 when thestatement X is true and 0 when X is false.
Considering the limited fuel budget, we assume that [Figure] we have a fixed cost for each action taken by the agent here, and doing nothing brings no cost.
Then,to describe the goal of the game, we define the function of the final status by hasLanded(), which is0 when not landing; 1 when the agent has landed softly on the landing pad; and − 1 when the landerruns out of fuel or loses contact with the pad on landing.
The reward function rewards the agentfor reducing its distance to the landing pad, decreasing its speed to land smoothly and keeping theangular speed at a minimum to prevent rolling.
Additionally, it penalises the agent for running outof fuel and deters the agent from taking off again after landing.
By introducing such a reward function with minimally bounded costs, our goal was to test if LICRAcan exploit the optimal policy.
In Fig.
3, we observe that the LICRA agent outperforms all the baselineagents, in terms of both the sample efficiency and the average test return (the total of rewards in eachtimestep).
Moreover, we observe that LICRA enables more stable training than PPO, PPO-Lagrangianand CPO.
On the other hand, we observe that SAC is more suitable than PPO in this scenario, alsowhen adopting impulse control.
Conclusion
We presented a novel method to tackle the problem of learning how to select when to act in additionto learning which actions to execute.
Our framework, which is a general tool for tackling problemsof this kind seamlessly adopts RL algorithms enabling them to efficiently tackle problems in whichthe agent must be selective about when it executes actions.
This is of fundamental importance inpractical settings where performing many actions over the horizon can lead to costs and underminethe service life of machinery.
We demonstrated that our solution, LICRA which at its core has asequential decision structure that first decides whether or not an action ought to be taken under theaction policy can solve tasks where the agent faces costs with extreme efficiency as compared toleading reinforcement learning methods.
In some tasks, we showed that LICRA is able to solveproblems that are unsolvable using current reinforcement learning machinery.
We envisge that thisframework can serve as the basis extensions to different settings including adversarial training forsolving a variety of problems within RL.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]