A CTING IN D ELAYED E NVIRONMENTS WITH N ON -S TATIONARY M ARKOV P OLICIES
Anonymous authors
Paper under double-blind review
A BSTRACT
The standard Markov Decision Process (MDP) formulation hinges on the assumption that an action is executed immediately after it was chosen.
However, assuming it is often unrealistic and can lead to catastrophic failures in applications such as robotic manipulation, cloud computing, and ﬁnance.
We introduce a framework for learning and planning in MDPs where the decision-maker commits actions that are executed with a delay of m steps.
The brute-force state augmentation baseline where the state is concatenated to the last m committed actions suffers from an exponential complexity in m , as we show for policy iteration.
We then prove that with execution delay, Markov policies in the original state-space are sufﬁcient for attaining maximal reward, but need to be non-stationary.
As for stationary Markov policies, we show they are sub-optimal in general.
Consequently, we devise a nonstationary Q-learning style model-based algorithm that solves delayed execution tasks without resorting to state-augmentation.
Experiments on tabular, physical, and Atari domains reveal that it converges quickly to high performance even for substantial delays, while standard approaches that either ignore the delay or rely on state-augmentation struggle or fail due to divergence.
To preserve anonymity, the code will be shared upon publication.
I NTRODUCTION
The body of work on reinforcement learning (RL) and planning problem setups has grown vast in recent decades.
Examples for such distinctions are different objectives and constraints, assumptions on access to the model or logged trajectories, on-policy or off-policy paradigms, etc.
(Puterman, 2014).
However, the study of delay in RL remains scarce.
It is almost always assumed the action is executed as soon as the agent chooses it.
This assumption seldom holds in real-world applications (DulacArnold et al., 2019).
Latency in action execution can either stem from the increasing computational complexity of modern systems and related tasks, or the infrastructure itself.
The wide range of such applications includes robotic manipulation, cloud computing, ﬁnancial trading, sensor feedback in autonomous systems, and more.
To elaborate, consider an autonomous vehicle required for immediate response to a sudden hazard on the highway.
Driving at high speed, it suffers from perception module latency when inferring the surrounding scene, as well as delay in actuation once a decision has been made.
While the latter phenomenon is an instance of execution delay , the former corresponds to observation delay .
These two types of delay are in fact equivalent and can thus be treated with the same tools (Katsikopoulos & Engelbrecht, 2003).
Related works.
The notion of delay is prominent in control theory with linear time-invariant systems (Bar-Ilan & Sulem, 1995; Dugard & Verriest, 1998; Richard, 2003; Fridman, 2014; Bruder & Pham, 2009).
While the delayed control literature is vast, our work intersects with it mostly in motivation.
In the above control theory formulations, the system evolves according to some known diffusion or stochastic differential equation.
Differently, the discrete-time MDP framework does not require any structural assumption on the transition function or reward.
A few works consider a delay in the reward signal rather than in observation or execution.
Delayed reward has been studied on multi-armed bandits for deterministic and stochastic latencies (Joulani et al., 2013) and for the resulting arm credit assignment problem (Pike-Burke et al., 2017).
In the MDP setting, Campbell et al.
(2016) proposed a Q-learning variant for reward-delay that follows a Poisson distribution.
Katsikopoulos & Engelbrecht (2003) considered three types of delay: observation, execution, and reward.
Chen et al.
(2020b) studied execution delay on multi-agent systems.
The above works on MDPs employed state-augmentation with a primary focus on empirical evaluation of the degradation introduced by the delay.
In this augmentation method, all missing information is concatenated with the original state to overcome the partial observability induced by the delay.
The main drawback of this embedding method is the exponential growth of the state-space with the delay value (Walsh et al., 2009;
Chen et al., 2020a) and, in the case of (Chen et al., 2020b), an additional growth that is polynomial with the number of agents.
Walsh et al.
(2009) avoided state-augmentation in MDPs with delayed feedback via a planning approach.
By assuming the transition kernel to be close to deterministic, their model-based simulation (MBS) algorithm relies on a most-likely present state estimate.
Since the Delayed-Q algorithm we devise here resembles to MBS in spirit, we highlight crucial differences between them: First, MBS is a conceptual algorithm that requires the state-space to be ﬁnite or discretized.
This makes it highly sensitive to the state-space size, as we shall demonstrate in Sec.
7[Fig. 4(c)], prohibiting it from running on domains like Atari. Differently, Delayed-Q works with the original, possibly continuous state-space. Second, MBS is an ofﬂine algorithm: it estimates a surrogate, non-delayed MDP from samples, and only then does it solve that MDP to obtain the optimal policy (Walsh et al., 2009)[Alg. 2, l. 16]. This is inapplicable to large continuous domains and is again in contrast to Delayed-Q.
Recent studies considered a concurrent control setting where action sampling occurs simultaneously with state transition (Ramstedt & Pal, 2019; Xiao et al., 2020).
Both assumed a single action selection between two consecutive observations, thus reducing the problem to an MDP with execution delay of m “ 1 .
Chen et al.
(2020a) have generalized it to an arbitrary number of actions between two observations.
Hester & Stone (2013) addressed execution delay in the braking control of autonomous vehicles with a relatively low delay of m ď 3 .
All these works employ state-augmentation to preserve the Markov property of the process, whereas we are interested whether this restriction can be lifted.
Additionally, they studied policy-gradient (policy-based) methods, while we introduce a Q-learning style (value-based) algorithm.
Likewise, Firoiu et al.
(2018) proposed a modiﬁed version of the policy-based IMPALA (Espeholt et al., 2018) which is evaluated on a single video game with delay values of m ď 7 .
To the best of our knowledge, our work is the ﬁrst to tackle a delayed variant of the popular Atari suite (Bellemare et al., 2013).
Contributions.
Revisiting RL with execution delay both in theory and practice, we introduce:
An analytical study of delay on an example MDP quantifying the trade-off between stochasticity and delay.
The ﬁrst tight upper and lower complexity bounds on policy iteration for action-augmented MDPs.
We stress that this is also a contribution for the general RL theory on standard non-delayed MDPs.
A new formalism of execution-delay MDPs that avoids action-embedding.
This leads to (i) the formal understanding that restricting to non-stationary Markov policies is sufﬁcient for optimality in delayed MDPs, and (ii) a Bellman-type recursion for a delayed value function .
A model-based DQN-style algorithm that yields non-stationary Markov policies.
Our algorithm outperforms the alternative standard and state-augmented DDQN in 47 ofexperiments spanning over 3 environment categories and delay of up to m “ 25 .
P RELIMINARIES : N ON -D ELAYED S TANDARD MDP
Here, we describe the standard non-delayed MDP setup.
Later, in Sec. 5, we introduce its generalization to the delayed case.
We follow and extend notations from (Puterman, 2014)[Sec. 2.1.].
An inﬁnite horizon discounted MDP is a tuple p S , A , P, r, γ q where S and A are ﬁnite state and action spaces, P : S
ˆ A Ñ ∆ S is a transition kernel, the reward r : S ˆ A Ñ R is a bounded function, and γ P r 0 , 1 q is a discount factor.
At time t , the agent is in s t and draws an action a t according to a decision rule d t that maps past information to a probability distribution q d t over the action set.
Once a t is taken, the agent receives a reward r p s t , a t q .
A decision rule can be history-dependent (H) or Markovian (M) , and randomized (R) or deterministic (D).
Denote by H t the set of possible histories up to time t.
Then, a history-dependent decision-rule is given by d t : H t Ñ ∆
A with h t ÞÑ q d t p h t q p¨q .
A Markovian decision-rule, on the other hand, maps states to actions, i.e., d t : S Ñ ∆ A with s ÞÑ q d t p s q p¨q .
A policy π : “ p d t q t ě 0 is a sequence of decision rules whose type dictates that of the policy.
It can be either Markovian deterministic ( Π MD ) or randomized ( Π MR ), history-dependent deterministic ( Π HD ) or randomized ( Π HR ).
It is stationary if its decision rules do not depend on time, i. e., d t “ d for all t ě 0 .
This deﬁnes the smaller class of stationary policies: deterministic ( Π SD ) and randomized ( Π SR ).
Note that stationary policies are inherently Markovian.
Indeed, at time t “ 0 , d : H 0 Ñ ∆ A is state-dependent because H 0 “ S .
Since the policy is stationary, i. e., d t “ d @ t , subsequent decision rules are also state-dependent, thus
Markovian.
This makes Π HR the most general set and Π SD the most speciﬁc.
We denote probability model by P π 0 , where the subscript 0 stands for the delay value m “ 0 .
The related random variables are denoted by ˜ s t P S , ˜ a t P A and ˜
h t
P p S ˆ A q t ˆ S .
The value function given policy π P Π HR is deﬁned as v π p s q “ E π
[Equation]
, where the expectation is taken with respect to (w.r.t.)
P π 0 p¨| ˜ s
“ s q .
Let the optimal value function
[Equation]
Our goal is to ﬁnd a policy π ˚ that yields v ˚ , and it is known that focusing on stationary deterministic policies π P Π SD is sufﬁcient for reaching the optimum in (1) (Puterman, 2014)[Thm.
MDP S WITH D ELAY : A D EGRADATION E XAMPLE
[Equation]
In an MDP with execution delay 1 m , any action chosen at time t is executed at t ` m .
Therefore, at each step, the agent witnesses the current state and action being executed, but selects a new action that will be applied in a future state.
We assume that m decided actions are already awaiting execution at t “ 0 , so at any given time, the queue of pending actions is of constant length m .
As we illustrate in the next example, having a delay generally comes at a price.
Example 3.1 (Two-state MDP) .
Consider the MDP in Fig.
It has two states and two actions: S “ t s 0 , s 1 u , A “ t a 0 , a 1 u .
The transition kernel is independent of the action: for all s, s 1 P S s.t.
s ‰ s 1 , P p s 1 | s, a q “ P p s 1 | s q “ p where p P r 0 .
5 , 1 s .
The reward is positive for one of the two actions only: [Equation]
We inspect the return obtained from the commonly used set of stationary deterministic policies Π SD .
As expected, the highest possible return is attained when m “ 0 , but monotonically decreases with the delay, m, and increases with the level of certainty,
p. We analytically quantify this effect in the following and give a proof in Appx.
A.1.
Proposition 3.1.
For delay m P N and p P r 0 .
5 , 1 s , the optimal return of π ˚ P Π SD is 1 `p 2 p ´ 1 q m 2 p 1 ´ γ q .
Remark 3.1.
This result demonstrates a clear tradeoff between stochasticity and delay.
For p Ñ 0 .or
m Ñ 8 , the return goes to its minimal value of 0 .
5 {p 1 ´ γ q .
Contrarily, for p Ñ 1 or m Ñ 0 , it goes to its maximal value of 1 {p 1 ´ γ q .
T HE A UGMENTATION A PPROACH
In this section, we consider state-augmentation for solving MDPs with execution delay.
We begin with deﬁning an equivalent MDP with a larger state space that memorizes all missing information for an informed decision.
Due to the full observability, the resulting optimal augmented policy attains the optimal return in the original delayed MDP.
Deﬁnition 4.1 ( m -AMDP) .
Given MDP p S , A , P, r, γ q and m P N , an m -Augmented MDP ( mAMDP) is a tuple p X m , A , F, g, γ q such that X m : “ S ˆ A m is the augmented state-space,
A the original action-space, F is the transition matrix given in Appx.
B.1, (14) , and g is the reward function given in Appx.
B.1, (15) .
The pending action queue is concatenated to the original state to form an augmented state x t : “ p s t , a ´ 1 t , ¨ ¨ ¨ , a ´ m t q P X m , where a ´ i t is the i -th pending action at time t .
It means that in the following step, t ` 1 , action a ´ m t will be executed independently of the present action selection, the queue will shift to the right, and the newly selected action will be at the second coordinate.
By construction, the m -AMDP is non-delayed; it directly accounts for execution delay through its state-representation, as opposed to our coming formulation in Sec.
We further deﬁne a stationary deterministic policy ¯ π P ¯Π SD m with corresponding decision rule ¯ d : X m Ñ ∆ A and augmented value function v ¯ π p [Equation] q : “ E [Equation]
“ř [Equation] [Equation] p
[Equation]
t , ˜ a t q| ˜ x 0 “ x ‰ .
As in (1), our goal is to solve ¯
[Equation] We now analyze the classical Policy Iteration (PI) algorithm (Howard, 1960) for m -augmented
MDPs and provide a ﬁnite-time analysis of its convergence.
We refer to it as m A-PI and provide its pseudo-code in Appx.
B.2.
We consider PI since it is a canonical representative upon which many other algorithms are built.
Admittedly, we did not ﬁnd any other formal result quantifying the effect of augmentation on a planning or learning algorithm, other than a PAC upper bound for R-max with (cid:15) -optimal
policies
Walsh et al., 2009).
A proof for the next result is given in Appx.
B.4.
Theorem 4.1 (Lower Bound for m A-PI) .
The number of iterations required for m A-PI to converge in m -AMDP
M m is Ω p| X m |q “ Ω p| S
||
A | m q . Thm.
4.1 does not take advantage of the special delay problem structure but rather is an application of our more general result to augmented MDPs (Appx.B.4). As pointed out in Scherrer et al. (2016), the lower-bound complexity of PI is considered an open problem, at least in the most general MDP formulation. Lower-bounds have been derived in speciﬁc cases only, such as deterministic MDPs (Hansen & Zwick, 2010), total reward criterion (Fearnley, 2010) or high discount factor (Hollanders et al., 2012).
Even though we did not intend to directly address this open question, our lower bound result seems to be a contribution on its own to the general theory of non-delayed MDPs. Next, we show that the above lower bound is tight (up to a factor of |A| and logarithmic terms) and m A-PI is guaranteed to converge after ˜ O p |S||A| m ` 1 q .
A proof is given in Appx.
B.5.
Theorem 4.2 ( m A-PI Convergence) .
The m A-PI algorithm converges to the optimal value-policy pair p ¯ v ˚ , ¯ π ˚ q in at most |S||A| m p |A|
[Equation]
E XECUTION -D ELAY MDP: A N EW F ORMULATION
In this section, we introduce and study the stochastic process generated by an MDP with execution delay, without resorting to state-augmentation.
In the ED-MDP we consider, the probability measure changes according to the delay value m .
We assume that during the m initial steps, actions are sequentially executed according to a ﬁxed queue ¯ a : “ p ¯ a 0 , ¨ ¨ ¨ , ¯ a m ´ 1 q P
A m .
Unlike m -AMDPs, the initial queue of pending actions here plays the role of an exogenous variable that is not embedded into the state-space.
A policy π P Π HR induces a probability measure P πm that is deﬁned through a set of equations which, for brevity, we defer to Appx. C[(16)-(19)].
We just mention that for t ă m , decision rules do not depend on the history, while for t ě m , they depend on the history up to t ´ m only.
Let µ be an initial state distribution and δ a Dirac distribution.
Using this and the notations from Sec. 2, we can explicitly write the probability of a sample path.
See proof in Appx.
C.1.
Proposition 5.1.
For policy π : “ p d 0 , d 1 , ¨ ¨ ¨ q P Π HR , the probability of observing history h t : “ p s 0 , a 0 , s 1 , a 1 ¨ ¨ ¨ , a t ´ 1 , s t q is given by:
[Equation]
From Prop. 5.1 we deduce that, differently than the standard MDP setting where any Markov policy induces a Markov process, the delayed process is not Markovian even for stationary policies (see
Appx.
C.2 for a formal proof).
Next, we show that for any history-dependent policy and starting state, there exists a Markov policy (not necessarily stationary) that generates the same process distribution.
Consequently, despite execution delay, one can restrict attention to Markov policies without impairing performance.
See proof in Appx.
C.3.
Theorem 5.1.
Let π : “ p d 0 , d 1 , ¨ ¨ ¨ q P Π HR be a history dependent policy.
For all s 0 P S , there exists a Markov policy π 1 : “ p d 1 0 , d 1 1 , ¨ ¨ ¨ q P Π MR that yields the same process distribution as π , i.e.,
[Equation]
The formal proof is given in Appx.
C.3.
It builds on the concept that for each history-dependent policy π HR , one can choose a sequence of Markov decision rules that reconstruct the same time-dependent action distribution in the process induced by Π HR .
This result proves attainability of the optimum over Π MR , but not how one can efﬁciently ﬁnd an optimal policy.
In Appx.
C.5, we formally deﬁne a delayed value function and show that it satisﬁes a non-stationary Bellman-type recursion.
The questions of how to efﬁciently ﬁnd an optimal non-stationary Markov policy and the role of the delayed value function in that process remain open.
Degradation due to stationarity.
As to complement the ﬁnding that a Markov policy can be optimal for any ED-MDP, we show that restricting to stationary policies impairs performance in general.
Therefore, while in non-delayed MDPs it is enough to focus on stationary policies, in ED-MDPs the restriction should be to the more general class of Markov policies.
Proposition 5.2.
There exists an m -ED-MDP for which all stationary policies are sub-optimal.
This result follows from computing the optimal return for stationary and non-stationary policies in the ED-MDP from Example 3.1 using simulation.
We elaborate on this further in Appx.
C.4.
There, we also conﬁrm that our theoretical return from Prop. 3.1 matches closely with simulation.
[Equation]
[Figure]
In this section, we introduce a new algorithm capable of successfully handling tasks with execution delay.
To solve such delayed MDP, the agent infers the future m -step state whenever it chooses an action. Algorithm Description.
The algorithm is depicted in Fig.
As a ﬁrst stage, to select an action a t to be executed in a future state s t ` m , we infer that future state ˆ s t ` m using the current state s t and the queue of pending actions p a t ´ m , . . .
, a t ´ 1 q .
This is done by successively applying an approximate forward model m times: ˆ s t ` 1 “ f p s t , a t ´ m q , . . .
, ˆ s t ` m “ f p ˆ s t ` m ´ 1 , a t ´ 1 q .
More details on the forward models used in the three environment types are given next, in Sec.
Compared to other model-based algorithms such as tree-search methods, the approximate model here is simpler since it does not require access to the reward function.
Also, only a single trajectory is sampled rather than exponentially many w.r.t.
the horizon length.
We do note this method relies on the environment not being entirely stochastic (Walsh et al., 2009).
Still, as we shall show next, it performs well even in moderately noisy environments.
As a second stage, we select an action according to a policy a t “ π p ˆ s t ` m q .
The two stages of this procedure can be represented as a non-stationary Markov policy π t p s t q , where the non-stationarity stems from the time-dependency via the action queue, and the Markov property from the policy being applied on s t and no other prior history.
Notably, the Q-function here does not take as input the action history, as opposed to the augmentation approach in Sec. 4.
To better stress the non-stationarity, we note that applying the policy on the same state at different times can yield different action probability outcomes.
Lastly, for training, we maintain a sample-buffer of length m which we use to shift action a t into the tuple p s t ` m , r t ` m , a t , s t ` m ` 1 q prior to each insertion to the replay buffer. We note that during the course of this work, we also experimented with a model-free variant. Instead of ‘un-delaying’ the Q-function with the forward-model, we deﬁned a delayed Q-function trained on sequences whose actions were shifted m steps forward. However, the results were unsatisfactory, assumingly because of the incapacity of the Q-function to implicitly learn the m -step transition.
Point-Estimate Approaches.
For completeness, we also mention alternatives to using a ‘most-likely’ state estimate, such as taking an expected future state instead.
To demonstrate why point-estimate prediction can be devastating, consider an MDP where two parameters deﬁne the state s “ p x, t q : position and time, respectively. Starting from p 0 , 0 q , t progresses deterministically, while x behaves like a random walk with momentum; i. e., if x ą 0 , then x ` 1 is more likely than x ´ 1 , while if x ă 0 , x ´ 1 is more likely than x ` 1 . The process obviously diverges with time. Now, suppose there are two actions: one is good in extreme values (when | x | is big) and one is good in values close to 0 (when | x | is small).
Suppose also that the delay m is large.
The PDF of the state is bi-modal and symmetric around p Z, m q and p´ Z, m q for some ﬁnite Z , but a point estimate (e. g., ML or
MAP) would yield a value of p 0 , m q .
In addition to this example, we observe that in our Ex. 3.1, any alternative to a ’most-likely’ state estimate is worse: there, the optimal policy applies actions based on the most-likely state (see proof of Prop. 3.1), while it is easy to see that any other policy weighing future state probabilities leads to lower reward.
We perform experiments in a wide range of domains: tabular, physical, and image-based Atari.
All of them include stochasticity: In the maze we inject noise to actions; in the physical domains we perturb the masses at each step; and Atari is stochastic by nature.
We compare our algorithm with two baselines: Oblivious-Q and Augmented-Q .
Oblivious-Q is the standard Q-learning that ignores delay and assumes each decided action to be immediately executed.
Augmented-Q acts on the m ´ AMDP introduced in Def. 4.1.
We test all domains on delays m P t 0 , 5 , 15 , 25 u with 5 seeds per each run.
All results are summarized in Fig. 9, and in a more detailed version with standard deviations in Appx.
D.1, Table 2.
Tabular Maze Domain.
We begin with testing Delayed-Q on a Maze domain (Brockman et al., 2016)
[ tinyurl.com/y34tmfm9 ].
It is based on textbook tabular Q-learning and enables us to study the merits of our method decoupled from the DDQN added complexities.
Moreover, it conveys the exponential complexity of Augmented-Q.
The forward-model we construct is naturally tabular as well: it predicts a state s 1 according to the highest visitation frequency given p s, a q .
The objective in
Maze is to ﬁnd the shortest path from a start position to a goal state in a randomly-generated N ˆ N maze.
Reaching the goal yields a reward of 1 , and ´ 1 {p 10 N 2 q per step otherwise.
The maximal episode length is 10 N 2 steps, so the cumulative reward is in r´ 1 , 1 s .
We also create a Noisy Maze environment that perturbs each action w.p.
p P r 0 , 0 .
5 s .
Convergence plots are given in Fig.
Delayed-Q outperforms the rest for all delay values m , while
Oblivious-Q fails in all runs for m ą 0 .
Since the augmented state-space grows exponentially with m , Augmented-Q converges more slowly as m increases.
In fact, for m ą 15 the simulation fails to run due to memory incapacity for the Q-table; this explains its absence in Figs.
To conﬁrm the exponential complexity growth of Augmented-Q and compare it with that of Delayed-Q, we trained both agents on Maze with increasing delay values, and reported the number of training episodes each one required before reaching a cumulative reward of 0 . 5 . Fig. 3 clearly demonstrates the exponential (resp.
linear) dependence of Augmented-Q (resp. Delayed-Q) in the delay value. The linear dependence of Delayed-Q in m is not surprising: algorithmically, it is identical to Q-learning, except for two parts: the m -step forward-model calls, and the replay buffer shift of m samples.
To further analyze the sensitivity of tabular Delayed-Q to the state-space size, we ran Delayed-Q with increasing maze sizes, for a ﬁxed m “ 5 .
As Fig. 4(c) shows, the performance drops exponentially fast, suggesting high sensitivity of Delayed-Q to the state-space size and highlighting one shortcoming of the MBS method (Walsh et al., 2009)
(see Sec. 1).
Physical Domains.
Next, we test our approach on two continuous domains: CartPole 2 and Acrobot.
The CartPole task requires balancing a pole connected to a cart that actuates left or right.
In Acrobot, one needs to swing up the lower of two links connected by a joint above a certain height.
The agent receives a reward of 1 if the pole stays above a certain angle in Cartpole, and in Acrobot it receives ´ 1 until it reaches the goal.
The episode length is 500 steps in both tasks.
We also create noisy [Figure]
versions of both tasks: At each step, normal additive noises are independently added to each physical component’s mass, with std of 0 .
1 of the nominal mass.
To solve these tasks, we extend the DDQN algorithm (Van Hasselt et al., 2015), which is a known and common baseline.
Nonetheless, our method is general and can be seamlessly integrated into any Q-learning based algorithm.
Our one-step forward-model is implemented with a neural network of the same architecture as the one used to estimate the Q-function.
Namely, it consists of two hidden layers, each of width 24, with ReLu activations.
The input of the forward-model NN is the concatenation of p s, a q and its output is s 1 .
Training the forward-model NN is conducted together with the Q-network training with the same hyperparameters and sample batches; this makes the implementation easy and simple.
For Augmented-Q, a concatenation of the pending actions to the state is fed to the Q-network.
We depict the performance of the three algorithms for different values of m for Noisy Cartpole in Fig.
As expected from a physical domain, ignoring the delay gives catastrophic results even for m “ 5 .
Augmented-Q performs moderately up to m “ 15 , but fails with larger delays.
Delayed-Q performs the best for all m values and performs well, even for the challenging task of balancing a noisy pole with m “ 25 .
We observed similar behavior in all Cartpole and Acrobot experiments, as shown in Fig.
Moreover, in Fig.
6, we demonstrate the relative robustness of Delayed-Q to different delay values on Maze, Noisy Cartpole, and Noisy Acrobot.
All tested environments exhibit superior performance of Delayed-Q for a wide range of delays. In Noisy Acrobot, Delayed-Q with very high delay ( m “ 25 ) performs better than the alternatives with very small delay ( m “ 2 ).
In
Figs. 4(a)-4(b), we see a clear trade-off between noise and delay: noise and delay adversely affect performance, as we also discuss in Rmk. 3.1.
When the delay is signiﬁcant, the agent is much more sensitive to an increase in stochasticity.
To quantify the dependence of Delayed-Q on the quality of the model, we compared the learned model to a perfect one, i.e., the environment itself.
As seen in Fig.
8, model error is small enough to not affect performance for m “ 5 , but starts impairing it for m “ 15 and m “ 25 . Thus, for large delays, a better model can potentially improve reward by 20-30%. Further, we test the robustness of Delayed-Q to misspeciﬁed delay. Namely, we train it with m “ 10 , but then evaluate on additional delay values it has not seen before.
Fig.
7 shows the evaluation performance for m P t 5 , . .
, 15 u .
It demonstrates the robustness of our method – varying performance in evaluation (for good or bad) does not stem from delay misspeciﬁcation.
Instead, the delay is ‘forgotten’ after training, and Fig.
7 depicts the general effect of execution delay on performance.
For shorter delay than the training one, i. e., m ă 10 , performance even improves.
The reason is that, ﬁrst, during training, the Q-function is ‘un-delayed’ due to the replay buffer shift that relates the actions to the correct execution time.
Second, the forward-model is trained based on single-step transitions and only during inference is it queried m times.
Thus, these two networks composing the agent are oblivious to the delay they were trained on.
Atari Domains. We further investigate the Atari Learning Environment (Bellemare et al., 2013).
We inspect 10 arbitrary games that were successfully tackled with the original Q-network architecture and hyperparameters of DDQN (Van Hasselt et al., 2015).
Since a learned forward-model for images conditioned on actions is a hanging question in the research frontier, we leave it for future research and use the simulator itself for prediction.
It is stochastic in nature and thus encompasses approximation error.
For Augmented-Q, we concatenate the action queue to the output of the CNN part of the
Q-network; the extended vector is then fed into the subsequent fully-connected part of it.
We train all games for 1M steps.
Fig.
5 shows convergence plots for Boxing.
Delayed-Q is consistently better than Augmented-Q for all m values, which is, in turn, better than Oblivious-Q. Although the gap between all three algorithms is relatively small for m “ 5 , it increases with m .
For m “ 25 , the delay is too large for the augmentation to have a positive effect compared to Oblivious-Q, and they perform the same.
This behavior is representative of all Atari games, as can be seen in Fig.
Lastly, we compared Delayed-Q with a fourth algorithm which uses an RNN-based policy that is unaware of the delay value. The results are given in Appx. D.2, showing that a recurrent policy does not improve upon Augmented-Q or Oblivious-Q. This result is not surprising though: as stated in Thm.
5.1, the history sequence s t ´ m , s t ´ m ´ 1 , . . .
does not aid the policy any further than only using s t ´ m .
D ISCUSSION
The main message of this work is that dealing with execution delay requires non-stationary Markov policies.
The standard state augmentation approach, though optimal in terms of expressiveness, is intractable for all but the shortest delays.
The oblivious approach that ignores delay suffers from inferior performance and breaks down at some point when the system is stochastic enough.
We derived a Q-learning based algorithm that generates a Markov policy.
Our approach is based on combining a forward model with a policy learning one.
A thorough simulation study demonstrates the efﬁcacy of the proposed method.
While we focused on execution delay, we note that execution delay and observation delay where at time t the agent observes the state at t ´ m but the action is performed immediately are equivalent (Katsikopoulos & Engelbrecht, 2003).
As such, our results apply to the observation delay model.
The forward-model we used produces a simple future-state estimate.
Incorporating probabilistic estimates and other improvements such as integration of image-based action-dependent learned forward-models (Kim et al., 2020), are left for future research.
As for real-world applications, extensions of our work that are likely to be important are unknown or varying delay values.
In the ﬁrst case, a good prior for the delay value can often be used, e. g., for autonomous vehicles, as the latency statistics of the different hardware and software components are well studied (Zhao et al., 2019; Niu et al., 2019), while in production systems they are almost constant (Toschi et al., 2019). Our algorithm is also readily extendable to the second case of varying delay.
Differently than the augmentation approach, our 1-step forward-model decouples the algorithm from the delay used for training, as Fig. 7 depicts. Also, quantiﬁcation of the delay is not essential as long as the forward model can operate with variable delay values.
Finally, our framework can be extended to policy-gradient based methods that are particularly useful for continuous control, where observation delay is inherent.