R EINFORCEMENT L EARNING WITH R ANDOM D ELAYS
Ramstedt simonramstedt@gmail.com
Pal Jonathan Binas
A BSTRACT
Action and observation delays commonly occur in many Reinforcement Learning applications, such as remote control scenarios.
We study the anatomy of randomly delayed environments and show that partially resampling trajectory fragments in hindsight allows for off-policy multi-step value estimation.
We apply this principle to derive Delay-Correcting Actor-Critic ( DCAC ), an algorithm based on Soft ActorCritic with signiﬁcantly better performance in environments with delays.
This is shown theoretically and also demonstrated practically on a delay-augmented version of the MuJoCo continuous control benchmark
I NTRODUCTION
[Figure] [Figure]
This article is concerned with the Reinforcement Learning ( RL ) scenario depicted in Figure 1, which is commonly encountered in real-world applications (Mahmood et al., 2018; Fuchs et al., 2020; Hwangbo et al., 2017).
Oftentimes, actions generated by the agent are not immediately applied in the environment, and observations do not immediately reach the agent.
Such environments have mainly been studied under the unrealistic assumption of constant delays (Nilsson et al., 1998;
Ge et al., 2013; Mahmood et al., 2018).
Here, prior work has proposed different planning algorithms which naively try to undelay the environment by simulating future observations (Walsh et al., 2008; Schuitema et al., 2010; Firoiu et al., 2018).
We propose an off-policy, planning-free approach that enables lowbias and low-variance multi-step value estimation in environments with random delays.
First, we study the anatomy of such environments in order to exploit their structure, deﬁning Random-Delay
Markov Decision Processes ( RDMDP ).
Then, we show how to transform trajectory fragments collected under one policy into trajectory fragments distributed according to another policy.
We demonstrate this principle by deriving a novel off-policy algorithm ( DCAC ) based on Soft Actor-Critic ( SAC ), and exhibiting greatly improved performance in delayed environments.
Along with this work we release our code, including a wrapper that conveniently augments any
OpenAI gym environment with custom delays.
D ELAYED E NVIRONMENTS
We frame the general setting of real-world Reinforcement Learning in terms of an agent , random observation delays , random action delays , and an undelayed environment .
At the beginning of each time-step, the agent starts computing a new action from the most recent available delayed observation.
Meanwhile, a new observation is sent and the most recent delayed action is applied in the undelayed environment.
Real-valued delays are rounded up to the next integer time-step.
For a given delayed observation s t , the observation delay ω t refers to the number of elapsed time-steps from when s t ﬁnishes being captured to when it starts being used to compute a new action.
The action delay α t refers to the number of elapsed time-steps from when the last action inﬂuencing s t starts being computed to one time-step before s t ﬁnishes being captured.
We further refer to ω t + α t as the total delay of s t .
As a motivating illustration of real-world delayed setting, we have collected a dataset of communication delays between a decisionmaking computer and a ﬂying robot over WiFi, summarized in
Figure 2.
In the presence of such delays, the naive approach is to simply use the last received observation.
In this case, any delay longer than one time-step violates the Markov assumption, since the last sent action becomes an unobserved part of the current state of the environment.
To overcome this issue, we deﬁne a Markov Decision
Process that takes into account the communication dynamics.
2. R ANDOM D ELAY M ARKOV D ECISION P ROCESSES
To ensure the Markov property in delayed settings, it is necessary to augment the delayed observation with at least the last K sent actions.
K is the combined maximum possible observation and action delay.
This is required as the oldest actions along with the delayed observation describe the current state of the undelayed environment, whereas the most recent actions are yet to be applied (see
Appendix C).
Using this augmentation sufﬁces to ensure that the Markov property is met in certain delayed environments.
On the other hand, it is possible to do much better when the delays themselves are also part of the state-space.
First, this allows us to model self-correlated delays, e.g. discarding outdated actions and observations (see Appendix A.1).
Second, this provides useful information to the model about how old an observation is and what actions have been applied next.
Third, knowledge over the total delay allows for efﬁcient credit assignment and off-policy partial trajectory resampling, as we show in this work.
Deﬁnition 1.
A Random Delay Markov Decision Process RDMDP ( E, p ω , p α ) =
( X, A, ˜ µ, ˜ p ) augments a Markov Decision Process E =
( S, A, µ, p ) with:
(1) state-space X =
S ×
A K × N 2 , (2) action-space A , (3) initial state distribution ˜ µ ( x 0 )
=
˜ µ ( s, u, ω, α ) =
µ ( s ) δ ( u − c u ) δ ( ω − c ω ) δ ( α − c α ) , (4) transition distribution ˜
[Equation]
where s ∈ S is the delayed observation,
u ∈
A K is a buffer of the last K sent actions, ω ∈ N is the observation delay , and α ∈ N is the action delay as deﬁned above.
To avoid conﬂicting with the subscript notation, we index the action buffers’ elements using square brackets.
Here, u [1] is the most recent and u [ K ] is the oldest action in the buffer.
We denote slices by u [ i : j ] =
( u [ i ] , . . .
, u [ j ] ) and u
[ i : − j ] = ( u [ i ] , . . .
, u [ K − j ] ) .
We slightly override this notation and additionally deﬁne u [0]
= a .
The constants c u ∈
A K and c ω , c α ∈ N initialize u, ω, α , since δ is the Dirac delta distribution.
The transition distribution itself is composed of three parts: (1) The observation delay distribution p ω modelling the evolution of observation delays.
Note that this density function must represent a discrete distribution (i.e. be a weighted sum of Dirac delta distributions).
Furthermore, this process will repeat observations if there are no new ones available.
This means that the observation delay can maximally grow by one from one time-step to the next.
The transition distribution for the action buffer p u ( u (cid:48) | u, a ) =
δ ( u (cid:48) − ( a, u [1 : − 1] )) .
The distribution f ∆ describing the evolution of observations, rewards and action delays (Deﬁnition 2).
Deﬁnition 2.
For each change in observation delays ( ∆= ω − ω (cid:48) )
we deﬁne a variable step update distribution f ∆ as
[Equation]
The base case of the recursion is f − 1 ( s (cid:48) , α (cid:48) , r (cid:48) | s, u, ω, α, a ) =
δ ( s (cid:48) − s ) δ ( α (cid:48) − α ) δ
( r (cid:48) ) .
Here, p α is the action delay distribution which, similar to p ω , must be discrete.
The transition distribution of the underlying, undelayed MDP is p .
The r (cid:48) − r ∗ term accumulates intermediate rewards in case observations are skipped or repeated (see Appendix A.4).
Since the observation delay cannot increment by more than one, f − 1 is used when ω is increasing, whereas f 0 is used when there is no change in observation delay.
A simple special case of the RDMDP is the constant observation and action delay case with p ω ( ω (cid:48) | ω )
=
δ ( ω (cid:48) − c ω ) and p α ( α (cid:48) | α )
=
δ ( α (cid:48) − c α ) .
Here, the RDMDP reduces to a Constantly Delayed Markov
Decision Process, described by Walsh et al.
In this case, the action and observation delays α, ω can be removed from the state-space as they don’t carry information.
Examples of RDMDP dynamics are visualized in Figure 3 (see also Appendix C).
R EINFORCEMENT L EARNING IN D ELAYED E NVIRONMENTS
Delayed environments as described in Section 2 are speciﬁc types of MDP , with an augmented statespace and delayed dynamics.
Therefore, using this augmented state-space, traditional algorithms such as Soft Actor-Critic ( SAC ) (Haarnoja et al., 2018a)(Haarnoja et al., 2018b) will always work in randomly delayed settings.
However, their performance will still deteriorate because of the more difﬁcult credit assignment caused by delayed observations and rewards, on top of the exploration and generalization burdens of delayed environments.
We now analyze how to compensate for the credit assignment difﬁculty by leveraging our knowledge about the delays’ dynamics.
One solution is to perform on-policy multi-step rollouts on sub-trajectories that are longer than the considered delays.
On the other hand, on-policy algorithms are known to be sample-inefﬁcient and therefore are not commonly used in real-world applications, where data collection is costly.
This motivates the development of off-policy algorithms able to reuse old samples, such as SAC .
Intuitively, in delayed environments, one should take advantage of the fact that actions only inﬂuence observations and rewards after a number of time-steps relative to the beginning of their computation (the total delay ω + α ).
Since the delay information is part of the state-space, it can be leveraged to track the action inﬂuence through time.
However, applying conventional off-policy algorithms in delayed settings leads to the following issue: the trajectories used to perform the aforementioned multi-step backups have been sampled under an outdated policy, and therefore contain outdated action buffers.
In this section, we propose a method to tackle this issue by performing partial trajectory resampling .
We make use of the fact that the delayed dynamics are known to simulate the effect they would have had under the current policy, effectively transforming off-policy sub-trajectories into on-policy sub-trajectories.
This enables us to derive a family of efﬁcient off-policy algorithms for randomly delayed settings.
3. P ARTIAL T RAJECTORY R ESAMPLING IN D ELAYED E NVIRONMENTS
One important observation implied by Figure 3 is that, given the delayed dynamics of RDMDP s , some actions contained in the action buffer of an off-policy state did not inﬂuence the subsequent delayed observations and rewards for a number of time-steps.
Therefore, if an off-policy sub-trajectory is short enough, it is possible to recursively resample its action buffers with no inﬂuence on the return.
We propose the following transformation of off-policy sub-trajectories:
Deﬁnition 3.
The partial trajectory resampling operator recursively updates action buffers as follows
[Equation]
with trivial base case
[Equation]
This operator recursively resamples the most recent actions of each action buffer in an input subtrajectory τ n , according to a new policy π .
Everything else stays unchanged.
A visual example is provided in Figure 4 with n = 2 and an action buffer of two actions.
When resampled actions are delayed and would not affect the environment, they do not "invalidate" the sub-trajectory.
The resampled trajectories can then be considered on-policy.
Theorem 1.
The partial trajectory resampling operator σ πn (Def. 3) transforms off-policy trajectories into on-policy trajectories
[Equation]
on the condition that none of the delayed observations depend on any of the resampled actions, i.e.
[Equation]
where t indexes the trajectory τ ∗ n = ( s ∗ 1 , u ∗ 1 , ω ∗ 1 , α ∗ 1 , r ∗ 1 , . .
, s ∗ n , u ∗ n , ω ∗ n , α ∗ n , r ∗ n ) from 1 to n .
The condition in Equation 4 can be understood visually with the help of Figure 3.
In the constant delay example it is fulﬁlled until the third time-step.
After that, the observations would have been inﬂuenced by the resampled actions (starting with a 0 ).
3. M ULTI - STEP O FF -P OLICY V ALUE E STIMATION IN D ELAYED E NVIRONMENTS
We have shown in Section 3.1 how it is possible to transform off-policy sub-trajectories into on-policy sub-trajectories in the presence of random delays.
From this, we can derive a family of efﬁcient off-policy algorithms for the randomly delayed setting.
For this matter, we make use of the classic on-policy Monte-Carlo n -step value estimator:
Deﬁnition 4.
The n-step state-value estimator is deﬁned as
[Equation]
where ˆ v 0 is a state-value function approximator (e.g. a neural network).
Indeed, in γ -discounted RL , performing on-policy n-step rollouts to estimate the value function reduces the bias introduced by the function approximator by a factor of γ n :
Lemma 1.
The n-step value estimator has the following bias:
[Equation]
A simple corollary of Lemma 1 is that the on-policy n-step value estimator is unbiased when the function approximator ˆ v 0 is unbiased.
On the other hand, Theorem 1 provides a recipe for transforming sub-trajectories collected under old policies into actual on-policy sub-trajectories.
From a given state in an off-policy trajectory, this is done by applying σ πn to all the subsequent transitions until we meet a total delay ( ω i + α i ) that is shorter than the length of the formed sub-trajectory.
Consequently, the transformed sub-trajectory can be fed to the on-policy n-step value estimator, where n is the length of this sub-trajectory.
This does not only provide a better value estimate than usual 1-step off-policy estimators according to Lemma 1, but it maximally compensates for the multi-step credit assignment difﬁculty introduced by random delays.
Indeed, the length of the transformed sub-trajectory is then exactly the number of time-steps it took the ﬁrst action of the sub-trajectory to have an inﬂuence on subsequent delayed observations, minus one time-step.
As opposed to other unbiased n-step off-policy methods, such as importance sampling and Retrace (Munos et al., 2016), this method doesn’t suffer from variance explosion.
This is because the presence of delays allows us to transform off-policy sub-trajectories into on-policy sub-trajectories, so that old samples don’t need to be weighted by the policy ratio.
Although we use a multi-step state-value estimator, the same principles can be applied to action-value estimation as well.
In fact, the trajectory transformation described in Deﬁnition 3 enables efﬁcient off-policy n-step value estimation in any value-based algorithm that would otherwise perform 1-step action-value backups, such as DQN , DDPG or SAC .
In the next section, we illustrate this using SAC .
Figure 5 summarizes the whole procedure in a simple 1D-world example.
The maximum possible delay is K = 3 here, and the agent can only go ‘left’ or ‘right’.
An initial augmented state xis sampled from the replay memory, along with the 3 subsequent augmented states and rewards.
The condition of Theorem 1 is satisﬁed for n ≤ 2 .
It follows that τ n = τ 2 = ( x 1 , x 2 ) .
This offpolicy trajectory fragment is partially resampled, which yields the corresponding on-policy trajectory fragment τ ∗ n = τ ∗ 2 .
This on-policy trajectory fragment can then be used to compute an unbiased n-step value estimate of the initial state
x 0 = x ∗ 0 .
D ELAY -C ORRECTING A CTOR -C RITIC
We have seen in Section 3 how it is possible, in the delayed setting, to collect off-policy trajectories and still use on-policy multi-step estimators in an unbiased way, which allows us to compensate for the more difﬁcult credit assignment introduced by the presence of random delays.
We now apply this method to derive Delay-Correcting Actor-Critic ( DCAC ), an improved version of Soft Actor-Critic (Haarnoja et al., 2018a;b) for real-time randomly delayed settings.
4. V ALUE A PPROXIMATION
Like SAC , DCAC makes use of the entropy-augmented soft value function (Haarnoja et al., 2018a):
Lemma 2.
In a RDMDP ( E, p ω , p α ) the soft value function is:
[Equation] [Equation]
It can be estimated by augmenting the reward function in Deﬁnition 4 with an entropy reward:
Deﬁnition 5.
The delayed on-policy n-step soft state-value estimator, i.e. the n-step state-value estimator with entropy augmented rewards under the current policy π , is
[Equation]
where ˆ v soft 0 is a state-value function approximator (e.g. a neural network).
Given the off-policy trajectory transformation proposed in Section 3, Deﬁnition 5 directly gives
DCAC ’s value target.
To recap, we sample an initial state x 0 ( = x ∗ 0 ) and a subsequent trajectory τ n (= x 1 , r 1 , . . .
x n , r n ) from a replay memory.
The sampling procedure ensures that n is the greatest length so that the sampled trajectory τ n does not contain any total delay
ω i + β i < i .
This trajectory was collected under an old policy µ , but we need a trajectory compatible with the current policy π to use ˆ v soft n in an unbiased way.
Therefore, we feed τ n to the partial trajectory resampling operator deﬁned in Deﬁnition 3.
This produces an equivalent on-policy sub-trajectory τ ∗ n with respect to the current policy π according to Theorem 1, while maximally taking advantage of the bias reduction described by Lemma 1.
This partially resampled on-policy sub-trajectory is fed as input to ˆ v soft n ( x 0 ; τ ∗ n ) , which yields the target used in DCAC ’s soft state-value loss:
Deﬁnition 6.
The DCAC critic loss is
[Equation]
where x 0 , τ n are a start state and following trajectory, sampled from the replay memory, and satisfying the condition of Theorem 1.
4. P OLICY I MPROVEMENT
In addition to using the on-policy n-step value estimator as target for our parametric value estimator, we can also use it for policy improvement.
As in SAC we use the reparameterization trick (Kingma &
Welling, 2013) to obtain the policy gradient from the value estimator.
However, since we use our trajectory transformation and a multi-step value estimator, this involves backpropagation through time in the action buffer.
Deﬁnition 7.
The DCAC actor loss is
[Equation]
where x 0 , τ n are a start state and following trajectory, sampled from the replay memory, and satisfying the condition of Theorem 1.
Proposition 1.
The DCAC actor loss is a less biased version of the SAC actor loss with
[Equation]
assuming both are using similarly biased parametric value estimators to compute the loss,
i.e.
[Equation]
E XPERIMENTAL RESULTS
To evaluate our approach and make future work in this direction easy for the RL community, we release as open-source, along with our code, a Gym wrapper that introduces custom multi-step delays in any classical turn-based Gym environment.
In particular, this enables us to introduce random delays to the Gym MuJoCo continuous control suite (Brockman et al., 2016; Todorov et al.), which is otherwise turn-based.
Compared algorithms.
A naive version of SAC would only use the unaugmented delayed observations, which violates the Markov assumption in delayed settings as previously pointed out.
Consequently, naive SAC exhibits near-random results in delayed environments.
A few such experiments are provided in the Appendix for illustration (Figure 9).
In order to make a fair comparison, all other experiments compare DCAC against SAC in the same
RDMDP setting, i.e. all algorithms use the augmented observation space deﬁned in Section 2.1.
Since
SAC is the algorithm we chose to improve for delayed scenarios, comparing DCAC against it in the same setting provides a like-for-like comparison.
We also found it interesting to compare against RTAC (Ramstedt & Pal, 2019).
Indeed, DCAC reduces to this algorithm in the special case where observation transmission is instantaneous ( ω =0) and action computation and transmission constantly takes one time-step ( α =1).
Whereas DCAC performs variable-length state-value backups with partial trajectory resampling as explained in Section 4 , RTAC performs 1-step state-value backups, and SAC performs the usual 1-step action-value backup described in its second version (Haarnoja et al., 2018b).
All hyperparameters and implementation details are provided in Section B of the Appendix.
For each experiment, we perform six runs with different seeds, and shade the 90% conﬁdence intervals.
Constant delays.
Our ﬁrst batch of experiments features simple, constantly delayed scenarios.
Figure 6 displays the results of the most difﬁcult of these experiments (i.e. where the delays are longest), while the others are provided in Section D.2 of the Appendix.
The advantage of using DCAC is obvious in the presence of long constant delays.
Note that DCAC reduces to the RTAC (Ramstedt & Pal, 2019) algorithm when ω = 0 and α = 1 and behaves as an evolved form of RTAC in the presence of longer constant delays.
Real-world random delays.
Our second batch of experiments features random delays of different magnitudes.
The experiment we chose to present in Figure 7 is motivated by the fact that our approach is designed for real-world applications.
Importantly, it provides an example how to implement DCAC in practice (see Appendix A and B for more details).
We sample the communication delays for actions and observations from our real-world WiFi dataset, presented in Figure 2.
When action or observation communications supersede previous communications, only the most recently produced information is kept.
In other words, when an action is received in the undelayed environment, its age is compared to the action that is currently being applied.
Then, the one that the agent most recently started to produce is applied.
Similarly, when the agent receives a new observation, it only keeps the one that was most recently captured in the undelayed environment (see the right-hand side of
Figure 3 for a visual example).
We discretize the communication delays by using a time-step of 20ms.
Importantly, note that Figure 2 has been cropped to 60ms, but the actual dataset contains outliers that can go as far as 1s.
However, long delays (longer than 80ms in our example) are almost always superseded and discarded.
Therefore, when such information is received, we clip the corresponding delay with no visible impact in performance: in practice, the maximum acceptable delays are design choices, and can be guided by existing probabilistic timing methods (Santinelli et al., 2017).
R ELATED WORK
We trace our line of research back to Katsikopoulos & Engelbrecht (2003), who provided the ﬁrst discussion about Delayed Markov Decision Processes.
In particular, they were interested in asynchronous rewards, which provides interesting insights in relation to Appendix A.4.
Walsh et al.
(2008) later re-introduced the notion of “Constantly Delayed Markov Decision Process”.
While recent advances in deep learning enable implementations of what the authors call an “augmented approach”, this was considered intractable at the time because the size of the action buffer grows with the considered delay length.
Instead, they studied the case where observations are retrieved with a constant delay and developed a model-based algorithm to predict the current state of the environment.
Similarly, Schuitema et al.
(2010) developed “memory-less” approaches based on SARSA and vanilla
Q-learning, taking advantage of prior knowledge about the duration of a constant control delay.
Hester & Stone (2013) adopted the action buffer-augmented approach to handle random delays, and relied on a decision-tree algorithm to perform credit assignment implicitly.
By comparison, our approach relies on delay measurements to perform credit assignment explicitly.
More recently, Firoiu et al.
(2018) introduced constant action delays to a video game to train agents whose reaction time compares to humans.
Similar to previous work, the authors used a state-predictive model, but based on a recurrent neural network architecture.
Ramstedt & Pal (2019) formalized the framework of
Real-Time Reinforcement Learning ( RTRL ) that we generalize here to all forms of real-time delays.
Initially designed to cope with the fact that inference is not instantaneous in real-world control, the
RTRL setting is equivalent to a constantly delayed MDP with α = 1 and ω = 0 .
Finally, Xiao et al.
(2020) adopted an alternative approach by considering the inﬂuence of the action selection time when action selection is performed within the duration of a larger time-step.
However, their framework only allows delays smaller than one time-step, whereas large time-steps are not compatible with high-frequency control.
C ONCLUSION AND FUTURE WORK
We proposed a deep off-policy and planning-free approach that explicitly tackles the credit assignment difﬁculty introduced by real-world random delays.
This is done by taking advantage of delay measurements in order to generate actual on-policy sub-trajectories from off-policy samples.
In addition, we provide a theoretical analysis that can easily be reused to derive a wide family of algorithms such as DCAC , whereas previous work mostly dealt with ﬁnding approximate ways of modelling the state-space in constantly delayed environments.
The action buffer is fundamentally required to deﬁne a Markovian state-space for RDMDP s , but it is of course possible to observe this action buffer approximately, e.g. by compressing it in the hidden state of an RNN , which is complementary to our work.
We have designed our approach with real-world applications in mind, and it is easily scalable to a wide variety of scenarios. For practical implementation, see Section 5 and Sections A and B of the Appendix.
See also rtgym, a small python helper that we use in future work to easily implement delayed environments in the real world.
To the best of our knowledge, DCAC is the ﬁrst deep actor-critic approach to exhibit such strong performance on both randomly and constantly delayed settings, as it makes use of the partially known dynamics of the environment to compensate for difﬁcult credit assignment.
We believe that our model can be further improved by making use of the fact that our critic estimates the state-value instead of the action-value function.
Indeed, in this setting, Ramstedt & Pal (2019) showed that it is possible to simplify the model by merging the actor and the critic networks using the PopArt output normalization (van Hasselt et al., 2016), which we did not try yet and leave for future work.
Our approach handles and adapts to arbitrary choices of time-step duration, although in practice time-steps smaller than the upper bound of the inference time will require a few tricks.
We believe that this approach is close to time-step agnostic RL and will investigate this direction in future work.
A CKNOWLEDGMENTS
We thank Pierre-Yves Lajoie, Yoshua Bengio and our anonymous reviewers for their constructive feedback, which greatly helped us improve the article.
We also thank ElementAI and Compute
Canada for providing the computational resources we used to run our experiments.
A P RACTICAL CONSIDERATIONS AND SCALABILITY A. S ELF - CORRELATED DELAYS
The separation between ω and α allows auto-correlated conditional distributions on both delays.
This is necessary to allow superseded actions and observations to be discarded.
In RDMDP s , the agent keeps the delayed observation that was most recently captured in the undelayed environment.
Ideally, it is also ensured by the undelayed environment that the applied action is the action that most recently started being computed by the agent.
In practice, this can be ensured by augmenting the actions with timestamps corresponding to the beginning of their computation, and observations with timestamps corresponding to the end of their capture.
Thus, the undelayed environment and the agent can keep track of the most recent received timestamp and discard outdated incoming information.
To measure the delays in practice, one possibility is to make use of the aforementioned timestamps. In addition to the augmentations described in A.1, one can augment each observation sent by the undelayed environment with the timestamp of the action that was applied before the end of observation capture.
When the agent receives an observation, this observation then contains two timestamps: one that directly corresponds to an action in the buffer (agent’s clock), and one that corresponds to when the observation ﬁnished being captured (undelayed environment’s clock). The identiﬁed action in the buffer directly gives the total delay.
If the agent and the undelayed environment have e.g. synchronized clocks, the current timestamp minus the timestamp corresponding to observation capture gives the observation delay (and thus we can deduce the action delay). A. S CALABILITY OF THE ACTION BUFFER
As seen in our WiFi experiment, the maximum delays are design choices in practice. The actual maximum delays can be prohibitively long (e.g. inﬁnite when packets are lost) and would require a long action buffer to be handled in the worst-case scenario.
However, in random delays scenarios, long delays are likely to be superseded by shorter delays. Therefore, observations reaching the agent with a total delay that exceeds the chosen K value should simply be discarded, and a procedure implemented to handle the unlikely edge-case where more than K such observations are received in a row.
Also note that, although we used a simple action buffer in this work, more clever representations are possible in the presence of long delays, e.g. run-length encoding.
A. D ELAYED REWARDS
We have implicitly made a choice when deﬁning the rewards for RDMDP s .
Indeed, keep in mind that observations can be dropped (superseded) at the level of the agent.
In such cases, we chose to accumulate the rewards corresponding to the lost transitions.
When an observation gets repeated because no new observation is available, the corresponding reward is 0, and when a new observation arrives, the corresponding reward contains the sum of intermediate rewards in lost transitions.
In practice, this is ensured for example by making the assumption that the remote robot (i.e. the undelayed environment) can observe its own instantaneous reward.
This allows the robot to compute its cumulative reward and send it to the agent along with the observation.
The agent can then compute the difference between the last cumulative reward it received from the remote robot and the new one for each incoming observation (NB: outdated observations are discarded so the agent only sees cumulative rewards with time-increasing timestamps).
Alternatively, the practitioner can choose to repeat the delayed rewards along with the repeated delayed observations at the level of the agent (this is what we use to do in earlier versions of the paper).
When a trick similar to the aforementioned cannot be implemented, this can be done instead, with no impact on our analysis.
However, the reward signal will inherently have a higher variance.
A. L ONG OBSERVATION CAPTURE
In practice, it is often the case that observation capture is not instantaneous.
In such situation, one should increase the size of the action buffer so that it always includes the actions for which it is unclear whether they have inﬂuenced the observation yet or not.
Indeed, when observation capture is not instantaneous it is not possible to know which undelayed state(s) it describes.
The length of the multi-step backup performed by DCAC doesn’t need to be adapted, because it only cares about the ﬁrst action that is known to not have inﬂuenced the delayed observation.
Equivalently, if observations are formed of several combined parts that were captured at different times, the action buffer must be long enough to always include the ﬁrst action that has not inﬂuenced the oldest sub-observation yet (i.e. be as long as the maximum possible combined total delay).
B I MPLEMENTATION D ETAILS B. M ORE INFORMATION AS INPUT TO THE MODEL
The action delay α identiﬁes the action that was applied during the previous time-step.
It is needed to deﬁne RDMDP s and thus is used by DCAC .
However, in practice we can include another piece of information on top of α : the delay of the action that is going to be applied in the undelayed environment when the captured observation is sent.
We use this additional information as input of the model for all tested algorithms.
B. M ODEL A RCHITECTURE
The model we use in all our experiments is composed of two separate multi-layer perceptrons (MLPs): a critic network, and an actor network.
Both MLPs are built with the same simple architecture of two hidden layers, 256 units each.
The critic outputs a single value, whereas the actor outputs an action distribution with the dimension of the action-space, from which actions are sampled with the reparameterization trick.
This architecture is compatible with the second version of SAC described in Haarnoja et al.
(2018b).
The only difference from the DCAC model is that the SAC critic tracks q ( x ) , and not v ( x ) .
Indeed, differently from usual actor-critic algorithms, the output of DCAC ’s critic approximates the state-value v ( x ) (instead of the action-value q ( x ) ), as it is sufﬁcient to optimize the actor loss described in Deﬁnition 7.
Weights and biases are initialized with the default Pytorch initializer.
Both the actor and the critic are optimized by gradient descent with the Adam optimizer, on losses L DCAC ( π ) (Equation 10) and L DCAC ( v ) (Equation 9), respectively.
Classically, we use twin critic networks (Van Hasselt et al., 2015; Fujimoto et al., 2018) with target weight tracking (Mnih et al., 2015) to stabilize training.
B. H YPERPARAMETERS
Other than our neural network architecture, our implementations of SAC , RTAC and DCAC all share the following hyperparameters:
[Table]
NB: the target weights are updated according to the following running average: θ ← τθ + (1 − τ ) θ
C V ISUAL EXAMPLES
D A DDITIONAL E XPERIMENTS D. I MPORTANCE OF THE AUGMENTED OBSERVATION SPACE
D. C ONSTANT DELAYS
Figure uniformly sampled delays): This experiment is perhaps even more difﬁcult than the WiFi experiment featured in the main paper, because it gives equal probability to all possible delays in the speciﬁed ranges (but delays are smaller here which makes it easier for RTAC , because these delays are closer to 1). All tested approaches fail on randomly delayed Ant. For other tasks, the advantage of DCAC is very clear over SAC .
E D EFINITIONS
Deﬁnition 8.
The n -step state-reward distribution for an environment E =
( S, A, µ, p ) and a policy π is deﬁned as
[Equation]
with the base case p π 0 ( s ) = 1 and the ﬁrst iterate p
π 1 ( s (cid:48) , r
[Equation]
| s )
= (cid:82) A p ( s
[Equation]
, r
[Equation]
| s,
[Equation]
) π ( a | s [Equation] da .
Deﬁnition 9.
A 1-step action-value estimator is deﬁned as
[Equation]
Part of this estimator is usually another parametric estimator ˆ q 0 (e.g. a neural network trained with stochastic gradient descent).
F O THER M ATHEMATICAL R ESULTS F. L EMMA ON S TEADY -S TATE V ALUE E STIMATION B IAS
Lemma 3.
The expected bias of the n-step value estimator under the steady-state distribution (if it exists) is
[Equation]
Proof.
We remind ourselves that the steady state distribution observes
[Equation]
According to Lemma 1 we then have
[Equation]
F. L EMMA ON A D IRAC DELTA PRODUCT DISTRIBUTION
Lemma 4.
For p ( u, v ) =
δ ( u − c ) q
( u, v ) if q ( u, v ) < ∞ for u =
c then p ( u, v ) =
δ ( u − c ) q
( c, v ) .
Proof.
If u = c then p ( u, v ) =
δ ( u − c ) q
( c, v ) , otherwise p ( u, v ) =
= δ ( u − c ) q
( c, v )
F. L EMMA ON F
Lemma 5.
The dynamics described by f depend neither on the input action nor on a range of actions in the action buffer:
[Equation]
with x 0 = s 0 , u 0 , ω 0 , α 0 and x ∗
=
s ∗ 0 , u ∗ 0 , ω ∗ 0 , α ∗ 0 , given that s 0 , ω 0 , α
= s ∗ 0 , ω ∗ 0 , α ∗ 0 and given u 0 [ ω ∗ 0 − δ + α ∗ 1 ] =
u ∗
[ ω ∗ 0 − δ + α ∗ 1 ] for all δ ∈ { ∆ , ∆ − 1 , . . .
Proof.
We prove by induction.
The base case ( ω ∗ 0 − ω ∗ 1 = − 1 ) is trivial since it does not depend on the inputs that differ.
For the induction step we have
[Equation]
Because of our condition on u 0 and u ∗ 0 and the fact that ω 0 = ω ∗ 0 this is equal to
[Equation]
We can now use the induction hypothesis since the conditions on s 0 , u 0 , ω 0 , α 0 are still met when ∆ ← ∆ − 1 .
[Equation]
F. L EMMA ON PARTIAL RESAMPLING
Lemma 6.
Partially resampling trajectories collected under a policy µ according to σ πn transforms them into trajectories distributed according to π .
[Equation]
with x 0 = s 0 , u 0 , ω 0 , α 0 and x ∗
=
s ∗ 0 , u ∗ 0 , ω ∗ 0 , α ∗ 0 , on the condition that s 0 , ω 0 , α
= s ∗ 0 , ω ∗ 0 , α ∗ 0 and on the condition that the actions in the initial action buffers u 0 and u ∗ 0 that are applied in the following trajectory are the same, i.e. u 0 [ k : end ]
= u {+∗ 
+}
[ k : end ] with k = min
i ( ω ∗ i +
+ α ∗
i +1 − i ) for i ∈ { 0 , n − 1 } and for the trajectory τ ∗ n = ( s ∗ 1 , u ∗ 1 , ω ∗ 1 , α ∗ 1 , . . .
, s ∗ n , u ∗ n , ω ∗ n , α ∗ n ) .
Proof.
We start with the induction base for n = 0 .
The theorem is trivial in this case since we have 0-length trajectories () and p µ 0 (() | x 0 ) =
σ π 0 (() | x ∗ 0 ; ())
=
p π 0 (() | x ∗ 0 ) = 1 .
For the induction step we start with the left hand side of the lemma’s main equation.
[Equation]
with
[Equation]
Plugging that and solving the integral over u 1 yields
[Equation]
Rolling out σ πn by one step and integrating out s 1 , ω 1 , α 1 , r 1 yields
[Equation]
Reordering terms and substituting s 0 , ω 0 , α 
 = s
[Equation]
[Equation] 0 , α [Equation] 0 yields
[Equation]
We can substitute the f term according to Lemma 5 since the condition between x 0 and x ∗ 0 is met.
More precisely the condition on u 0 and u ∗ 0 is met because k ≤ ω ∗ 0 − ∆ + α ∗ 1 = ω ∗ 1 + α ∗ 1 .
After the substitution we have
[Equation]
We can substitute the induction hypothesis in the following form.
[Equation]
on the condition that [Equation]
[Equation] i [Equation]
i [Equation] [Equation] 2 } for the trajectory τ ∗ n − 1 =
( s ∗ 2 , u ∗ 2 , ω ∗ 2 , α ∗ 2 , . . .
, s ∗ n , u ∗ n , ω ∗ n , α ∗ n ) .
To check that this condition is met we observe that u 1 = ( a µ 0 , u 0 [1 : − 1]) and substitute u ∗ 1 = ( a π 0 , u ∗ 0 [1 : − 1]) (made possible by
Lemma 4) which means that[-u 
-]
[Equation] end ]
[Equation]
k = min i ( ω ∗ i +2 + α ∗ i +2 − i ) for i ∈ { 0 , n − 2 }
Substituting the induction hypothesis yields
[Equation]
(26) which is
[Equation]
G P ROOFS OF THE RESULTS FROM THE MAIN PAPER
A
Theorem 1.
The partial trajectory resampling operator σ πn (Def. 3) transforms off-policy trajectories into on-policy trajectories
[Equation]
on the condition that none of the delayed observations depend on any of the resampled actions, i.e.
[Equation]
where t indexes the trajectory τ ∗ n = ( s ∗ 1 , u ∗ 1 , ω ∗ 1 , α ∗ 1 , r ∗ 1 , . .
, s ∗ n , u ∗ n , ω ∗ n , α ∗ n , r ∗ n ) from 1 to n .
Proof.
The theorem is a special case of Lemma 6 with x 0 = x ∗ 0 .
This allows us to simplify the condition in the lemma as we show next.
Since u 0 = u ∗
we can allow all k ≥ 1 which is the minimum allowed index for u .
Therefore we must ensure 1 ≤ min i ( ω ∗ i {++
+} + α ∗ i +1 − i ) .
Since the min must be larger than 1 then all arguments must be larger than 1 which means this is equivalent to
[Equation]
This can be transformed into
[Equation]
Lemma 1.
The n-step value estimator has the following bias:
[Equation]
Proof.
[Equation]
[Equation]
B
Lemma 2.
In a RDMDP ( E, p ω , p α ) the soft value function is:
[Equation]
Proof.
The soft value function for an environment ( X, A, ¯ µ, ¯ p ) is deﬁned as
[Equation]
where
[Equation]
If ( X, A, ¯ µ, ¯ p ) =
RDMDP ( E, p ω , p α ) =
( X, A, ˜ µ, ˜ p ) with E =
( S, A, µ, p ) this is q
[Equation]
37) and
[Equation]
Proposition 1.
The DCAC actor loss is a less biased version of the SAC actor loss with
[Equation]
assuming both are using similarly biased parametric value estimators to compute the loss,
i.e.
[Equation]
Proof.
Note that for simplicity, we also assume that the states in the replay memory are distributed according to the steady-state distribution, i [Equation] e. D ∼ p π ss .
This assumption could be avoided by making more complicated assumptions about the biases of the state-value and action-value estimators.
We now start with the bias of the DCAC loss with respect to an unbiased SAC loss using the true action-value function,
[Equation]
where
[Equation]
| Theorem
[Equation]
and
[Equation]
Substituting these we have
[Equation]
| Lemma
[Equation]
[Equation]
| using D ∼ p π ss and Lemma
[Equation] [Equation] Equation [Equation]