Paper: Tree Search-Based Policy Optimization under Stochastic Execution Delay (ICLR-2024)

Reviewer comment (Novelty):
It is not clear that this is a new MDP variant, or it is a particular version of the one with random delays considered in the Bouteiller article (cited in the paper) - with no observation delays.

Author response:
**W1: What is the difference from the formulation of (Bouteiller et al., 2020)?** 
In fact, there is a crucial difference between our work and (Bouteiller et al., 2020). The formulation in the latter included an augmentation of the state space with the action queue and delay values: X=S×A^K×N^2. This makes the decision process Markovian and therefore all standard results on non-delayed RL hold naturally at the expense of an exponentially larger new (embedded) state space. In our SED-MDP, however, the state space remains unembedded. We therefore devise the theoretical machinery needed to be able to prove that non-Markov policies are as beneficial as history-dependent policies, as long as they can be non-stationary.

Reviewer comment (Theory):
Personally, I would find it more natural if the action delays were not limited to discrete values, and were executed after the delay. In many physical systems, the evolution of the system is not restricted to discrete steps anyway, even if the decisions are. This would avoid the drop and duplication of the actions.

Author response:
**Q1: What about continuous time delay and estimation of it without observing it?** We appreciate the useful real-world direction. We agree and added this direction to the summary as relevant future work.   We mentioned another approach when dealing with unobserved stochastic delays. Using a robust criterion with respect to multiple possible realizations of delays may lead to conservative policies, adapted to the delay complexity.

Reviewer comment (Theory):
We might not necessarily observe the realization of execution delay, unless we can reverse engineer it from a known transition model (when it is deterministic).

Author response:
**Q1: What about continuous time delay and estimation of it without observing it?** We appreciate the useful real-world direction. We agree and added this direction to the summary as relevant future work.   We mentioned another approach when dealing with unobserved stochastic delays. Using a robust criterion with respect to multiple possible realizations of delays may lead to conservative policies, adapted to the delay complexity.

Reviewer comment (Theory):
Since the ‘proper’ way to incorporate stochastic action delays in the MDP is not well established, I would like to see some (more) motivating discussion on the choice, and not only how it is handled.

Author response:
**Q2: Can you add motivating examples to having stochastic delay?** Sure. Perhaps the most prominent example of why delay can be stochastic is in real world systems that rely on transmission of data. Often, there is interference in transmission that can stem both from internal or external sources. Internal sources may be due to sensing HW that can be dependent on the temperature and pressure conditions. Whereas external sources can stem in the case of a robot or AV whose policy infers actions remotely (e.g., in the cloud). We added this motivation to the introduction.

Reviewer comment (Evaluation):
I may be misunderstanding the graphs, but it looks as if SD-EZ scores worse than the other algorithms in most of the games in the plots in Fig 3a and 3b. Also, there are no confidence intervals or significance testing of any kind.

Author response:
**W1 Mistake in Figure 3** 
Good catch, the legend in Figure 3 was indeed wrong; we fixed it in the revised version. Also, we added a table in the appendix, summarizing all the scores, including standard deviations for the four seeds.

Reviewer comment (Novelty):
This paper adopts the ED-MDP formulation of (Derman et al., 2021) that sidesteps state augmentation, and extend it to the random delay case. This extension appears to be a direct application of the previous formulation, authors are expected to explain the technical challenges compared to the constant delay formulation, and be clear about their technical contribution in terms of this formulation.

Author response:
**W1 How is the extension technically challenging compared to the existing ED-MDP formulation of Derman et al., (2021)?**
While both our work and that of Derman et al., (2021) tackle RL under delayed execution, we present new theoretical and algorithmic contributions. We understand the reviewer is asking regarding the theoretical part, so we answer it here. Unlike the fixed delay case of Derman et al., (2021), in an SED-MDP, the decision time of an executed action at $t$ is *random*. As a result, for a given policy, the process distribution can be different depending on the realization of the delay process. This is not the case of constant delay, as the SED-MDP process distribution stays the same once the policy is given. Yet, in Thm. 4.2, we show that *even* when the delay values are generated from a random process, for any history-dependent policy, we can achieve the same process distribution under a Markov policy. The technicalities of the proof require introducing the notion of *effective decision time* at time $t$, i.e., the effective time at which an action executed at $t$ has been drawn. Thus, instead of having $a_t\sim \pi_{t-m}$ for a constant execution delay $m$, we now have $a_t\sim\pi_{\tau_t}$, where $\tau_t}$ is the effective decision time – the time at which the action performed at time $t$ was previously decided. In some sense, the resulting process becomes a stopped process in the random delay case.

Reviewer comment (Novelty):
This paper mainly develops based on the ED-MDP formulation of (Derman et al., 2021), random delay formulation in (Bouteiller et al., 2020), and EfficientZero (Ye et al., 2021). The technical novelty in terms of the algorithm appears to be limited. Authors are expected to highlight the technical challenges and novelties.

Author response:
**W2 Technical challenges and novelties in Delayed EfficientZero** 
As opposed to the Delay-Correcting Actor-Critic of Bouteiller et al., (2020), we leverage a model-based approach, which is different from their solution: Bouteiller et al., (2020) work on an augmented state space, a continuous action space, and effectively small delay values compared to ours (observation and execution delays are smaller than 7 in any of their wifi delay sampler).  The technicalities in our work also differ from Delay-Correcting Actor Critic as we describe here. We address our approach by leveraging the popular model-based approach of Ye et al., (2021), but this requires a significant effort to extend to the delayed setting. When delays exist in tree search-based algorithms, one has to perform these searches successively while keeping parallelism efficiency.  We do so by implementing a parallelized MCTS search for predicted states, game trajectory manipulation, and matching stochastic delays after observation are obtained as described in Section 5 and in the added Appendix B.

Reviewer comment (Experiments):
While authors provide a thorough experimental study on 15 Atari benchmarks, they only consider small delays by setting $M = \{5, 15, 25\}$. It is desirable to see how the performance can be when delays are large.

Author response:
**W3 Longer delay values**
Thank you for your suggestion. We added experiments per your request on two Atari domains with larger delay values of {35,45}. The learning curves for Asterix and Hero can be found in the following link: 
https://ibb.co/q7npffz 
We also added them to Appendix C.3 in the revised version and the result explanations at the end of section 6.