Paper: DEER: A Delay-Resilient Framework for Reinforcement Learning with Variable Delays (ICLR-2024)

Reviewer comment (Evaluation):
It's unclear why breaking the end-to-end decision process into an encoding phase and a decision-making phase increases interpretability. I don't believe this claim is supported by the current experiments. The paper should clarify this contribution.

Author response:
Information states consist of the most recently observed states and their corresponding action sequences. The SACAS method utilizes information states as decision states to design value and policy functions. When there is significant delay, the dimensionality of the action part within the information states can be much larger than that of the states themselves. The SACAS method operates as a black-box process, disregarding the mentioned issue and completing training end-to-end, with the quality depending on the neural network's capacity and parameter settings. However, in the initial phase of our experiments, the delayed state information corresponding to delayed actions is actually known. Using this information, an encoder-decoder can map the information states into a latent space. This process, unlike SACAS's end-to-end random exploration of network parameters, is more directive, compelling the model to start from a good state representation. The improved interpretability is reflected here.

Reviewer comment (Novelty):
Since DEER essentially learns a dynamics model that predicts a sequence of states given an initial state and sequence of actions, my understanding is that DEER still suffers the same drawbacks as model-based approaches.

Author response:
Model-based algorithms typically involve constructing a dynamics model through real-time interactions with the environment, followed by planning using this model. Consequently, these algorithms are often vulnerable to model and iteration errors. DEER, however, leverages existing data from delay-free environments to learn an encoder-decoder for multi-step prediction. During the decision-making phase, DEER employs the encoder's representation of input states and action sequences, termed as context representation, rather than the final state from multi-step predictions. This representation extracts decision-relevant information from the most recent observed states and action sequences, thereby mitigating sensitivity to deviations from the true state caused by model and iteration errors.

Reviewer comment (Experiments):
DEER seems to have an unfair advantage over the baselines in Figures 4 and 5. In particular, DEER assumes access to an offline dataset, but the baselines do not. I believe DEER could in principle train its encoder in an online fashion -- if so, then an online variant of DEER would provide a fairer comparison across algorithms.

Author response:
Due to constraints in computational resources and time, our comparative experiments were exclusively performed on the Hopper and Walker2d tasks, each configured with a delay set at 4. The online version of DEER encompasses two primary modules: an encoder-decoder and a decision-making component. Initially, the encoder-decoder network is randomly initialized, and training data is gathered during real-time interactions within the environment. The encoder-decoder is updated every 300,000 steps. Regarding the decision-making aspect, the SAC framework is employed, wherein the replay buffer stores context representations derived from the encoder's information states. The entire training duration extends to 1 million steps. The comparative results are shown in the following table:

| Version of Deer | Online        | Offline       |
|-----------------|---------------|---------------|
| Hopper          | 751.75±623.66 | 2935.62±312   |
| Walker2d        | 141±159.8     | 2623.14±542.8 |

Reviewer comment (Presentation):
Figure 1 and 3 are essentially the same. I suggest replacing Figure 1 with Figure 3, and then include the “undelayed trajectory buffer” icon from figure 1.

Author response:
The aforementioned content will be updated in my newly uploaded PDF.

Reviewer comment (Writing):
In the abstract and conclusion, it is unclear what “tasks involving delays” and “delay issues” mean. These phrases are immediately clear from the introduction, but some researchers (such as myself) read the abstract and conclusion before reading the full paper. I suggest clarifying that you’re referring to when the learning agent receives its next observation after taking an action.

Author response:
The aforementioned content will be updated in my newly uploaded PDF.

Reviewer comment (Writing):
GRU not defined at its first occurrence.

Author response:
The aforementioned content will be updated in my newly uploaded PDF.

Reviewer comment (Presentation):
The “Offline assisted Online RL” section would be more readable if you used separate paragraphs for (1) and (2).

Author response:
The aforementioned content will be updated in my newly uploaded PDF.

Reviewer comment (Writing):
It would help readers if you briefly stated how Seq2Seq effectively handles delays.

Author response:
For addressing the delay issue, the Seq2Seq model in the paper serves two purposes: encoder pretraining and policy training. In the encoder pretraining phase, the dataset from the delay-free environment is initially randomly reconstructed to create information state sets with various delay sizes and their corresponding delayed state sequence sets. Specifically, assuming time $t$ and delay $d$ where $d \in [d_I, d_I+d_M]$, with $D=d_I+d_M$, the information state is denoted as $I_t = (s_{t-d}, a_{t-d},a_{t-d+1}, \cdots, a_{t-1})$，and the delayed state sequence is $g_t=(s_{t-d+1}, \cdots, s_{t-1})$.  For information states with a delay smaller than $D$, to facilitate batch training, these information states are zero-padded until their length reaches $D$. Next, a Seq2Seq encoder-decoder model is employed to predict the state based on the input information state and output the delayed state sequence. Specifically, a multi-layer perceptron encodes each element of the information state to generate corresponding embeddings. These embeddings are then fed into a GRU module to generate a hidden feature vector of a dimension determined by hyper-parameters. Finally, the Seq2Seq model is optimized using MSE loss to improve the accuracy of state sequence prediction, thereby refining the context representation of the information states. These hidden states are the ones utilized for policy learning.
During the policy training phase, the agent interacts with the environment in real-time, causing the length of the information state $I_t$ to dynamically change due to delays. We employ the pretrained Seq2Seq encoder, where information states of varying lengths $I_t$ are fed into the encoder. After several iterations, the encoder extracts delayed state information and maps it to fixed-dimensional context representations. Subsequent policy training relies on these context representations rather than the original state space. Moreover, there is no need for additional adjustments to the chosen policy algorithm due to changes in delays.

Reviewer comment (Presentation):
Last paragraph at section 4 is an experimental detail and would fit better at beginning of paragraph 2 in section.

Author response:
The aforementioned content will be updated in my newly uploaded PDF.

Reviewer comment (Writing):
First line of page 2: What does “dynamic environments” mean?

Author response:
“dynamic environments” means non-deterministic environments. The corresponding sentence is revised as: While effective in deterministic environments, their adaptability in nondeterministic environments needs further improvement.

Reviewer comment (Writing):
What the difference between d_M and D? They seem to represent the same quantity.

Author response:
$d_I \in Z^{+}$ denotes the intrinsic delay value resulting from long-distance transmission or heavy data transfers. Meanwhile, $d_M \in Z^{+}$ represents the maximum number of additional dropping steps, ensuring that $d_I + d_M$ remains within the agent’s capacity. The sum of these values is defined as $D = d_I + d_M$.

Reviewer comment (Evaluation):
The paper mentions the computation drawback of existing approaches. How does DEER compare computationally to the baselines considered?

Author response:
To address this inquiry, we conducted a comparative analysis of the time and performance of four algorithms—DEER, RLRD, DATS, and SACAS. Specifically, within the Hopper task, a fixed delay of 4 was established, with each algorithm undergoing three runs of 1 million environmental steps using different seeds. All algorithms were executed on the same NVIDIA GeForce RTX 4090 graphics card, and the time and performance comparison are illustrated in the Table.
| Algorithm | Time(hour)  | Scaled Return |
|-----------|-------------|---------------|
| DEER      | 5.771±0.012 | 0.8±0.11      |
| RLRD      | 70.67±0.4   | 0.85±0.05     |
| DATS      | 81.2±0.087  | 0.61±0.13     |
| SACAS     | 5.01±0.078  | 0.88±0.03     |

Reviewer comment (Evaluation):
How does breaking the end-to-end decision-making process into two stages improve interpretability? How is DEER more interpretable than say RLRD and DATS?

Author response:
Information states consist of the most recently observed states and their corresponding action sequences. The SACAS method utilizes information states as decision states to design value and policy functions. When there is significant delay, the dimensionality of the action part within the information states can be much larger than that of the states themselves. The SACAS method operates as a black-box process, disregarding the mentioned issue and completing training end-to-end, with the quality depending on the neural network's capacity and parameter settings. However, in the initial phase of our experiments, the delayed state information corresponding to delayed actions is actually known. Using this information, an encoder-decoder can map the information states into a latent space. This process, unlike SACAS's end-to-end random exploration of network parameters, is more directive, compelling the model to start from a good state representation. The improved interpretability is reflected here.

Reviewer comment (Reproducibility):
Does the encoder increase or decrease the dimensionality of the augmented state (s_{t-d}, a_{t-d}, …, a_t)? Does DEER intend to *compress* the MDP?

Author response:
From our experimental results, mapping the information state to a higher-dimensional context representation resulted in better performance of the trained policy in delayed environments. Moreover, the improvement in performance does not have a direct proportional relationship with the increase in dimensions. In other words, higher dimensions do not necessarily yield better results. To clarify this conclusion further, we extended our experiments on the Hopper and Walker2d tasks by encoding the information state into dimensions similar to the original information state (32 and 64 dimensions) under a delay of 4. Subsequently, we employed the SAC framework and the same set of parameters for policy training. The final outcomes of the encoders with different dimensions on the Hopper and Walker2d tasks are presented in the table below.

| Dimension | 32            | 64              | 128            | 256            | 512            |
|-----------|---------------|-----------------|----------------|----------------|----------------|
| Hopper    | 363.99±19.15  | 2496.68±1011.77 | 2993.60±325.07 | 2988.32±432.15 | 2413.60±562.31 |
| Walker2d  | 607.68±392.12 | 728.15±122.54   | 1645.92±286.45 | 2289.55±968.70 | 623.75±209.45  |

Based on the data in the table above, it's evident that when the dimension of DEER's pre-trained encoder is too small, such as setting it to dim=32, the encoder's capability is too weak to effectively represent delay information within the information state, resulting in decreased performance. Conversely, when the dimension of DEER's pre-trained encoder is excessively large, for instance, with dim=512, although the encoder has numerous parameters and enhanced capabilities, there is a risk of potential overfitting to the current dataset, leading to decreased performance. Moreover, as the dimension shifts from 64 to 256, the performance improves for both tasks, indicating that the representation capability of the context representation indeed influences the final outcome.

Reviewer comment (Novelty):
The novelty of DEER is moderate. From my understanding, the key novelty is a framework combining "offline-assisted online RL" and "Encoder in RL".

Author response:
**[A1]** The reviewer has indeed grasped the core concept of the DEER method. However, summarizing DEER as a mere fusion of "offline-assisted online RL" and "Encoder in RL" is somewhat simplistic. Firstly, DEER stands as the pioneering approach that bridges data from delay-free tasks with applications in delayed tasks. Secondly, DEER doesn't necessitate modifications to subsequent policy algorithms; it seamlessly applies existing algorithms. Lastly, although DEER utilizes a simple Seq2Seq model, context representation derived from its encoder  effectively extracts delay-related information, significantly bolstering decision-making capabilities.

Reviewer comment (Theory):
The definition of RDDMP is quite complex and probably unrealistic. One concern is about the random delay value $z_t$ with dropping probablity $\mu$. $z_t = d_I$ if $z_{t-1} = d_I + d_M$, which means the $d_M$ steps of delay suddenly "disappear". How can this happen in reality? My suggestion is to modify it as (this ensures that there is no sudden change of delay steps):  
   - $z_t = z_{t-1} - 1$ if $z_{t-1} > d_I$ and with probability $ \mu_{catch} $  
   - $z_t = z_{t-1} + 1$ if $z_{t-1} < d_I + d_M$ and with probability $ \mu_{lag}$  
   - $z_t = z_{t-1}$ otherwise

Author response:
**[A2]** The reviewer's statement is accurate. In our defined scenario, when the cumulative count of consecutive frame drops reaches $d_M$, it resets. There are two perspectives to explain this setup:
1.The probability of dropping frames, $\mu$, for each instance, compounded over $d_M$ consecutive drops, equates to $d_M^{\mu}$. This value tends to be relatively small, signifying that it is an infrequent occurrence.
2.In real-world scenarios, when an agent encounters $d_M$ consecutive instances of not receiving a state, it surpasses the agent's capacity to handle delay. Hence, it necessitates the remote end to ensure the appearance of the next state.
The proposed scenario by the reviewer is indeed more reasonable but requires a slight modification: when the agent observes the returned state, the current delay reverts to $d_I$, representing the intrinsic delay value resulting from long-distance transmission or heavy data transfers. Hence, $z_t = d_I$ if $z_{t-1} > d_I$ with a probability of $\mu_{catch}$. Based on this stochastic delay setting, we compared four algorithms (DEER, RLRD, DATS, SACAS) in the Hopper and Walker2d tasks, setting $d_I=2$, $d_M=4$, and $\mu=0.2, 0.4, 0.6$. Each algorithm was experimented with three different random seeds for 1 million environment steps in each environment configuration. The comparative results are shown below.

| Tasks($\mu=0.2$) | DEER            | RLRD           | DATS            | SACAS        |
|----------------|-----------------|----------------|-----------------|--------------|
| Hopper         | 2583.90±915.03  | 2391.16±473.86 | -1064.50±426.87 | 820.94±15.58 |
| Walker2d       | 3940.05±1539.67 | 3013.65±571.69 | -1196.97±418.48 | 670.84±74.90 |
|                |                 |                |                 |              |

| Tasks($\mu=0.4$) | DEER            | RLRD           | DATS            | SACAS        |
|----------------|-----------------|----------------|-----------------|--------------|
| Hopper         | 2542.54±960.73  | 1609.96±403.12 | -754.47±225.30  | 609.33±15.80 |
| Walker2d       | 4247.60±1441.24 | 2062.80±198.73 | -1237.48±350.60 | 518.13±47.12 |
|                |                 |                |                 |              |

| Tasks($\mu=0.6$) | DEER            | RLRD          | DATS           | SACAS        |
|----------------|-----------------|---------------|----------------|--------------|
| Hopper         | 2129.59±995.17  | 638.93±230.15 | -735.94±197.38 | 398.93±48.83 |
| Walker2d       | 2578.52±1614.18 | 908.23±222.47 | -672.75±238.66 | 360.65±28.46 |
|                |                 |               |                |              |

From the table above, it's evident that even under the requested random delay settings by the reviewer, the performance of DEER significantly surpasses other algorithms (although there might be instances where it closely aligns with RLRD scores). This underscores the significance of state representation in decision-making, further emphasizing that the pre-trained encoder in the DEER algorithm indeed extracts delay-related information from the information state.

Reviewer comment (Experiments):
DEER should also be compared with vanilla SAC in delay=0 case. This is important to understand the effect of pretrained encoder in deep RL and probably explain why DEER significantly outperforms expert in Swimmer (Figure 4).

Author response:
**[A3]** Whether in fixed or random delay settings, the legend labeled "Expert" in the graph corresponds to the performance of vanilla SAC trained with delay=0, as depicted in the Figure 4 and Figure 5 .

Reviewer comment (Presentation):
Fonts are too small in Figure 3. I cannot recognize the subscripts on the printed paper.

Author response:
**[A7]** The aforementioned content will be updated in my newly uploaded PDF.

Reviewer comment (Presentation):
A table summarizing the numerical results (including the converged task return of each task and average performance of all tasks) could help the readers get easier access to the empirical results.

Author response:
**[A7]** The aforementioned content will be updated in my newly uploaded PDF.

Reviewer comment (Writing):
A typo in the title "resillient".

Author response:
**[A7]** The aforementioned content will be updated in my newly uploaded PDF.

Reviewer comment (Writing):
In the end of the first paragraph, it was mentioned that minor delays markedly amplify the risk in self-driving. Is there any reference? Following this question, are there more examples (better with data) of practical scenarios that suffer heavily from delays?

Author response:
**[A5]** In the '5G + Intelligent Transportation' series white paper (link provided below [4]), clear indicators are specified for the latency requirements in four scenarios of intelligent driving, as detailed below:
Scenario	Assisted Driving	Remote Driving	Autonomous Driving	Vehicle Convoy

| Scenario             | Assisted Driving                             | Remote Driving | Autonomous Driving | Vehicle Convoy   |
|----------------------|----------------------------------------------|----------------|--------------------|------------------|
| Maximum Latency (ms) | 20ms/100ms（V2V） 100ms（V2I） 1000ms（V2N） | 20ms           | 3ms-100ms          | 10ms-25ms        |
|                      |                                              |                |                    |                  |

Additionally, in literature [5], practical observations indicate a delay of 0.2 to 0.5 seconds between steering signals from the vehicle's computer and the actual steering wheel operation. While the vehicle remains in motion, this leads to substantial deviations in path tracking, potentially resulting in a loss of control when traveling at high speeds. Furthermore, literature [6] notes that hydraulic vehicle braking systems typically require over 0.4 seconds to achieve the necessary deceleration, significantly impacting the planning and control modules of Connected and Autonomous Vehicles (CAVs).
In various other domains such as control of robotic systems [7], distributed computing [8], and management of traffic networks [9], delays have demonstrated adverse effects on the performance of corresponding intelligent agents."

**Reference**

[1] https://github.com/baimingc/dambrl

[2] https://github.com/rmst/rlrd

[3] https://github.com/pranz24/pytorch-soft-actor-critic

[4]http://13115299.s21d-13.faiusrd.com/0/1/ABUIABA9GAAglcTHjQYojIWZswY.pdf?f=%E4%B8%AD%E5%9B%BD%E8%81%94%E9%80%9A5G+%E8%BD%A6%E8%BD%BD%E5%AE%9A%E5%88%B6%E5%8C%96%E6%A8%A1%E7%BB%84+%E7%99%BD%E7%9A%AE%E4%B9%A62019.pdf&v=1639047701

[5]赵建辉, 高洪波, 张新钰, 等. 基于时间延迟动态预测的自动驾驶控制[J]. 2018.

[6]Bayan, Fawzi P., et al. "Brake timing measurements for a tractor-semitrailer under emergency braking." SAE International Journal of Commercial Vehicles 2.2009-01-2918 (2009): 245-255.

[7]Imaida, Takashi, et al. "Ground-space bilateral teleoperation of ETS-VII robot arm by direct bilateral coupling under 7-s time delay condition." IEEE Transactions on Robotics and Automation 20.3 (2004): 499-511.

[8]Hannah, Robert, and Wotao Yin. "On unbounded delays in asynchronous parallel fixed-point algorithms." Journal of Scientific Computing 76 (2018): 299-326.

[9]Dollevoet, Twan, et al. "Delay propagation and delay management in transportation networks." Handbook of Optimization in the Railway Industry (2018): 285-317.

Reviewer comment (Evaluation):
Some shortcomings of previous methods, like inference time, model precision and cumulative errors also exist in DEER. DEER uses an RNN which is not very efficient in terms of the inference time.

Author response:
DEER leverages existing data from delay-free environments to learn an encoder-decoder for multi-step prediction. During the decision-making phase, DEER employs the encoder's representation of input states and action sequences, termed as context representation, rather than the final state from multi-step predictions. This representation extracts decision-relevant information from the most recent observed states and action sequences, thereby mitigating sensitivity to deviations from the true state caused by model precison and cumulative errors. 

**Reference**  
[1] Zhang, Haichao, Wei Xu, and Haonan Yu. "Policy Expansion for Bridging Offline-to-Online Reinforcement Learning." The Eleventh International Conference on Learning Representations. 2022.

Reviewer comment (Experiments):
RNN also has cumulative errors. Except for using more data (the offline dataset), why is DEER more effective than previous methods?

Author response:
DEER leverages existing data from delay-free environments to learn an encoder-decoder for multi-step prediction. During the decision-making phase, DEER employs the encoder's representation of input states and action sequences, termed as context representation, rather than the final state from multi-step predictions. This representation extracts decision-relevant information from the most recent observed states and action sequences, thereby mitigating sensitivity to deviations from the true state caused by model precison and cumulative errors. 

**Reference**  
[1] Zhang, Haichao, Wei Xu, and Haonan Yu. "Policy Expansion for Bridging Offline-to-Online Reinforcement Learning." The Eleventh International Conference on Learning Representations. 2022.