Paper: Addressing Real-Time  Fragmentary Interaction Control Problems via Muti-step Representation Reinforcement Learning (ICLR-2024)

Reviewer comment (Experiments):
The experiemnt part may not fully match with the motivation of the paper. Generally speaking, simulation environments such as Mujoco do not require the use of framentary control.

Author response:
## The experiment part may not fully match the motivation of the paper. Generally speaking, simulation environments such as Mujoco do not require the use of fragmentary control. 
Fragmented interactions exist in many virtual scenarios. For example, remote control of NPCS: In the game, the cloud server needs to control the terminal NPC in real time. NPC stalling can significantly degrade the user experience. So we follow works with related evaluation scenarios (e.g.RTAC[1]RDAC[2]). We used mainstream virtual scenarios (Mujoco, D4RL) to simulate remote control of NPC tasks instead of using Mujoco directly. The details of constructing fragmented interactions simulation tasks are described in the experiment section Sec 4.2: In real-world tasks, an execution time for one step is about 0.05s to 0.25s. And most of the prohibited interaction duration is 0.5s to 2s. Thus, we set the interval step length as 8 to simulate the real-world setting.  We will highlight the task settings in the new version. And put it at the top of the experiment section.

[1] Chris Pal, et al. "Real-time reinforcement learning." NeurIPS (2019).

[2] Ramstedt, Simon, et al. "Reinforcement learning with random delays." arXiv preprint arXiv:2010.02966 (2020).

Reviewer comment (Evaluation):
Why does the method significantly outperform TD3 with advanced decision? Please explain the comparison between the baselines in more details.

Author response:
## Why does the method significantly outperform TD3 with advanced decision?
Explanation:
- Td3-advance decision needs to output the whole action sequence (concatenate c steps), and the long action sequence will increase the output dimension (output_dim=c * single_action_dim). This increases the difficulty of the action space exploration. Thus, the agent cannot learn the optimal policy (mentioned in the introduction).
- Our method represents diverse action sequences in low-dimensional space. Our method reduces the difficulty of exploration and performs better.
- Ours is a plug-in method and can be applied to all major reinforcement learning methods. To ensure experimental fairness, all methods use Td3. This eliminates the framework advantages that Td3 brings to advance deisicion.

Verified by experiment:
- Following results show the sensitivity of three methods to the action sequence dimension. The first table runs on Walker and the second runs on Maze-hard.
- The exploration ability of Td3-advance decision decreases significantly with the increase of sequence length.
- TD3-frameskip has better exploration ability than TD3- advanced decision. This is because frameskip reduces the spatial dimension of exploration by sacrificing action variety (repeating the same action).
- Our method has the best performance by reducing the dimension of exploration space and ensuring the diversity of action through the representation of action space.

| method     | 12 dim| 24dim| 36 dim|
| :-----------: | :-----------: | :------------: | :-----------: |
| Ours |271.2 ± 11.4|258.3 ±15.2|246.5 ± 21.3|
| TD3- frameskip |231.2 ± 14.6|112.1 ± 3.2|88.7 ± 2.5|
| TD3- advanced decision  |83.5 ± 5.7|31.9 ± 7.4|85.1 ± 2.8|

| method     | 18 dim| 36 dim| 54 dim|
| :-----------: | :-----------: | :------------: | :-----------: |
| Ours |4715.6 ± 343.1|4613.2 ± 362.7|4215.9 ± 428.3|
| TD3- frameskip |3714.7 ± 252.1|941.6 ± 603.2|691.3 ± 122.4|
| TD3- advanced decision |2368.2 ± 316.4|913.2 ± 592.1|718.3 ± 176.8|

Reviewer comment (Experiments):
What is the interaction pattern between the executor and the agent in the real-world experiment?

Author response:
## The real-world robotic control experiment lacks important details, including the interaction pattern between the executor and the agent in the real-world experiment.
The 21-joint snake robotic uses a rolling gait. The algorithm is arranged in the cloud server, and the decision end interacts with the snake robot through the local area network. [This is the interaction pattern figure.](https://anonymous.4open.science/r/ICLR2024-C0F6/interaction_pattern.png)

Snake robot tracking is a periodic information acquisition task. The reason for this phenomenon is not packet loss or delay, but the unique rolling gait of the snake robot. This gait of the snake robot is the most efficient gait for outdoor work, which includes the rolling of the head sensor. However, this gait causes the next observation to be acquired only when the head sensor rolls to a position parallel to the ground. If the rolling motion is interrupted midway, the next state cannot be obtained, leading to ineffective shaking of the robot. 

The snake robot needs to scroll from the starting point to the target point in a scene with a size of 5 square meters. Each observation step is 54d, and the action sequence time step is 20, so the action sequence dimension is 1.08k. The control frequency of the real interaction is 20 Hz. We describe the snake robot tracking task in detail and provide several [visualization results](https://anonymous.4open.science/r/ICLR2024-C0F6/) in the new version.

Reviewer comment (Novelty):
The authors might need to connect more with existing problem formulations. The FIMDP looks related to partially observable MDPs and MDP with reward delays. I can get a rough sense that there are differences between FIMDP and these existing problem formulations, but not very clear. The authors should add a clear discussion to distinguish FIMDP from the existing related problem formulations.

Author response:
## The questions are also related to weakness, what are the differences between FIMDP and POMDP, or MDP with reward delays?
We compare FIMDP with reward delayed MDP and POMDP in the new version:
- The main difference between FIMDP and delay MDP: In FIMDP All environmental information is delayed (observation, action sequence, reward). Besides, agents are not allowed to pause midway. However, in reward delay MDP agents just need to address the reward delay. And reward delay MDP does not consider the harm caused by the agent stalled in the middle.
- POMDP does not involve delay, the agent gets a local observation at each step. In contrast, the FIMDP has a delay in obtaining observations, but the agent can obtain global observations

Reviewer comment (Novelty):
What are the differences between FIMDP and POMDP, or MDP with reward delays?

Author response:
## The questions are also related to weakness, what are the differences between FIMDP and POMDP, or MDP with reward delays?
We compare FIMDP with reward delayed MDP and POMDP in the new version:
- The main difference between FIMDP and delay MDP: In FIMDP All environmental information is delayed (observation, action sequence, reward). Besides, agents are not allowed to pause midway. However, in reward delay MDP agents just need to address the reward delay. And reward delay MDP does not consider the harm caused by the agent stalled in the middle.
- POMDP does not involve delay, the agent gets a local observation at each step. In contrast, the FIMDP has a delay in obtaining observations, but the agent can obtain global observations

Reviewer comment (Experiments):
If we have/learned a world model for the environment, can we do the model-based predictions, like model predictive control to solve the FIMDP? (this might be the most straightforward method that first came into my mind.) How does this compare to learning the multi-step representations?

Author response:
## 2. If we have/learned a world model for the environment, can we make the model-based predictions, like model predictive control to solve the FIMDP? (this might be the most straightforward method that first came into my mind.) How does this compare to learning the multi-step representations?
This question is valuable. In fact, we originally wanted to build a world model to achieve comprehensive environmental perception. However, the interval of FIMDP is too long and some tasks have random intervals. It is difficult to construct effective dynamics models and reward models under delayed conditions, which can be summarized as we cannot model random delays effectively. Therefore, we chose a method that is easier to train and robust to delay: construct an action latent space. To prove that our approach is more efficient than the model-based approach in FIMDP tasks. The following results show the performance of the original MBPO[1] based world model method (Average of the 10 runs. Interval is 10). In the future, we will further think about how to make Model-based methods effective in FIMDP tasks.

| Methods     | Walker| maze-hard| HalfCheetah|
| :-----------: | :-----------: | :------------: | :-----------: |
| MBPO-based |3781.2 ± 217.5|203.6 ± 15.8|6153.1 ± 381.2|
| Ours  |4715.6 ± 343.1|271.2 ± 11.4|7012.1 ± 131.4|

[1] Janner, Michael, et al. "When to trust your model: Model-based policy optimization." Advances in neural information processing systems 32 (2019).

Reviewer comment (Theory):
Why is clustering representations of actions that have similar environmental effects better than clustering action sequences with similar values or rewards? Can you provide a more in-depth explanation and analysis?

Author response:
## Why is clustering representations of actions that have similar environmental effects better than clustering action sequences with similar values or rewards? 
Explanation is as follows (highlight in new version):
- Get accurate Q value is difficult: In sparse reward environments (such as FIMDP), reward and Q are difficult to obtain and the evaluation of Q values in the early stage of training is inaccurate. In contrast, environmental dynamic is more reliable and accessible.
- Environmental dynamic contains more information: The same reward or Q value may correspond to different environmental changes, but the same environmental change must have the same reward or Q value.
- Environmental dynamic is reward-agnostic: In FIMDP, rewards are sparse. Environment dynamics do not require a per-step reward. Therefore, environmental dynamic representation is more robust in FIMDP.

Further, we compare these 3 representational learning methods. We only changed the clustering representations.

|Method|Halfcheetah|Walker|maze-hard|
| :- | :- | :- | :- |
|Env dynamic (ours)|7012.1±131.4|4821.6±427.6|311.4±16.3|
|Q|6386.1±412.7|4021.6±313.7|275.2±13.7|
|reward|6618.1±372.7|4188.3±185.5|253.9±21.5|

Reviewer comment (Experiments):
In the random FIMDP tasks mentioned in the paper, is the number of decision steps fixed within one trial or randomly decided during execution? As shown in Figure 11, a larger interval leads to lower performance, how will the method perform if trained with longer action sequences but executed with a shorter interval, compared with training with shorter action sequences?

Author response:
## Is the number of decision steps fixed within one trial? How will MARS perform if trained with longer action sequences but executed with a shorter interval, compared with training with shorter action sequences?
The number of decision steps is fixed. The following results show that reducing the number of steps as much as possible can improve scores (Average of the 10 runs. Interval is 6). As the number of steps increases, both the sequence dimension and the reward sparsity increase. These lead to the difficulty in exploration.
|task|6 step|12 step|18 step|
| :-| :-| :- | :- |
|Walker|4715.6±343.1|4613.2±362.7|4215.9±428.3|
|maze-hard |271.2±11.4|258.3±15.2|246.5±21.3|

Reviewer comment (Evaluation):
I noticed some interesting results. Why does it appear that MARS has a more pronounced advantage over frame-skips in simpler tasks than in more complex tasks?

Author response:
## MARS has a more advantage over frame-skips in simple tasks than in complex tasks?
Because the VAE hyperparameters of MARS are not optimized in difficult tasks. We adjust the VAE hyperparameters while keeping the remaining hyperparameters unchanged for both methods （following table). By doing this MARS has a more pronounced advantage on complex tasks. (Old score of MARS in maze-Hard:315.2±13.9 maze-Hard:6417.3±317.3)

| method| maze-hard|HalfCheetah|
| :- | :- | :-| 
|Ours|356.4±16.3|8631.5±265.2|
|TD3- frameskip|285.7±12.5|5842.3±306.7|

Reviewer comment (Reproducibility):
The application form of the method needs further clarification. Does the decoder need to be run on the execution device? If so, does this mean that latent representations will also be lost?

Author response:
## Does the decoder need to be run on the execution device? Does this mean that latent representations will also be lost?
We do not deploy the decoder to the executor. The agent makes c step decision based on each timestep information, so each time step (t+i) will receive c times in the future. Besides, our decision steps are set at maximum intervals to ensure that the executor receives the next sequence before the previous sequence is completed.

Reviewer comment (Presentation):
Should the "Musar" method in Fig.5 and 6 be referred to as "MARS"?

Author response:
## Should the "Musar" be referred to as "MARS"?
Correct this in the new version.