Paper: Can Agent Learn Robust Locomotion Skills without Modeling Environmental Observation Noise? (ICLR-2024)

Reviewer comment (Experiments):
The paper shows the algorithm boosts the stability of the training process. However, it does not consider whether the method is robust when the test environment has random noises with different distributions. The authors should consider if the policy is robust in unseen noise distribution, which is much more essential for using RL in the real-world.

Author response:
As suggested, we add sufficient descriptions about unseen noise distributions in section 5.3 as below：

Generalization is an important metric for deep learning algorithms. In this experiment, we evaluate if SIMA can be generalized to the environments with various unseen environmental noise. Specifically, we adopt both stationary noise (high-frequency noise) and non-stationary noise (intermittent disturbance), each with 9 groups of different noise characteristics. The stationary noise groups follow gaussian process with frequency mean as $f=[16,32,64]$ (khz), and amplitude variance as $A=[0.4,0.7,1.0]$. The corresponding results are illustrated in Figure 6 (a). Alternatively, the non-stationary noise groups are set to be unpredictable in occurance and duration periods. In such case, we name it as intermittent disturbance which follows uniform distribution with duration period characteristics as $T=[10,15,20]$ (steps), and amplitude characteristics as $A=[5,10,15]$. All these noise durations occur in range of $0$ to $T$ randomly as illustrated in Figure 6 (b). All experimental results are collected from HalfCheetah environments. It can be seen that SIMA outperforms RL-vanilla by $95\%$ in task success rate (we define task success once an agent never falls down during running). Furthermore, SIMA also performs significantly smaller variance in average return than RL-vanilla, which indicates SIMA maintains more stable under various unseen environmental noise.

Reviewer comment (Experiments):
The paper mainly compares whether the training curves of various RL algorithms is better than SIMA when combined with some relatively simple encoders (filters, lstm). However, I think this is slightly unfair for state representation learning. The process of SIMA is obviously more complicated and uses more training objectives. The author should compare it with more solid works on state representation learning for locomotion, such as [1][2].

Author response:
As suggested, we add an additional experiment to make comparisons with other well-adopted robotics locomotion state representation learning baselines in Appendix J as below:

In this section, we mainly focus on comparing the performance of SIMA with Belief\_AE [1] and RMA [2]. 

Figure 15 and Figure 16 show experimental results under single-sensor noise conditions, i.e., during entire locomotion task episodes, observation noise only appears in one signal-input channel of the sensors. Figure 16 shows the average return performance among SIMA, Belief\_AE, and RMA by 50 episodic trials. It can be clearly seen in Figure 16, SIMA and Belief\_AE performs better than RMA. 

Figure 17 and Figure 18 show experimental results under multi-sensor noise conditions. Figure 18 shows that SIMA also outperforms Belief\_AE by $25\%$ in average return. More specifically, the "Belief state encoder" in Belief\_AE leverages an attentional gate explicitly controls which aspects of exteroceptive data to pass through. However, the Belief\_AE could handle noises only in one-demensional observaton (e.g., the elevation map in [1]), and lacks of an important mechanism to assess the noise scale for all sensors input.

Reviewer comment (Experiments):
Can the SIMA still perform better on a totally unseen noise distribution (after the student policy learning stage)? Is it still robust when noise distribution is shifted?

Author response:
As suggested, we add sufficient descriptions about unseen noise distributions in section 5.3 as below：

Generalization is an important metric for deep learning algorithms. In this experiment, we evaluate if SIMA can be generalized to the environments with various unseen environmental noise. Specifically, we adopt both stationary noise (high-frequency noise) and non-stationary noise (intermittent disturbance), each with 9 groups of different noise characteristics. The stationary noise groups follow gaussian process with frequency mean as $f=[16,32,64]$ (khz), and amplitude variance as $A=[0.4,0.7,1.0]$. The corresponding results are illustrated in Figure 6 (a). Alternatively, the non-stationary noise groups are set to be unpredictable in occurance and duration periods. In such case, we name it as intermittent disturbance which follows uniform distribution with duration period characteristics as $T=[10,15,20]$ (steps), and amplitude characteristics as $A=[5,10,15]$. All these noise durations occur in range of $0$ to $T$ randomly as illustrated in Figure 6 (b). All experimental results are collected from HalfCheetah environments. It can be seen that SIMA outperforms RL-vanilla by $95\%$ in task success rate (we define task success once an agent never falls down during running). Furthermore, SIMA also performs significantly smaller variance in average return than RL-vanilla, which indicates SIMA maintains more stable under various unseen environmental noise.

Reviewer comment (Evaluation):
In Appendix E, how are the scores calculated? Why can scores above 99 be achieved under clean experimental settings?

Author response:
This part was indeed not explained clearly. All corresponding experiment descriptions are revised totally in Appendix F.

We extensively train individual locomotion control policies in six clean environments (no environmental noise), and record the average returns (e.g., CartPole-200, Halfcheetah-2500). We then adopt these results as benchmarks. To map the experimental results from these different algorithms into a unified comparison range, we collect returns from 30 trials that are all normalized into values between 0.0 and 100.0 as shown in Table 1, in which clean indicates there is no observation noise in the evaluation environments.

All these algorithms achieve above 99 scores in all six clean environments. It indicates all these algorithms perform normally without  noise.

Reviewer comment (Presentation):
How did the authors draw policy distribution? Is it t-SNE visualization of action? How the actions are sampled?

Author response:
Yes, we utilize t-SNE to visualize policy distributions. 

Firstly, we train a teacher policy with state truth as a reference (illustrated as red dots scattered in Figure 5), a student policy to reconstruct the optimal policy embeddings under these unexpectable disturbance, and a vanilla DRL policy to serve as the control group. 

Secondly, these policies are deployed in two evaluation environments, with stationary noise and non-stationary noise, respectively. The teacher policy can directly observe the state truth, while the SIMA policy and the vanilla DRL policy make decisions by the observations with environmental noise. 

Consequently, actions of 30 trials are recorded to compare the differences between the distribution of the three policies.

Reviewer comment (Experiments):
The main experiment shows marginal improvements over the baseline. Figure 4 is quite suspicious: SIMA mostly performs the same as the baselines but will have sudden drops in performance, after which the performance will rise again. This suggests a mistake in the experimental setting or visualization.

Author response:
**For Weaknesses 1,**

This part was indeed not explained clearly and the performance drop is normal.  It's worth noting that SIMA consists of three training stages. In the first two stages, SIMA utilizes clean observations to train a teacher policy and a de-noising module. Due to the fact that the teacher policy of SIMA agent in the first two stages was trained without environmental observation noise, it performs similar to other algorithms in the first half of the training curves in Figure 4.

In the last stage, we initialize a student policy network to complete robust locomotion control tasks in noisy environments, which utilizes optimal state representations learned in the first two stages. Since the student policy lacks of guidance from the teacher policy in the first two stages, a notable performance drop pops-up in the middle section of the learning curves as shown in Figure 4. Consequently, after a short warm-up period intervened by the teacher policy, the performance of SIMA outperforms other algorithms. To give a clearer description of the three stages of SIMA, we provide a schematic diagram in Figure 8 in Appendix E.

Reviewer comment (Novelty):
The idea of masked modeling for reinforcement learning is not new. See [1], [2]. Masked modeling itself could help improve the sample efficiency and robustness to noises. The paper does not make comparisons with these methods.

Author response:
**For Weaknesses 2,**

The overall motivation, pipeline, and architecture among Masked World Models (MWM), Multi-View Masked World Models (MV-MWM), and SIMA are totally different. One thing in common is that they all use masks. The differences can be summuarized as three-fold:

(1) Motivation: MWM and MV-MWM mainly focus on sample efficiency enhancement for model-based RL, but SIMA mainly focuses on enhancing system robustness against environmental noise, especially for non-stationary noise with model-free RL. 

(2) Pipeline: MWM and MV-MWM employ a 2-tier sampling-training cycles, and iterate untill converge. In contrast, SIMA employs a "never turn back" training mode with 3 stages.

(3) Architecture: MWM and MV-MWM employ an end-to-end MBRL learning architecture, but SIMA proposes a novel teacher-student de-noising policy distillation architecture.

To make these differences descriptions clearer, we add an additional experiment to make further comparations about this in Appendix H. Here, we find an interesting phenomenon in Figure 13. From this figure, it can be clearly seen that SIMA shows significant improvements in learning efficiency compared to MWM in the first two training stages. We speculate the proper reason is the teacher policy training (stage 1) can directly observe the system state truth, thereby makes SIMA a quicker learner in locomotion skills. The corresponding locomotion skills bring correct state and action distributions that are critical to de-noising module learning performance in stage 2 subsequently. In contrast, MWM cannot directly observe system state truth in all training stages. Afterwards, we found that the SIMA algorithm shows a significant performance improvement on average returns after being connected to student policy distillation (stage 3). The essential reason is that SIMA fully learned the optimal state representations under environmental noise conditions by robot running procedures during the first two stages, thus effectively suppress the environmental noise encountered.

To further demonstrate this conclusion, we conduct an additional experiment on the core differences between MWM and SIMA. In case of MWM, the sampling-training cycles of MWM last for many rounds, and in each new round, due to the drastic change of robot running policy distribution. MWM needs to re-adapt to brand new world models and de-noising modules in all training rounds. In view of this, we list the probability distributions of states and actions for multiple MWM training rounds in Figure 13. It can be clearly seen that there have been significant changes in the probability distributions of states and actions corresponding to the new trained robot's running skills in each round, resulting in unstable updates to the MWM policy. In contrast, SIMA employs a "never turn back" training mode throughout the entire training process. Once the first two stages have properly learned de-noising skills, stage 3 only needs to complete single round of student policy distillation based on this. This ensures that SIMA only needs to adapt to new de-noising probability distribution of states and actions once, and thus achieves a better learning performance. This novel learning pipeline of SIMA brings significant improvement in learning curves observed in Figure 14.

Furthermore, Multi-View Masked World Models (MV-MWM) mainly enhances multi-viewport image inputs based on MWM, and the essential difference with SIMA is described as aforementioned.

Reviewer comment (Writing):
The writing of the paper could be further improved, by reducing the use of long abbreviations, e.g. VICOR, RMA (the name is also used in a different sim2real transfer for locomotion paper), MASOR

Author response:
**For Weaknesses 3,**

We abandon the complex abbreviations, e.g., VICOR, RMA, and MASOR, etc as suggested. Now the overall expression looks clearer and easy to comprehend.

Reviewer comment (Evaluation):
In section 5.1 LEARNING PERFORMANCE UNDER ENVIRONMENTAL OBSERVATION NOISE (RQ1), the performance of the proposed algorithm shows sharp declines in all locomotion control tasks. I suggest the authors to add more contents to analyze the causes in the manuscript. What's more, more training steps should be visualized since the performance of some baselines is still increasing such as RL-DR and RL-vanilla in the Walker2D environment.

Author response:
This part was indeed not explained clearly and performance drop is normal. 

It's worth noting that SIMA consists of three training stages. In the first two stages, SIMA utilizes clean observations to train a teacher policy and a de-noising module. Due to the fact that the teacher policy of SIMA agent in the first two stages was trained without environmental observation noise, it performs similar to other algorithms in the first half of the training curves as illustrated in Figure 4.
In the last stage, we initialize a student policy network to complete robust locomotion control tasks in noisy environments, which utilizes optimal state representations learned in the first two stages. Since the student policy lacks of guidance from the teacher policy in the first two stages, a notable performance drop pops-up in the middle section of the learning curves as shown in Figure 4. Consequently, after a short warm-up period intervened by the teacher policy, the performance of SIMA outperforms other algorithms. 

To give a clearer description of the three stages of SIMA, we provide a schematic diagram in Figure 8 in Appendix E. Furthermore, all corresponding experiment descriptions in Section 5.1 are revised totally and carefully to provide a clearer expression. 

As suggested, we add more training steps in the Walker2D-v0 ($260 \times 10^{4}$ to $300 \times 10^{4}$ ) and  Halfcheetah-v0 ($200 \times 10^{4}$ to $240 \times 10^{4}$). The results in Figure 4 show that all algorithms have converged. It can be clearly seen that SIMA outperforms other state-of-the-art methods in learning performance.

Reviewer comment (Writing):
There are so many abbreviations throughout the paper, e.g., DRPOP, SIMA, VICOR, RMA, and MASOR, making them difficult to understand. In my humble opinion, one or two abbreviations are good.

Author response:
We abandon the complex abbreviations, e.g., VICOR, RMA, and MASOR, etc as suggested. Now the overall expression looks clearer and easy to comprehend.

Reviewer comment (Experiments):
I suggest the authors to attempt BetaVAE and Beta-TC-VAE as the generative model to encode masked state observation into the the latent state representation and reconstruct state observation from latent state representation.

Author response:
As suggested, we conduct a new experiment to evaluate BetaVAE, and Beta-TC-VAE as the generative models to encode masked state observation in the third paragraph of Appendix G as below: 

Generative Models. We attempt BetaVAE, and Beta-TC-VAE as the generative models to encode masked state observation into the the latent state representation. Experimental environment is Halfcheetah-v0 with observation noise. For BetaVAE model, we set three sets of parameters $\beta = 1$ (i.e., vanilla VAE), $\beta = 4$, and $\beta = 50$. For Beta-TC-VAE model, we set the parameters as $\alpha=1$, $\beta = 1$, and $\gamma=6$. The results can be seen in Figure 11. The four generative models have almost the same impact on SIMA performance, with both mean and variance being close. BetaVAE and Beta-TC-VAE aim to balance latent channel capacity and independence constraints with reconstruction accuracy. The locomotion control experimental scenarios in this paper belong to continuous dynamic system. Good disengagement could not result in a significant improvement in downstream continuous control tasks. Experimental results indicate that vanilla VAE can satisfy the requirements of SIMA.