Paper: Hybrid Internal Model: Learning Agile Legged Locomotion with Simulated Robot Response (ICLR-2024)

Reviewer comment (Experiments):
Many technical details are poorly described or totally missing: what’s the exact task legged robot need to solve (follow forward velocity or follow angular velocity or anything else), what command is included in the user-command, what’s the linear velocity and angular velocity range for the input.

Author response:
We have fixed the typos in our paper. The task is to tracking the user command including linear velocity in x-axis, linear velocity in y-axis and yaw angular velocity, The range differs from terrain type, for example, the command of linear velocity in x-axis on slopes and rough slopes is [-3.0, 3.0] m/s and [-1.0, 1.0] m/s on stairs, the range of linear velocity in y-axis is [-1.0, 1.0] m/s on all terrains, the range of yaw angular velocity on slopes and rough slopes is [-3.0, 3.0] rad/s and [-1.0, 1.0] rad/s on stairs. We also update a more clear decription of technical details in $\color{red}{section 3}$ in current version.

Reviewer comment (Experiments):
There is no quantitative comparison with any baseline in the paper (no result in simulation and no result in real world).

Author response:
We have done more evaluation of our method.
* We conduct quantitative comparison with various baselines in both simulation and real world in $\color{red}{section 4.3}$ and $\color{red}{section 4.4}$ in current version.

* We already clearify the task description of both simulation and real world benchmarks. In simulation benchmarks, we test the trackng error across various terrains and command ranges. In real world benchmarks, we set up three types of terrains to evaluate the performance of real world deployment, including success rate, number of stairs, maximum weights of payload, and other metrics. You can checkout the details in $\color{red}{section 4.3}$ and $\color{red}{section 4.4}$ in current version.

* We compared our method with the baselines including the regression method, RMA, MoB, Built-in MPC and ours w/o auxiliary loss, the experimental results show that our method outperform other baselines in both simulation and real-world. The experimental details are in $\color{red}{section 4.3}$ and $\color{red}{section 4.4}$.

Reviewer comment (Evaluation):
The only comparison result (The training curve in Figure 6) provides limited information, since there is no specific description of the task. It’s hard to see the actual performance gap of different methods in terms of the “task target” like velocity tracking and angular.

Author response:
We already clearify the task description of both simulation and real world benchmarks. In simulation benchmarks, we test the trackng error across various terrains and command ranges. In real world benchmarks, we set up three types of terrains to evaluate the performance of real world deployment, including success rate, number of stairs, maximum weights of payload, and other metrics. You can checkout the details in $\color{red}{section 4.3}$ and $\color{red}{section 4.4}$ in current version.

Reviewer comment (Experiments):
There is no actual baseline compared in the experiment section, since the method uses only proprioception observation of the robot. At least RMA, MoB should be compared quantitatively, and provide corresponding training curve. Since this work claims the proposed method provides better performance and sample efficiency against two-stage methods, at least comparison is needed.

Author response:
We compared our method with the baselines including the regression method, RMA, MoB, Built-in MPC and ours w/o auxiliary loss, the experimental results show that our method outperform other baselines in both simulation and real-world. The experimental details are in $\color{red}{section 4.3}$ and $\color{red}{section 4.4}$.

Reviewer comment (Evaluation):
There is no performance analysis of the method or baselines across different terrains, or across different command ranges or different command types, or across different command targets.

Author response:
We conduct performance analysis of our method and other baselines across different terrains, command ranges and command types. For our tasks, the only target is to track the velocity commands. You can checkout the details in $\color{red}{section 4.3}$ and $\color{red}{section 4.4}$ in current version.

Reviewer comment (Experiments):
This work claims contrastive learning is better than purely regression, but there is no experiment to support the claim.

Author response:
We also did extensive ablation studies ablation studies on  contrastive learning and regression. To be more specificly, we perform ablation study with the following methods:

* **Baseline**: Training the policy with no auxiliary task, but with a history encoder of the same architecture training as our method, the history encoder is trained with policy update.
* **Ours w/o Velocity Input**: Training the policy the same as our method, but set the position of velocity input to 0 for the policy network.
* **Ours w/o Velocity Loss**: Training the policy the same as our method, but without the loss of velocity estimation and set the position of velocity input to 0 for the policy network.
* **Ours w/o Latent Input**: Training the policy the same as our method, but set the position of latent input to 0 for the policy network.
* **Ours w/o Latent Loss**: Training the policy the same as our method, but without the loss of dynamics-aware estimation and set the position of latent input to 0 for the policy network.
* **Regression**: Training policy with velocity estimation and dynamics-aware estimation, but the auxiliary is to simply regress them in latent space.
* **Oracle**: Training policy with a history of full observations.

The training curve is in $\color{red}{section4.5}$, which can not be included in the comments due to the limitation of openreview. However, the real-world experiments(same as the comparision method) can show the improvement of our dynamics-aware modeling and contrastive learning method:
| Benchmarks | Environments | Metrics | Ours | Ours w/o  Vel. Inp. | Ours w/o  Vel. Loss | Ours w/o  Lat. Inp. | Ours w/o  Lat. Loss | Regression | Baseline |
| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| Stairs | Short-range  (A1) | Success rate  $(\mathrm{percent})$ | **100** | 85 | 80 | 50 | 55 | 85 | 10 |
|  | Long-range  (Aliengo) | Number  of stairs | **176.5 $\pm$ 7.81** | $51.45 \pm$ 25.67 | $54.6 \pm 24.68$ | $13.3 \pm 13.42$ | $10.6 \pm 9.68$ | $155.7 \pm 13.48$ | $0.0 \pm 0.0$ |
| Unseen  terrains | Compositional  Terrain  (Aliengo) | Success rate  $(\mathrm{percent})$ | **85** | 70 | 75 | 50 | 50 | 70 | 15 |
|  | Deformable Slope  (A1) | Success rate  $(\mathrm{percent})$ | **55** | 30 | 25 | 0 | 0 | 30 | 0 |
| Anti-  disturbance | Dragging  Obstacle  (A1) | Maximum  weight  $(\mathrm{Kg})$ | **10** | **10** | **10** | 5 | 5 | **10** | 4 |
|  | Vertical Hit  (A1) | Maximum  weight  $(\mathrm{Kg})$ | **8** | 7 | 6.5 | 4 | 4 | **8** | 5.5 |
|  | Payload  (A1) | Maximum  weight  (Kg) | **8** | 7 | 7 | 3.5 | 3.5 | **7.5** | 3 |
|  | Missing steps  (Aliengo) | Success rate  $(\mathrm{percent})$ | **100** | 90 | 90 | 5 | 0 | 65 | 0 |

Reviewer comment (Experiments):
More baselines need to be compared.

Author response:
We compared our method with the baselines including the regression method, RMA, MoB, Built-in MPC and ours w/o auxiliary loss, the experimental results show that our method outperform other baselines in both simulation and real-world. The experimental details are in $\color{red}{section 4.3}$ and $\color{red}{section 4.4}$.

Reviewer comment (Evaluation):
More specific performance analysis is required: performance across terrain or different physical properties.

Author response:
We conduct performance analysis of our method and other baselines across different terrains, command ranges and command types. For our tasks, the only target is to track the velocity commands. You can checkout the details in $\color{red}{section 4.3}$ and $\color{red}{section 4.4}$ in current version.

Reviewer comment (Writing):
The paper seems to be written in a rush with lots of typos and incorrect references.

Author response:
Thanks for your advice, we carefully checked the typos and incorrect reference in our paper, we have already corrected them in a the current version.

Reviewer comment (Novelty):
Seems to me the proposed model is more of a state estimation model (or a state compression model) instead of a dynamics model. For example, one would expect a dynamics model to be used to do rollouts instead of using a simulator during training. It will be nice to have some comments about it.

Author response:
Yes, our "dynamics model" is not the typical "dynamics model" or "world model" in Reinforcement Learning. We do not rollout on "dynamics model" to generate synthetic interaction data. Instead, the "dynamics model" is trained to help the policy understanding that under current state, if we do not take any action, how will the robot move. This in fact reflects the properties of the physical world, letting the policy determine how to react to achieve our command. More specifically, we treat the external environment as a kind of system disturbance(the impact of friction, resitution, terrain, etc), then use the "dynamics model" or "internal model" in our work to simulate the successor state of the system in order to estimate the potential disturbance. We think that "dynamics-aware model" should be a more accurate description, we already corrected this term in current version.

Reviewer comment (Experiments):
A key missing ablations is to compare the proposed contrastive learning with a regression model.

Author response:
We have done the ablation study between contrastive learning and regression, it shows that contrastive learning performs better, the superior performance comes from the utlization of batch level information and discriminative representation. Furthermore, we also conduct ablation studies on different variants of our method. You checkout the details in $\color{red}{section 4.5}$.

Reviewer comment (Experiments):
It would be nice to have more analysis of the learned latent model as this is the key contribution. For example, examining the effect of the number of prototypes.

Author response:
Thank you for your advice on experiments, we conduct the ablation on the number of prototypes in $\color{red}{section 4.6}$, we also investigate the influence of velocity estimation(both input and auxiliary loss) and latent (both input and auxiliary loss) in $\color{red}{section 4.5}$, which shows that the latent(dynamics-aware) representaion plays a main role in the performance.

Reviewer comment (Novelty):
There isn't a big delta from previous works, i.e. the ETH 2020 science paper also demonstrates similar capabilities.

Author response:
W1: Previous works[1, 2, 3] use a mimic learning framework to predict unavailable terrain information in latent space, which entails additional performance reductions.Our method directly treats different terrains as perturbations, and try to learn the best response under different observations with the help of critic module, which is able to sense terrain information. A vivid analogy is that previous work trains a dog that can see the terrain and then blindfolded it to achieve prior performance; the dog we trained has never seen the terrain, and learns to react directly to a variety of different situations.

Reviewer comment (Novelty):
While it is applaud to see good results on blind locomotion walking, without perception the policy is still fundamentally limited on difficult terrains. For example, the policy has to hit the vertical edges to sense the terrain before the hybrid internal model can "sense" the change in the environment. And there are already good perceptive locomotion works there.

Author response:
W2: Perceptive locomotion require to utilize a wealth of sensor information, such as depth cameras, ToF cameras or LIDAR, to capture height information about the surrounding terrain, which plays a crucial role. However, most of the sensors mentioned above are very expensive and fragile, and the vast majority of quadruped robots are not equipped with these sensors and can only use feedback information from a basic IMU and motors. For example, Mini-cheetah only equips IMU and joint encoder, and Unitree Go1's default depth camera is hard to tackle. Besides, our policy do not always hit the vertical edge, in most of the environments, it spontaneously use its front leg to sense the front terrain, the videos in demo shows this strategy.

Reviewer comment (Evaluation):
The experiment result section of the paper is thin. For example, there is no quantitative results on success rate etc.

Author response:
W3: We are very thankful for your advice on experiments, we have already conducted extensive experiments in current version, including simulation benchmarks $\color{red}{section 4.3}$, real-world benchmarks $\color{red}{section 4.4}$, ablation study $\color{red}{section 4.5}$ and influence of prototypes $\color{red}{section 4.6}$.

Reviewer comment (Experiments):
In the ablation study, the oracle policy seems learning slower than the proposed method, which is surprising.

Author response:
W4: We are sorry for the wrong setting of oracle, we only include the current oracle observation in the policy input rather than a history of oracle observations, which may decrease the performance and learning speed of oracle policy, the new curve is also included in the paper $\color{red}{section 4.5}$.

Reviewer comment (Reproducibility):
Consider publishing the code with the paper. Right now there is a supplementary pdf which is a duplication of the main paper.

Author response:
W5: We are working on refine and adding comments to our code, we will try our best to release the code as soon as possible. In the current revision, we add our training hyper parameters in the appendix, which can be used directly for training.

Reviewer comment (Reproducibility):
Is the hybrid internal model co-trained with the policy or separately?

Author response:
Q1: The hybrid internal model is co-trained with the policy. In each iteration, we first collect trajectories, then train the hybrid intrernal model with the proposed method, after that, we froze the hybrid internal model and update the actor-critic part. We refined the describtion in our paper $\color{red}{section 3.5}$