Paper: Universal Humanoid Motion Representations for Physics-Based Control (ICLR-2024)

Reviewer comment (Experiments):
One of this work’s claims is that previous work has limited coverage of the learned latent space that can not cover the wide spectrum of possible human motion. But in Table 1, there is no comparison with ASE or CALM or Imitate&Repurpose on motion imitation performance. Though ASE, or CALM might be a bit hard to compare, Imitate&Repurpose should be reasonable to compare with. I’m not expecting it to perform better, just for completeness.

Author response:
**Comparison with Baselines on Imitation**

As ASE and CALM do not use a motion-tracking objective for training, we cannot compare motion imitation results directly. We show that they do not have good motion skill coverage in the VR controller tracking task, which is a generalization and arguably harder task than full-body motion imitation. The VR controller tracking task only has partial body observations, but requires the policies to produce the same full-body motion as the full-body imitation task. Between PULSE and imitate \& repurpose, the main difference is how the latent space is trained, where PULSE is through distillation and imitate \& repurpose is through RL. Imitate \& repurpose also does not use a Gaussian prior or learnable prior, though both PULSE and Imitate \& repurpose use the AR1 prior. We report the result of training using RL and no distillation in Table 6 of Appendix C.2. We can see that training without distillation can learn a large amount of motion in AMASS (76\%), but it cannot scale to the performance of PULSE. We hypothesize that random sampling for variational bottleneck and random sampling for RL hampers the learning process. This ablation can be viewed as imitate \& repurpose implemented in our framework and humanoid, and we show that it does not scale as well in the imitation task.

Reviewer comment (Experiments):
Though PHC+ exhibit great performance on motion imitation, it’s comparison with PHC might be a bit unfair, since one of the modification fo PHC+ is “removing” some hard-negative in the dataset, provide motion imitation result on modified dataset might be beneficial.

Author response:
**Comparison between PHC and PHC+**

Both PHC and PHC+ are evaluated on the same set of modified training and testing sequences in Table 1.  After removing sequences with issues, there are 11313 and 138 AMASS motion sequences for training and testing, and we re-run all PHC's evaluations on the modified dataset.

Reviewer comment (Presentation):
Some content is a bit hard to read: In Figure 3, it’s quite hard to see the human motion in the Figure 3(e) row, and the title for each row is really hard to see. (Minor issue)

Author response:
**Plot Content Visibility**

We have updated the plot for better visibility. Thanks!

Reviewer comment (Experiments):
Is the proposed method robust against methodology and dynamics changes? It would be interesting to see these results and potentially enable appling the proposed method to humanoid robots.

Author response:
**Answer to Questions**

- As sim2real from physics simulation [1, 2, 3, 4] continues work, we believe that a similar methodology should transfer to real humanoids. Although modifications to the humanoid kinematic tree, reward designs, and domain randomization will be required, the methodology should be applicable. PHC has also demonstrated motion imitation on diverse body shapes, which could also be distilled to a shape/dynamics-aware latent space. 



[1] Cheng, Xuxin, Kexin Shi, Ananye Agarwal, and Deepak Pathak. 2023. “Extreme Parkour with Legged Robots.” arXiv [Cs.RO]. arXiv. http://arxiv.org/abs/2309.14341.

[2] Zhuang, Ziwen, Zipeng Fu, Jianren Wang, Christopher Atkeson, Soeren Schwertfeger, Chelsea Finn, and Hang Zhao. 2023. “Robot Parkour Learning.” arXiv [Cs.RO]. arXiv. http://arxiv.org/abs/2309.05665.

[3] Fu, Zipeng, Ashish Kumar, Ananye Agarwal, Haozhi Qi, Jitendra Malik, and Deepak Pathak. 2021. “Coupling Vision and Proprioception for Navigation of Legged Robots.” http://arxiv.org/abs/2112.02094.

[4] Radosavovic, Ilija, Tete Xiao, Bike Zhang, Trevor Darrell, Jitendra Malik, and Koushil Sreenath. 2023. “Learning Humanoid Locomotion with Transformers.” arXiv [Cs.RO], March. https://doi.org/10.48550/ARXIV.2303.03381.

Reviewer comment (Evaluation):
Quantitative results on the VR-controller tracking task are a little bit disappointing compared to Scratch

Author response:
**VR Motion Tracking Results**

We acknowledge that the quantitative result on VR motion tracking is less performant on joint position errors though better on success rate. Similar performance degradation on joint errors can also be observed in Table 1, where after distillation with the variational latent space, the imitation performance is worse than the teacher's. This is a known trade-off between prior regularization and achieving good reconstruction performance in VAEs, where the reconstruction error usually does not reach 0 during training. As shown in Appendix C.3, if a non-variational latent space is used (e.g. spherical), a better distillation performance can be achieved (100\% success rate and 28.1 $E\_\text{mpjpe}$). However, such a latent space does not provide a variational prior, and random samples from it do not generate coherent motion. Thus, while our latent space has good coverage of the motor skills required to perform AMASS, it is not full coverage. Downstream task such as VR motion tracking is searching in a constrained solution space and can lose performance on precise motion tracking (though gains faster convergence). Future investigation is needed to achieve better motion imitation results while retaining the generative capabilities of the latent space.

Reviewer comment (Presentation):
The references section could be cleaned up and harmonised, notably for publication conferences

Author response:
The typos and the reference section are cleaned up in the revision. Thanks!

Reviewer comment (Writing):
Although the writing is clear, some typos remain (e.g. Guassian)

Author response:
The typos and the reference section are cleaned up in the revision. Thanks!

Reviewer comment (Evaluation):
In section 4.3, a failure case is described using the prior R but it is unclear to me how do we recover from it apart from restarting the task's policy learning? Does it happen often?

Author response:
**Failure Case not Using Prior R**

We believe that this refers to the cases "one can also supervise $\pi\_{\text{task}}$'s distribution using prior $\mathcal{R}$", which leads to an "adverse feedback loop". We apologize for the confusing statement. Here, we are referring to an alternative to using the residual action formulation. In Sec. 4.3,  we propose to let downstream policy produce a residual with respect to the learned prior $\boldsymbol{a}\_t = \mathcal{D}(\pi\_{\text{task}}(\boldsymbol{z}\_t | \boldsymbol{s}^{p}\_t, \boldsymbol{s}^{g}\_t) + \boldsymbol{\mu}^{p}\_t)$. The alternative is to directly output the latent code rather than the residual, and supervise the output distribution with the prior $\mathcal{R}$. We **do not use** this alternative due to the described issue. The paragraph is included as a justification for the residual action formulation. This failure case only applies to the alternative approach and does not apply to the residual action one.

Reviewer comment (Experiments):
No comparison with baselines that resemble the proposed method, e.g., Physics-based character controllers using conditional VAEs, by Won et al or ControlVAE: Model-based learning of generative controllers for physics-based characters. by Yao et al, which also uses VAEs.

Author response:
We acknowledge that PhysicsVAE [1] and ControlVAE [2] and ours all strive to learn a compact latent space using a variational bottleneck. While our structure resembles a VAE, we do not use auto-encoding/reconstruction as a learning objective. Our decoder outputs actions $\boldsymbol{a}\_t$ instead of the input state $\boldsymbol{s}\_t$. As a result, we do not require learning any world model during training. This design allows us to directly distill motor skills from a pretrained imitator that has achieved strong performance on a large-scale (40 hours) motion dataset. 

PhysicsVAE[1] has tested on the CMU dataset (a subset of AMASS) to further understand its scalability. They "observed that the model learned from the CMU dataset does not perform well when compared to other models learned from either the locomotion dataset or the dancing dataset" (Sec.6, third paragraph, page 11 in PhysicsVAE). The learned latent space from PhysicsVAE also requires an additional adaption layer to solve some of the harder tasks such as terrain traversal, which PULSE does not require. 

Similarly, ControlVAE [2] stated that "experiment on a large-scale dataset composed of 3.3 hours of motions" "converges to a lower reward", and the "learned skill embedding can still recover input motion with a larger visual discrepancy, but the performance on the downstream task are significance degrenrated" (Sec.5.7, last paragraph, page 14 in PhysicsVAE). 

Compared to PhysicsVAE and ControlVAE, we learn from a much larger dataset and demonstrate that our latent space can (1) scale to the motion dataset after distillation, (2) perform well for downstream tasks without any additional adaptations. PhysicsVAE and ControlVAE also use different humanoids and simulation environments (Bullet and ODE) from ours (SMPL humanoid and Isaac Gym), and the gap between environment and humanoid choices hinders the reimplementation and fair benchmarking.

Reviewer comment (Experiments):
The main difference between the proposed system and other systems is that the proposed system is able to scale to the whole AMASS dataset while other systems are mainly doing locomotion or something similar. However, this is not well demonstrated in the downstream tasks, which are mostly just comprised of locomotion tasks in addition to some simple reaching, which the other systems can already do pretty well (maybe less efficiently?).

Author response:
Our main motivation for scaling to AMASS is to have a latent space that covers a wide range of human motion. Thus, we include the VR controller tracking task, which requires tracking three 6DOF points in the global space. This is a generalized version of the full-body motion tracking task, where only partial observation is given to the humanoid to create full-body motion. This task requires the latent space to contain motor skills to match **arbitrary** user input in a continuous and streaming fashion. A latent space that has only seen small-scale locomotion datasets would not have the motor skills required to perform these free-form motions. Our supplement video (Sec. Motion Tracking Downstream Task) shows that the controller that uses our latent space can handle motions such as squatting, punching, dancing, running, rolling, etc. We also quantitatively demonstrate the tracking performance on the AMASS dataset in Table 2. Notice that the joint errors reported in Table 2 are **full-body** joint errors, and a low value means that the full-body motion generated from three-point input is close to the ground truth. Table 2 and Fig. 5 show that using our latent space, one can efficiently solve this task and achieve a high success rate on training, testing, and real-world data capture. Previously proposed latent spaces (ASE and CALM), when also trained with the AMASS dataset, do not obtain the coverage demonstrated in our latent space. We also note that few prior latent space methods have attempted a motion-tracking task due to the diversity and coverage required for free-form motion tracking. 

On generative tasks, we test simple ones ranging from reach, speed, and strike, but also challenging ones like complex terrain traversal. The terrain traversal task requires the humanoid to handle stairs, slopes, rough surfaces, and obstacles. Prior art  [3]  on this task has relied on GAIL to achieve natural motion. Using our latent space, we no longer require GAIL but can still learn human-like behavior. Note that previous latent space approaches either fail on a stairs-only environment [4] or require an adaptation layer to traverse rough terrains/slopes [1], while our latent space can be used to solve the task without modification, thanks to the wider range of motor skills it can draw upon.

Reviewer comment (Experiments):
It will be nice to showcase some scenarios where the benefit of learning the whole AMASS dataset is useful.

Author response:
We design the VR motion tracking task to show that our learned latent space can cover a large amount of motor skills. A controller using our latent space is able to achieve good full-body motion tracking performance on the AMASS train and test and also transfers well to real-world data captures using commercial VR headsets. If the latent space has only been trained on a handful of locomotion sequences, it will only have the skills to perform simple actions. Our result on complex terrain traversal also shows that scaling matters, where the humanoid will employ skills like jumping when traveling fast on stairs.

Reviewer comment (Evaluation):
What are the motions that PHC cannot handle?

Author response:
We will include failure cases for PHC+ from the AMASS test set and other data sources. In general, while PHC can imitate a large number of motion sequences, unseen highly dynamic ones (e.g. badminton jump smash) or noise motion estimated from videos can still cause the humanoid to fall.

Reviewer comment (Evaluation):
I feel like the motion quality produced in the downstream tasks is suboptimal/unnatural. It will be nice to have a metric to measure the motion quality generated in the downstream task for potential future improvement for future work.

Author response:
We agree that motion naturalness can be hard to evaluate and subjective, and we included a large number of video results for qualitative evaluation. The motion quality of a generative method is hard to quantify since there is no ground truth to compare against. Thus, we follow the prior art in the field and provide reward comparisons and video demonstrations. For the VR tracking task, we include pose estimation metrics, which are quantifiable ways to judge the quality of estimated motion. For unnaturalness in motion, we acknowledge that PULSE does not strictly constrain itself to certain types of motion when solving tasks. Since we train on a large-scale motion dataset, certain styles of motion that may be deemed unnatural for the speed task are also available in the latent space. For instance,  AMASS contains motions for "running fanatically" or "limping", whose motor skills are also learned in the latent space and can be used to solve the speed task. While training from scratch will often result in inhuman behavior, PULSE solves tasks using human-like behavior even when there are no constraints on the strategies used. To further constrain the solution space to natural behavior defined for certain tasks, one can enforce symmetry or incorporate additional regularizations.

Reviewer comment (Evaluation):
For the speed task (and other tasks as well, but speed task is one of the results in the ASE paper, so I will focus on this), looks like the motion quality of ASE is really bad, while the original ASE paper has pretty good motion quality (at least visually), any comment on the discrepancy?

Author response:
The discrepancy on baselines (ASE and CALM) mostly comes from the training data scale. For a fair comparison, all baselines are trained on the AMASS training set, the same as PULSE. The original ASE is trained on a specialized dataset that only contains locomotion or character movement (eg. strike). By construction, a latent space trained on specialized datasets will only include motor skills that are deemed natural for the downstream task. However, datasets like AMASS contain diverse full-body motion that are not specialized. Thus, motor skills for sequences such as "crouch walking" or "running fanatically" could also be learned in the latent space of ASE. When using these motor skills for downstream tasks, the resultant motion can appear unnatural, though still human-like (in the sense that a human can recreate these motions if desired).