Paper: H-GAP: Humanoid Control with a Generalist Planner (ICLR-2024)

Reviewer comment (Reproducibility):
The provided link in the abstract results in 404 not found, and this submission comes without supplementary videos. This, unfortunately, makes it difficult to assess the naturalness of the generated human motion, which is critical to humanoid motion generation.

Author response:
We apologize that the website didn’t work during your review. It was due to a privacy setting that was not properly set. We fixed it as soon as we noticed the issue during the reviewing period. We can confirm it is working now.

This is a quick response to this particular issue and we'll address your other concerns and questions later this week (together with the response to other reviewers).

Reviewer comment (Evaluation):
The paper claims that H-GAP provides a basis for a foundation model for humanoid control. However, only the performance of imitation learning improves with the model scale. The performance of downstream tasks drops for larger models. In my view, this does not constitute a foundation model's basis because supporting downstream tasks is the most important feature of a foundation model.

Author response:
Thank you for your insights. We acknowledge that the term 'foundation model' might have suggested our model is larger than it is. To clarify, we view H-GAP as an initial step towards a foundational model for humanoid control, given its generalist nature and training on a diverse dataset. We've adjusted our wording to 'an early attempt to build a foundation model for humanoid control' for clarity.

The scaling properties highlighted in our paper are part of our empirical contributions, underscoring the importance of data scaling relative to model scaling. This finding suggests prioritizing data enhancement to advance the generalist approach in humanoid control. Additionally, we delve into the reasons behind the limited impact of model scaling on downstream task performance in Section 3.3, providing a deeper understanding of these dynamics.

We want to highlight that our findings do not conflict with other empirical findings in foundation models for other modalities because:
The trend of imitation learning performance and validation loss of H-GAP do align with the scaling law reported in studies on large language models (LLMs) [1, 2].

Large foundation models are usually trained on a huge and diverse internet-scale dataset. For example, the whole training process of LLMs is usually only done for one or two epochs. If the training data is very limited, downstream tasks are not guaranteed to be improved just by scaling the model [3, 4].


	[1] Hoffmann et al. “Training Compute-Optimal Large Language Model”. 2022
	[2] Henighan et al. “Scaling Laws for Autoregressive Generative Modeling”. 2022
	[3] Muennighoff et al. “Scaling Data-Constrained Language Models”. Neurips 2023
	[4] Radford et. al. Robust Speech Recognition via Large-Scale Weak Supervision, 2022

Reviewer comment (Evaluation):
What are the metrics used in table 1 and table 2?

Author response:
The metrics used in table 1 are the average episodic clip tracking returns. The task is set up similar to the multi-clip tracking task [1], where the reward function measures the discrepancy between the agent’s joint and body positions and the target configurations in a reference trajectory. We adapt our implementation from the ReferencePosesTask from the DeepMind Control Suite [2]. We have added more explanation to the numerical results in section 3.1 of the manuscript (highlighted red).

The metrics used in downstream control tasks (table 3 in the latest manuscript) are average episodic returns on downstream tasks. 

[1] Hasenclever et al. “CoMic: Complementary task learning & mimicry for reusable skills.” ICML 2020.
[2] Tunyasuvunakool et al. “dm_control: Software and tasks for continuous control” 2020.

Reviewer comment (Experiments):
I would assume that the optimal number of samples $N$ corresponds to the length of the intended trajectory $T$. What is the choice of $N$ for your experiments when $T=M=16$ and $L=4$?

Author response:
We use S=256 when reporting the results in downstream control tasks (table 3 in the latest manuscript).

Reviewer comment (Reproducibility):
Does the temperature change throughout algorithm 1? What are the choices of $\Upsilon$ and $\rho$?

Author response:
The temperature is fixed throughout algorithm 1. We use \upsilon=4 and \rho=0.99 when reporting the results in downstream control tasks (table 3 in the latest manuscript). We have included a table of the planning hyperparameters and additional ablation results in the appendix.

Reviewer comment (Evaluation):
The paper could analyze the coverage of different motion skills by randomly sampling initial states and generating rollouts from the model. This would help quantify the diversity of motions that can be produced [1].

Author response:
In response to your suggestion, we would appreciate further details regarding the specific distribution you have in mind for sampling.
Uniform or Gaussian sampling in a high-dimensional state space often leads to unnatural poses and ineffective rollouts. 
On the other hand, if you're referring to sampling from the MocapAct dataset, our imitation learning experiments serve as an indirect assessment of the model's ability to capture a broad spectrum of behaviors. Here, initial states are drawn from a variety of clips, and Gaussian action noise is added for enhanced variability. If your suggestion falls outside these approaches, we'd appreciate further clarification to better address your point.

Reviewer comment (Evaluation):
For analyzing imitation abilities, the model could be conditioned on a small segment of reference trajectories rather than just initial states. This would better evaluate how well it can leverage larger context sequences. The current approach in Table 1 mainly reveals modeling/memorization capabilities rather than imitation abilities.

Author response:
In addressing your suggestion for conditioning the model on reference trajectories, we interpret this in two ways: conditioning on either a longer historical context or a target future trajectory. Using a longer historical context could enhance imitation performance, but may reduce the steerability for downstream control tasks. Our focus with H-GAP is on generalist control, where imitation learning primarily serves to validate model accuracy.

Conditioning on a target future trajectory, while potentially improving imitation, diverges from the norm in imitation learning, where policies typically access only current observations, not future trajectories. We believe our current approach maintains a fair balance in this context.

Additionally, to counteract mere memorization by the agent, we introduce Gaussian noise into the action selection process, ensuring that the model's performance extends beyond simple replication of data.

Reviewer comment (Experiments):
Additional experiments could be designed to isolate the planning contributions of the model beyond just trajectory modeling. For example, a conditional generation task could be constructed using out-of-distribution initial states in order to see if the model is able to “catch-up” to imitate a trajectory from a different initial polse from the original trajectory.

Author response:
Thank you for your valuable suggestions. It appears that your concerns focus on whether H-GAP, coupled with Model Predictive Control (MPC), is only used for to imitation. To address this, we'd like to emphasize the diverse and perturbed nature of initial states used in our experiments, as detailed in Section 3.2.1. These states, often not directly aligned with the specific downstream task, necessitate adaptive behaviors from the agent. For instance, in the 'rotate y' task in this rollout video: , the agent starts from a forward-walking initial state and must adjust to execute the rotation, demonstrating the model's capability to “piece together behaviors" in a non-trivial manner.

Looking ahead, we are committed to exploring long-horizon, sparse-reward tasks to further probe our model's potential as you suggested. Our approach's strength in encoding compact, high-dimensional humanoid trajectories positions us well to tackle these more complex, long-horizon challenges, demonstrating the adaptability and broad applicability of our method.

Reviewer comment (Experiments):
More complex downstream tasks requiring long-term planning could be explored to go beyond trajectory imitation. The current tasks are limited to state distributions from the training data. New tasks like going to specified locations, or following a particular heading and and orientation [1] would require guiding the motion generation process. This could reveal how well the model can piece together behaviors.

Author response:
Thank you for your valuable suggestions. It appears that your concerns focus on whether H-GAP, coupled with Model Predictive Control (MPC), is only used for to imitation. To address this, we'd like to emphasize the diverse and perturbed nature of initial states used in our experiments, as detailed in Section 3.2.1. These states, often not directly aligned with the specific downstream task, necessitate adaptive behaviors from the agent. For instance, in the 'rotate y' task in this rollout video: , the agent starts from a forward-walking initial state and must adjust to execute the rotation, demonstrating the model's capability to “piece together behaviors" in a non-trivial manner.

Looking ahead, we are committed to exploring long-horizon, sparse-reward tasks to further probe our model's potential as you suggested. Our approach's strength in encoding compact, high-dimensional humanoid trajectories positions us well to tackle these more complex, long-horizon challenges, demonstrating the adaptability and broad applicability of our method.

Reviewer comment (Experiments):
Overall, evaluating the model on tasks and scenarios that require leveraging the full context sequence in a generative way would provide better insight into the planning abilities. The current experiments focus primarily on trajectory modeling in different flavors (imitation based Downstream tasks and imitation benchmark).

Author response:
Thank you for your valuable suggestions. It appears that your concerns focus on whether H-GAP, coupled with Model Predictive Control (MPC), is only used for to imitation. To address this, we'd like to emphasize the diverse and perturbed nature of initial states used in our experiments, as detailed in Section 3.2.1. These states, often not directly aligned with the specific downstream task, necessitate adaptive behaviors from the agent. For instance, in the 'rotate y' task in this rollout video: , the agent starts from a forward-walking initial state and must adjust to execute the rotation, demonstrating the model's capability to “piece together behaviors" in a non-trivial manner.

Looking ahead, we are committed to exploring long-horizon, sparse-reward tasks to further probe our model's potential as you suggested. Our approach's strength in encoding compact, high-dimensional humanoid trajectories positions us well to tackle these more complex, long-horizon challenges, demonstrating the adaptability and broad applicability of our method.