Paper: HumanTOMATO: Text-aligned Whole-body Motion Generation (ICLR-2024)

Reviewer comment (Writing):
In section 2.1, $F$ should be $L$?

Author response:
- Q2: minor issues.
- A2: We apologize for these issues. We have revised them in the revision. Please check it.

Reviewer comment (Writing):
In section 2.2, the sentence is repeated twice:  
> In the encoding phase, we input hand and body motions, yielding hand and body tokens through the hand encoder EncH(·) and the body encoder EncB(·), respectively.

Author response:
- Q2: minor issues.
- A2: We apologize for these issues. We have revised them in the revision. Please check it.

Reviewer comment (Writing):
In Appendix C.1, Algorithm 1, line 4 in the for loop, $\hat{z}^B=..., \hat{z}^B));\mathcal{C}^B);$ the second $\hat{z}^B$ should be $z^B$.

Author response:
- Q2: minor issues.
- A2: We apologize for these issues. We have revised them in the revision. Please check it.

Reviewer comment (Writing):
What’s the reason behind the name hierarchical-GPT? The model seems to interleave instead of hierarchical to me.

Author response:
- Q3: What’s the reason behind the name hierarchical-GPT? The model seems to interleave instead of hierarchical to me.
- A3: Based on the Forward Kinematics (FK) and the skeleton tree, body motions can be treated as the next hierarchy of body motions. Based on these, we name it Hierarchical-GPT.

Reviewer comment (Theory):
Why is it that in H2VQ, the hand is encoded before the body, and in Hierarchical-GPT, the body token is predicted before the hand? Is there any reason besides empirical performance advantages?

Author response:
- Q4: Why is it that in H$^2$VQ, the hand is encoded before the body, and in Hierarchical-GPT, the body token is predicted before the hand? Is there any reason besides empirical performance advantages?
- A4: Similar to Q3, based on the FK and the skeleton tree, hand motions are highly dependent on body motions. This is the intuitive insight of this design.

Reviewer comment (Evaluation):
To what extent can the method generalize to OOD text descriptions?

Author response:
- Q5: To what extent can the method generalize to OOD text descriptions?
- A5: We greatly appreciate your insightful suggestions to help us explore the boundaries of model performance. We provide some OOD text testing, whose results are shown in the supplementary (`supplementary/OOD/`).
    - We test the robustness of different subject descriptions, like "the guy", "the woman".
    - We also test the robustness of different tenses, like simple tense and continuous tense.
    - We test some of the novel motion descriptions, which are few or do not appear in the dataset. Both success and failure cases are provided. **We find some cases interesting to share.** (1) For the "playing basketball" case, the generated results are about a man who is dunking. (2) For the "a man is sleeping" and "a man is sleeping on the ground" cases, sometimes it succeeds, sometimes it fails. Note that there are no cases describing the sleeping motion directly. The success case is similar to "the person is lying on the ground.", which we think is reasonable. The failure case is a swimming motion. We analyze this is because swimming and lying are similar motions. We also analyze the swimming and lying motions in the retrieval demo (`supplementary/retrieval-demo.mp4`).

Reviewer comment (Experiments):
In the context of vector quantization, the authors integrate the hand pose vector quantization with the body pose using a hierarchical strategy rather than directly quantizing the whole body pose.

Author response:
We sincerely acknowledge the discussion about the novelty of our work. We would like to clarify as follows.
    - For the VQ part, we explore how to improve the generation quality efficiently by enhancing reconstruction precision. However, increasing the codebook size is almost in vain. Therefore, we propose the H$^2$VQ to replace the traditional VQ, which benefits the whole-body motion generation task significantly (See Table 2 and Appendix G). Moreover, to verify our hierarchical modeling insight, we adapt our H$^2$VQ as RVQ to the body-only motion generation. The results verify that this modeling strategy also benefits body-only motion generation.

Reviewer comment (Experiments):
The T2M-GPT has been modified to decode the body pose code and hand pose code alternately, rather than directly outputting the whole body pose code.

Author response:
We sincerely acknowledge the discussion about the novelty of our work. We would like to clarify as follows.
    - For the Hierarchical-GPT contribution, as there are two (hierarchical) codebooks in the H$^2$VQ, it is also necessary to design the corresponding hierarchical token prediction mechanism.

Reviewer comment (Novelty):
The utilization of TMR for encoding textual descriptions is a more intelligent choice compared to CLIP embedding, and the incorporation of motion-text alignment supervision appears to be beneficial during training; however, text-motion alignment has been employed in various previous works (e.g., TEMOS), and the proposed TMR merely adopts a contrastive learning approach through a retrieval target.

Author response:
TEMOS only learns the alignment between positive pairs. However, TMR also learns the misalignment between positive pairs. TEMOS is not good enough for alignment. To verify the good alignment of TMR, we compare its retrieval ability with TEMOS As shown in the table the TMR enjoys a good alignment between texts and motions by the contrastive training objective, which makes it with a larger margin than TEMOS in retrieval. This good retrieval ability provides a better alignment of two modalities, and provide a better motion-text alignment for motion generation. Details are shown in Appendix D.3.
    
    
    | Protocol | R@1 (T2M) | R@2 (T2M) | R@3 (T2M) | R@5 (T2M) | R@10 (T2M) | R@1 (M2T) | R@2 (M2T) | R@3 (M2T) | R@5 (M2T) | R@10 (M2T) |
    | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
    | Protocol A (TEMOS) | 0.034 | 0.062 | 0.082 | 0.117 | 0.178 | 0.034 | 0.067 | 0.096 | 0.137 | 0.204 |
    | Protocol A (TMR) | 0.051 | 0.098 | 0.131 | 0.192 | 0.301 | 0.066 | 0.118 | 0.163 | 0.233 | 0.350 |
    | Protocol B (TEMOS) | 0.085 | 0.143 | 0.190 | 0.237 | 0.308 | 0.112 | 0.135 | 0.155 | 0.190 | 0.246 |
    | Protocol B (TMR) | 0.089 | 0.152 | 0.194 | 0.273 | 0.401 | 0.169 | 0.205 | 0.238 | 0.298 | 0.395 |
    | Protocol C (TEMOS) | 0.233 | 0.341 | 0.412 | 0.492 | 0.601 | 0.263 | 0.360 | 0.426 | 0.504 | 0.614 |
    | Protocol C (TMR) | 0.445 | 0.609 | 0.700 | 0.799 | 0.883 | 0.407 | 0.578 | 0.673 | 0.795 | 0.883 |
    | Protocol D (TEMOS) | 0.502 | 0.641 | 0.717 | 0.802 | 0.897 | 0.528 | 0.654 | 0.725 | 0.807 | 0.907 |
    | Protocol D (TMR) | 0.716 | 0.854 | 0.907 | 0.946 | 0.977 | 0.771 | 0.893 | 0.938 | 0.968 | 0.985 |

Reviewer comment (Novelty):
While the paper presents a substantial number of contributions, the technical designs lack novelty and a certain level of appeal from the reviewer’s perspective, leading to a recommendation of marginally above acceptance.

Author response:
We sincerely acknowledge the discussion about the novelty of our work. We would like to clarify as follows.
    - For the VQ part, we explore how to improve the generation quality efficiently by enhancing reconstruction precision. However, increasing the codebook size is almost in vain. Therefore, we propose the H$^2$VQ to replace the traditional VQ, which benefits the whole-body motion generation task significantly (See Table 2 and Appendix G). Moreover, to verify our hierarchical modeling insight, we adapt our H$^2$VQ as RVQ to the body-only motion generation. The results verify that this modeling strategy also benefits body-only motion generation.
    - For the Hierarchical-GPT contribution, as there are two (hierarchical) codebooks in the H$^2$VQ, it is also necessary to design the corresponding hierarchical token prediction mechanism.
    - For the TMR, our core contribution is not how to train a TMR, but how to use a TMR. Since April 2022, MotionCLIP has introduced the CLIP to generate motion. After, a lot of follow-up research (like MDM, MotionDiffuse, T2M-GPT, and MLD) took the CLIP as a text encoder and regarded it as a default setting. **However, the CLIP cannot include any dynamic clues of motion as it is aligned with images. The text2motion community seems not to realize the problem with using CLIP. We revisit the problem in this work and well discuss how to use TMR.** The results of our method benefit the motion generation and text-motion alignment significantly (see Table 3). We believe this will provide some new understanding to the community.
    
    We claim these are something valuable to peers and the community, which we think is novel [1]. 
    
    [1]: Micahel J. Black, Novelty in Science. https://perceiving-systems.blog/en/news/novelty-in-science.

Reviewer comment (Writing):
On page 3, where it mentions, "where $F$ and $d$ denote the number of frames and the dimension ...", it may be advisable to replace $F$ with $L$ to maintain consistent notation.

Author response:
We apologize for these issues. We have revised them in the revision. Please check it.

Reviewer comment (Novelty):
The technical contribution of this paper seems somewhat limited in the following: (1) the pipeline of the method is similar to the T2M-GPT where a VQ-VAE is used for motion reconstruction and a transformer-based GPT model is used for motion generation, while the tasks are different; (2) the pretraining of a motion encoder and a text encoder via aligning text and motion in a contrastive way is also not new such as TMR, and further using it to replace the clip is natural in the prediction stage.

Author response:
- Q1: About technical contributions.
- A1: Thanks for discussing our technical contributions. We would like to discuss them from two aspects.
    - About `similar to T2M-GPT` concern. The two-stage training pipeline (VQ-VAE + GPT prediction) in T2M-GPT is almost nothing new. The training pipeline has been widely explored by the vision community (Taming Transformer [1], Video GPT[2], DALL-E-1 [3]). T2M-GPT contributes to the motion community by introducing this fashion to motion generation and verifying its effectiveness. Therefore, we take this effective fashion as a base to generate motion. Our core contribution is (1) to propose a more effective VQ for whole-body motion generation; and (2) to explore how to use language prior properly and which kind of language prior is more proper. We think these two contributions will help the community to generate human motions.
    - For the H$^2$VQ design, we carefully designed it for our novel text-to-whole-body-motion task with hierarchical body-hand modeling, which reflects our key insights on the task.
    - For the TMR part, in April 2022, MotionCLIP introduced the CLIP to generate motion. After that, a lot of follow-up research (like MDM, MotionDiffuse, T2M-GPT, and MLD) took the CLIP as a text encoder and regarded it as a default setting. In this work, we revisit how to introduce better language prior to motion generation. We think our answer on **how to use** T-M-aligned language prior is essential to the community for the first time.

[1]: Esser et.al. Taming transformers for high-resolution image synthesis. *CVPR,* 2021.

[2]: Yan et.al. VideoGPT: Video Generation using VQ-VAE and Transformers. 2021.

[3]: Ramesh et.a. Zero-Shot Text-to-Image Generation. ICML, 2021.

Reviewer comment (Presentation):
From the example of the visualizations, the textual description for facial motions focused on emotion (like happily, angrily), but the generated face shown in the paper is static, rather than dynamic motions, which lacks the demonstration of the emotion dynamics.

Author response:
- Q2: About facial motion dynamics.
- A2: Thanks for your suggestions. In the paper, it is hard to present the dynamics of face motions. We provide more generated face motion videos in the supplementary. Please check it.

Reviewer comment (Experiments):
In Table 2, about the H2VQ, what is the size of the codebooks for reconstructing the body and hand motion? Besides, for vanilla VQ-VAE and RVQ, do you separately model the hand and body motions and then combine the motion as body-hand motions? How is the performance for VQ-VAE and RVQ when increasing from 1024 to 4096?

Author response:
- Q3: About the H$^2$VQ, what is the size of the codebooks for reconstructing the body and hand motion?
- A3: The codebook size of $Q_1$ and $Q_2$ are both 512.

---

- Q4: For vanilla VQ-VAE and RVQ, do you separately model the hand and body motions and then combine the motion as body-hand motions?
- A4: No. For vanilla VQ and RVQ, we take the body-hand motions as a whole during training. We provide an ablation on whether to model them separately in the following table. We compare the H$^2$VQ with 2 separate VQs. The generated results show that modeling body and hands separately will result in a large performance loss in whole-body motion generation. We also update this in Appendix J.
    
    
    |  | FID | TMR_top1 (256) | TMR_top2_RP (256) | TMR_top3_RP (256) | top1_RP (32) | top2_RP (32) | top3_RP (32) |
    | --- | --- | --- | --- | --- | --- | --- | --- |
    | separately | 2.209±0.047 | 0.306±0.003 | 0.459±0.002 | 0.552±0.002 | 0.359±0.002 | 0.551± 0.003 | 0.666±0.002 |
    | Ours | **1.174±0.015** | **0.399±0.000** | **0.555±0.005**  | **0.638±0.004** | **0.416±0.009**  | **0.603±0.007**  | **0.703±0.007** |

---

- Q5: How is the performance for VQ-VAE and RVQ when increasing from 1024 to 4096?
- A5: We conduct the experiments on the size of VQ, RVQ, and H$^2$VQ. Results are shown in the following table (on MPJPE metric). As shown in the table, the scaling of the codebook size is almost in vain. This observation supports the basic intuition on the designing of H$^2$VQ. We provide a detailed explanation in Appendix G.2.
    
    
    | Size | 512 | 1024 | 4096 |
    | --- | --- | --- | --- |
    | VQ | 140.7 | 139.3 | 134.2 |
    | RVQ | 110.9 | 111.2 | 116.8 |
    | H$^2$VQ | 93.0 | 83.9 | 84.9 |

Reviewer comment (Reproducibility):
Regarding facial generation, what the text embedding is used? Clip or TMR?

Author response:
- Q6: Regarding facial generation, what the text embedding is used? Clip or TMR?
- A6: None of both. We discuss the model in Appendix B.4 and the text encoder comes up with a pre-trained DistillBERT and a 6-layer transformer.

Reviewer comment (Evaluation):
Since the hand & body, and face motions are separately modeled, how is the coherence of the generated motions?

Author response:
- Q7: Since the hand & body, and face motions are separately modeled, how is the coherence of the generated motions?
- A7: We acknowledge that the coherence issue. The text2motion is a kind of one-to-many mapping and the face motion is not highly coherent with other motions. Besides, the face motions is more highly related to the speeches [1, 2]. Therefore, we treat them as highly disentangled and model them respectively. This fashion of motion modeling is also been used in previous research [3]. Moreover, the face motion is represented in expression parameters, which is different from skeleton-based motions. This makes it hard to build an all-in-one system.

    
    [1]: Richard, Alexander, et al. "Meshtalk: 3d face animation from speech using cross-modality disentanglement." *Proceedings of the IEEE/CVF International Conference on Computer Vision*. 2021.
    
    [2]: Fan, Yingruo, et al. "Faceformer: Speech-driven 3d facial animation with transformers." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2022.
    
    [3]: Habibie, Ikhsanul, et al. "Learning speech-driven 3d conversational gestures from video." *Proceedings of the 21st ACM International Conference on Intelligent Virtual Agents*. 2021.

Reviewer comment (Presentation):
The title '3.1.2 Evaluation' somehow is easily misleading. In this section, you introduced the evaluation metrics and compared methods. How about changing to 'Evaluation Details'?

Author response:
We sincerely appreciate your review of our contribution to the community and acknowledge for helping us polish this work. We promise to make all codes public if accepted. Additionally, wishing our TMR would play a similar role in the motion generation community like the CLIP in the vision-language area, we would release the training codes, pre-trained models, and retrieval demos if accepted.

- Q1: How about changing into 'Evaluation Details'?
- A1: We acknowledge your suggestion. We change the ‘Evaluation’ to ‘Evaluation Details’. Please check the revision of the manuscript.

Reviewer comment (Experiments):
I feel that the focus of this article is on how to generate physical movements in the hands. The discussion about facial cVAE is limited, and I have not found any experiments to analyze this module.

Author response:
We sincerely appreciate your review of our contribution to the community and acknowledge for helping us polish this work. We promise to make all codes public if accepted. Additionally, wishing our TMR would play a similar role in the motion generation community like the CLIP in the vision-language area, we would release the training codes, pre-trained models, and retrieval demos if accepted.

---
- Q2: discussion about facial cVAE.
- A2: We provide more visualization results in the supplementary, where some success cases and failure cases are well discussed. Due to the space limitation, we discuss the facial cVAE in Appendix F.1 (Table 9).

Reviewer comment (Evaluation):
From visual results, I noticed some physically implausible artifacts, such as foot sliding. Can you give some discussion on this point?

Author response:
We sincerely appreciate your review of our contribution to the community and acknowledge for helping us polish this work. We promise to make all codes public if accepted. Additionally, wishing our TMR would play a similar role in the motion generation community like the CLIP in the vision-language area, we would release the training codes, pre-trained models, and retrieval demos if accepted.

---
- Q4: Discussion on physically implausible artifact.
- A4: Thanks for the discussion on the physically implausible artifact. This mainly comes from three aspects.
    - Firstly, most text2motion efforts proposed by the community are hard to generate physically plausible motions [1]. Generating physically plausible motions requires post-processing in a simulation environment.
    - Secondly, whole-body motion generation is much more challenging to generate than previous body-only motion generation. The number of joints scales from 24 to 52, making generation more challenging. Notably, adapting the motion from body-only motion to whole-body motion following the motion representation setting in [2] also makes the model hard to learn due to the high-dimensional data. Therefore, we even explore which kind of motion representation is good for whole-body motion generation. The results are discussed in Appendix B.1, which shows the representation w/o rotations is a more efficient and effective motion representation. The exploration of motion representation might be helpful to the community and it is our efforts to explore how to generate realistic motions.

[1]: Yuan et.al., Physdiff: Physics-guided human motion diffusion model. ICCV, 2023

[2]: Guo et.al., Generating diverse and natural 3d human motions from text. CVPR, 2022.