Paper: BRIDGING THE GAP BETWEEN HUMAN MOTION AND ACTION SEMANTICS VIA KINEMATIC PHRASES (ICLR-2024)

Reviewer comment (Evaluation):
I find the lack of more analysis on current models and their failure modes on the KPG benchmark a missed opportunity. What are the main failure modes these methods are failing on and which part is the proposed method winning at? The analysis could provide far more valuable insight into the current human motion generation community, while the FID and diversity tell very little.

Author response:
1. **Analysis of failure modes.** In Appendix F.3.3 and Figure 10, we include a typical user-studied KPG failure case of current models. where redundant motion is generated. To demonstrate more, please refer to the general response part 2.

Reviewer comment (Experiments):
Are all the models (MDM/MotionGPT/etc.) trained with the same data? Is the KP-guided joint latent space trained on all the data in Table 1 or similar to prior methods?

Author response:
3. **Training data.** In Tab. 3, all methods with * are trained on HumanML3D only, including the KP-guided latent space. The rest is trained with KPB.

Reviewer comment (Reproducibility):
I do find the proposed method the weaker link in the submission. Not enough information and intuition are provided for the proposed method and no supplemental materials are provided. How is the alignment done? How does $D_m$ and $D_p$ take an arbitrary combination of $z_m$ and $z_p$ as input? How is the interpolation done based on the output latent codes?

Author response:
4. **Details of the adopted method.** Our model is mostly designed for a fast and lightweight effectiveness evaluation of KP.  
The results show that even by simply incorporating KP with a simple VAE structure, considerable improvements are achieved.

5. **Alignment.** For alignment, the losses in Eq. 4 and Eq. 6-7 are designed to align the motion latent space with the KP latent space.

6. **Decoder inputs.** During training, as the reconstruction loss in Eq. 4, an arbitrary combination of $z_m$ and $z_p$ are sent into the decoders, resulting in the flexible input of the decoders.

7. **Interpolation details.** Sorry for the potential ambiguity. The interpolation is not conducted directly via manipulation of the latent codes. Instead, we extract latent code from the corrupted motion and KP sequences. With self-supervised learning, our model is capable of reconstructing the complete KP sequence from the latent code extracted from the corrupted KP sequence. With the reconstructed KP sequence, we further feed it into the encoder for a latent with complete information, then reconstruct the whole motion with the motion decoder.

Reviewer comment (Evaluation):
I am willing to raise my score if more analysis on the current models and their failure modes on the KPG benchmark could be provided, as well as some more insight into the qualitative performance of the proposed method.

Author response:
1. **Analysis of failure modes**. Thanks a lot for your advice. We further provide detailed KPG results as shown in the following table. Most methods provide accurate motions for the relatively simpler PP, PDP, LAP, and GVP. However, T2M-GPT occasionally failed in some rather simple cases. The major challenge is the PRPP and LOP-related prompts. We identify two reasons. First, the temporal composition of KPs increases the difficulty. Second, some prompts are easy for humans but hard for current models, like the visualized "right shoulder is in front of left foot, then behind left foot". The corresponding table and descriptions are updated in Appendix F.6.
| Method | PP         | PRPP      | PDP        | LAP        | LOP       | GVP        | Overall   |
|---------|------------|-----------|------------|------------|-----------|------------|-----------|
| MDM     | **100.00** | 21.24     | **100.00** | **100.00** | 12.12     | **100.00** | 44.40     |
| MLD     | **100.00** | 21.43     | **100.00** | **100.00** | 15.15     | **100.00** | 44.76     |
| T2M-GPT | **100.00** | 26.45     | 98.77      | 87.50      | 19.70     | 83.33      | 47.86     |
| Ours    | **100.00** | **32.05** | **100.00** | **100.00** | **24.24** | **100.00** | **52.14** |

Reviewer comment (Novelty):
The claim that “Kinematic Phrases” bridges the gap between human motion and action is over-selling the paper: one could imagine a person either (1) slightly hammering (slight up-down motion of wrist) or (2) swiping through a book (slight left-right motion of wrist) while standing perfectly still and the proposed KP would not pick up the difference due to the small motion - this has also been acknowledged by the authors in the Discussion.

Author response:
**The claim of "bridging the gap"**. First, we believe that we should not let the perfect be the enemy of the good. Despite the limited granularity in the current version, it could be easily extended with the proposed framework. Second, we argue that the target of KP is not fully replacing language descriptions for action semantics representation. In essence, KP is designed as an **automatic symbolization system** for motion. We expect KP to become an interface for motion-semantics mapping, where the major burden of understanding KP is left to the machines, while humans could interfere with the process as assistance, given the easily interpretable atomic Phrases. Thus, humans are not required to be able to generate KP sequences from scratch.

Reviewer comment (Experiments):
Some information is missing, for example: In 3.1. the authors define various sets of Kinematic Phrases (36 PPs, 242 PRPPs, 81 PDPs, etc) - where do those numbers come from?

Author response:
**The number of Phrases**. Here we list the details for KP filtering. These will be updated in the appendix.
    - Position Phrase (PP). It is composed of 14 joints (except the pelvis and both eyes) and 3 reference vectors with originally 42 Phrases. Since there are joints that define the reference vectors (e.g. hips for the leftward vector). Specifically, shoulders and hips are excluded for the leftward vector, and hips are excluded for the upward vector, resulting in 42-6=36 PPs.
    - Pairwise Relative Position Phrase (PRPP). It is composed of 136 joint pairs and 3 reference vectors with originally 408 Phrases. 24 joint pairs that are linked by limbs are filtered out. 30 Eye-related pairs are filtered out except (for the left eye, and right eye) due to the others are covered by head-related pairs. 4 triplets that the relationship barely changes along the reference vector (e.g., right knee, left hip, and leftward vector) are filtered out. These result in 408-24*3-30*3-4=242 PRPPs.
    - Pairwise Distance Phrase (PDP). It is composed of 105 joint pairs (except for both eyes). 24 pairs that are linked by the human body are filtered out, resulting in 81 PDPs.
    - Limb Angle Phrase (LAP). The arms, legs, and their link with the upper body are included as 8 LAPs.
    - Limb Orientation Phrase (LOP). The 8 arm and leg limbs, head, collarbones, hips, torsos, and upper body, paired with the 3 reference vectors, result in 45 LOPs in total. 12 pairs that barely change along the reference vector (e.g., left-right hip, leftward vector) are filtered out, resulting in 45-12=33 LOPs.

Reviewer comment (Evaluation):
KP as evaluation alternative to common methods such as FID is also questionable: Even broken and unrealistic human motion would produce “valid” KP that would be difficult to differentiate from valid sequences.

Author response:
**KPG evaluation of naturalness**.  We agree that naturalness is necessary, and also provide user-study results on naturalness in the supplementary materials. However, KP-based accuracy is not an all-in-one metric. With its main target as semantic consistency evaluation, it is more an alternative to R-Precision instead of FID. We believe pursuing an interpretable naturalness metric like KP-based accuracy for semantic consistency is worth exploring.

Reviewer comment (Presentation):
I strongly suggest the authors to use equations on separate lines when describing the models in 4.2.

Author response:
**Equations**. Thanks for your advice. We will incorporate these in the revision.

Reviewer comment (Writing):
Minor issue - On Page 3 $r^f = r^r \times r^r$ should be $r^f = r^r \times r^u$.

Author response:
**Typos.** Thanks for pointing out. We will fix it.

Reviewer comment (Writing):
Minor issue - It is a bit unclear in 3.1 what the “Phrase” is exactly: is it the sign or is it the signal?

Author response:
**Phrase.** Sorry for the ambiguity. By Phrase, we mean the signs as $[-1, 0, 1]$. We will clarify this in the revised version.

Reviewer comment (Writing):
Question - What is the “human cognitive view” on Page 3?

Author response:
**Human cognitive view.** It means that the reference vectors are derived from human cognitive references for egocentric descriptions of actions. For example, we are more interested in which direction vectors humans adopt when saying "My hand is moving leftwards", and find the vector from right hip to left hip.

Reviewer comment (Experiments):
Question - It seems that the authors’ approach could be useful for approaches such as nearest neighbor: did the authors experiment with those approaches i.e. for action recognition or motion to text?

Author response:
**KP for action recognition.** Thanks for your constructive feedback. We conduct a simple experiment for action recognition on the BABEL validation set, where promising results are shown with Top-1 accuracy improving from 41.14 to over 45. The details are updated in the supplementary materials.

Reviewer comment (Evaluation):
The claim of "bridging the gap”: On the one hand the authors say that KPs are automatically extracted and that the “major burden of understanding KP is left to the machines”, on the other hand they maintain that KPs are “easily” interpretable - which I still disagree with. For example, in Figure 3 I would not be able to “easily” understand only from the KP what the poses would be. Interpretability is made difficult due to (1) the relatively large number of phrases and hierarchies and (2) the aggressive discretization.

Author response:
**The claim of "bridging the gap"**. First, we believe that we should not let the perfect be the enemy of the good. Despite the limited granularity in the current version, it could be easily extended with the proposed framework. Second, we argue that the target of KP is not fully replacing language descriptions for action semantics representation. In essence, KP is designed as an **automatic symbolization system** for motion. We expect KP to become an interface for motion-semantics mapping, where the major burden of understanding KP is left to the machines, while humans could interfere with the process as assistance, given the easily interpretable atomic Phrases. Thus, humans are not required to be able to generate KP sequences from scratch.

Reviewer comment (Theory):
There is insufficient detail in Chapter 4.2 compared to the previous chapters; the method should be described with equations and not just in text.

Author response:
**Equations**. Thanks for your advice. We will incorporate these in the revision.

Reviewer comment (Writing):
I do not find the reading particularly easy and clear. There are some parts that are very dense in technical details and/or that make reference to previous approaches. Indeed, the paper strongly relies on previous works and is not fully self-contained. While of course, this is understandable, I would suggest the authors be sure that at least the minimal information to understand and appreciate the work is included (just as an example: the meaning of the metrics in Tab. 3).

Author response:
**Presentation.** Thanks for your kind advice on making the paper more understandable. For the metrics in Tab. 3, we explain the details in Appendix F.2 due to the page limit.

Reviewer comment (Novelty):
I am not sure I fully understood the placement with respect to the literature: in what sense does the proposed method advance the state-of-art?

Author response:
**Comparison to SOTA.**  
Notably, our text-to-motion model could be identified as improving the latent-based diffusion proposed in MLD with the newly proposed KP.  
This choice is made to conduct a fast and light-weight effectiveness evaluation of KP.  
Therefore, a proper comparison target should be MLD, upon which considerable advantages are shown in the user-study results and KPG.  
For T2M-GPT and MotionGPT, which are the current SOTAs, we would like to emphasize the different paradigms it adopted, as well as the model scale.  
Both of them tokenized motions into a codebook, coupled the codebook with text tokens, and then adopted a different pipeline as an auto-regressive decoder-only generative model, which resembles GPT.  
Also, they possess a considerably bigger model scale. For example, the 18-layer transformer is adopted by T2M-GPT for the text-to-motion task, however, in comparison, only a 4-layer denoiser and a 4-layer with a frozen CLIP text encoder are adopted in our method.  
Therefore, we would claim that despite the diverse paradigm and model scale, by incorporating KP into an MLD-like pipeline, we manage to achieve comparable performance with SOTAs, proving the efficacy of KP.  
Furthermore, given the motion quantization ability of KP as mentioned by Reviewer DPfm, also its effortless conversion to natural language, we are optimistic about incorporating it with T2M-GPT for future work.

Reviewer comment (Writing):
I find that one of the main target tasks for this work, i.e. motion generation, should be better clarified from the very beginning, in the introduction (the impression that I have is that it becomes clear just after a while, by reading the next sections).

Author response:
**Clarifying the main target.** Thanks for your kind advice, and we will update our presentation in the revision.

Reviewer comment (Presentation):
The figures are not always enough explanatory. For instance, while Figure 2 is very clear, from Fig. 1 I would expect to have a better focus on what the authors mean when they say that there is a significant gap between motion and action semantics, but it is not clear to me.

Author response:
**Figures that are not explanatory enough.** Thanks for your kind advice, and we will polish the figure captions in the revision.

Reviewer comment (Writing):
"For objectivity and actuality, KP captures sign changes with minimal pre-defined standards" I am not sure I understand this statement.

Author response:
**"For objectivity and actuality, KP captures sign changes with minimal pre-defined standards."**  
By objectivity and actuality, we mean KPs are defined solely on the sign changes of the indicators calculated raw motion only, with little further human-defined thresholds, which could introduce unexpected subjectivity.

Reviewer comment (Evaluation):
"KP offers proper abstraction, which disentangles motion perturbations and semantics changes" How is it assessed in the experiments?

Author response:
**Proper abstraction of KP.** In the first place, the proper abstraction is a design motivation of KP. Compared to raw motion, it is designed as more abstract with interpretable atomic Phrases. Compared to natural language, it tends to preserve more detailed information though not as concise as text descriptions, which could be shown by the higher diversity while keeping competitive consistency as shown in the user studies.

Reviewer comment (Experiments):
The initial part of Sect. 3.1, where the amount of KP of each type is introduced, is a bit unclear to me. How such numbers are derived?

Author response:
**The derivation of the number of KPs.** Here we list the details for KP filtering. These will be updated in the appendix.  
    - Position Phrase (PP). It is composed of 14 joints (except the pelvis and both eyes) and 3 reference vectors with originally 42 Phrases. Since there are joints that define the reference vectors (e.g. hips for the leftward vector). Specifically, shoulders and hips are excluded for the leftward vector, and hips are excluded for the upward vector, resulting in 42-6=36 PPs.  
    - Pairwise Relative Position Phrase (PRPP). It is composed of 136 joint pairs and 3 reference vectors with originally 408 Phrases. 24 joint pairs that are linked by limbs are filtered out. 30 Eye-related pairs are filtered out except (the left eye, and right eye) because the others are covered by head-related pairs. 4 triplets that the relationship barely changes along the reference vector (e.g., right knee, left hip, and leftward vector) are filtered out. These result in 408-24*3-30*3-4=242 PRPPs.  
    - Pairwise Distance Phrase (PDP). It is composed of 105 joint pairs (except for both eyes). 24 pairs that are linked by the human body are filtered out, resulting in 81 PDPs.  
    - Limb Angle Phrase (LAP). The arms, legs, and their link with the upper body are included as 8 LAPs.  
    - Limb Orientation Phrase (LOP). The 8 arm and leg limbs, head, collarbones, hips, torsos, and upper body, paired with the 3 reference vectors, result in 45 LOPs in total. 12 pairs that barely change along the reference vector (e.g., left-right hip, leftward vector) are filtered out, resulting in 45-12=33 LOPs.

Reviewer comment (Writing):
“...we limit the criteria of KP as the indicator signs to minimize the need for human-defined standards (e.g., numerical criteria on the closeness of two joints) for objectivity and actuality “ I can not understand this statement.

Author response:
**"...we limit the criteria of KP as the indicator signs to minimize the need for human-defined standards (e.g., numerical criteria on the closeness of two joints) for objectivity and actuality."**  
The inherent idea is that for different motions, different joints, and different people, the definition of proximity could be varied. How close is close for two joints?  
If 15 centimeters is set as a criterion, it might be okay for feet, but for hands, it could be too large.  
Given this, we adopt indicator signs as the only criteria.

Reviewer comment (Reproducibility):
When describing the way the different types of KP are extracted, it is often reported: “After filtering...”. However, I missed what this filtering is.

Author response:
**KP filtering.** Please refer to QA-8.

Reviewer comment (Writing):
I am not sure I understand the difference between PRPP and LOP, the formula is apparently the same.

Author response:
**PRPP and LOP.** Sorry for the ambiguity. For PRPP, the joint pairs that are not linked by the skeleton are considered. While for LOP, we focus on the human limb orientations. We will update this in the revision.

Reviewer comment (Presentation):
Sec. 4.1: are these preliminaries reporting things already introduced in the previous sections? How are they related to them?

Author response:
**Section 4.1.** It introduced the representations for the motion understanding system based on motion and KP, which is first mentioned in Sec. 1. The KP has been introduced in Sec. 3.

Reviewer comment (Writing):
The description in “Model structure” in Sect. 4.2 is a bit dense and technical, I’m not sure it favours the understanding of a reader not fully familiar with the tools.

Author response:
**Readability of Section 4.2.** Thanks for your kind advice. We would keep revising it to make it more readable.

Reviewer comment (Writing):
Self-supervision: I did not get how the self-supervised approach is designed (in particular, what is the task to be addressed for the self-supervision).

Author response:
**Self-supervision.** The KP-Motion joint space is trained with the task of reconstructing missing KP given corrupted KP sequences, which is a simple self-supervised objective as illustrated in Fig. 3.

Reviewer comment (Novelty):
"Moreover, most current motion generation evaluations are performed on datasets (Guo et al., 2022a; Plappert et al., 2016; Ji et al., 2018) with considerable complex everyday actions, further increasing the difficulty." This should be better justified: in what sense the proposed approach is an advancement? And how it would extend to more complex actions?

Author response:
**"Moreover, most current motion generation evaluations are performed on datasets (Guo et al., 2022a; Plappert et al., 2016; Ji et al., 2018) with considerable complex everyday actions, further increasing the difficulty."**  
We argue that existing datasets with complex everyday actions are helpful, however, they are also hard to evaluate since there exists no reliable metrics on whether the generated motion is consistent with the given text prompt, especially when the prompt contains complex semantics. Instead, KPG provided a reliable metric for semantic consistency.  
Though the current KPG contains mostly simple prompts, we have included temporal compositions of KP, while the concurrent composition of KP is also easy to achieve.  
More complex compositions could explored for more complex actions, which we believe is worth future exploration.

Reviewer comment (Evaluation):
I fail to understand how the accuracy is computed. Giving an intuition, maybe with an example, would be beneficial.

Author response:
**Accuracy computation.** Given the generated motion X with t frames and the target KP $c_i \in [-1, 0, 1]$, we first extract the KP sequence corresponding to $c_i$ as $C_i \in [-1,0,1]^t$. Then,  $c_i \in C_i$ is recognized when there is the sequence $c_i, c_i, c_i, c_i, c_i$ is a consecutive sub-sequence of $C_i$. For PRPP and LOP evaluation, the sequence $c_i^0, c_i^0, c_i^0, c_i^0, c_i^0, c_i^1, c_i^1, c_i^1, c_i^1, c_i^1$ is the expected sub-sequence. We will update the details in the PDF.

Reviewer comment (Experiments):
“For each prompt, we generate one sample considering the annotation cost. We claim that the models should generate natural text-matching motion most of the time so that the one-sample setting would not hurt the fidelity of our user study.” I might misunderstand the statement, but I don’t think just one sample is enough to make considerations on the general behaviour of the method.

Author response:
**"For each prompt, we generate one sample considering the annotation cost. We claim that the models should generate natural text-matching motion most of the time so that the one-sample setting would not hurt the fidelity of our user study." I might misunderstand the statement, but I don't think just one sample is enough to make considerations on the general behavior of the method.**  
Here we clarify that the user study is designed with strict criteria with no tolerance for failure in semantic consistency. For general behavior modeling, we emphasize the considerably larger user-study scale (600 vs. less than 50 different prompts in previous works), which enables us to evaluate the models' behavior across a wide range of text prompts.

Reviewer comment (Evaluation):
In Tab. 5 the performance w/o Body KS seems slightly better on the accuracy. It would be interesting to provide a comment on that.

Author response:
**Accuracy w/o Body KP.** Sorry for the typo. The result should be 52.04. Since there are only three Body KPs, removing them only results in a marginal performance decrease.