Paper: TapMo: Shape-aware Motion Generation of Skeleton-free Characters (ICLR-2024)

Reviewer comment (Experiments):
The metrics are reported on a human dataset (HumanML3D), which does not show off the cross-category generation ability of the proposed method. Evaluating the method on a dataset with characters beyond humans would be beneficial, such as deforming things 4D [A].

Author response:
(1) **Data.** Thank you for your advice. The experiment on HumanML3D dataset focus on evaluating the motion quality of our text-based motion generation, which ignores the mesh quality of the animation. In addition, we conduct extensive experiments on Mixamo and ModelsResource-RigNet datasets, featuring a diverse set of non-human characters, to evaluate the mesh animation quality as Table 2 and Figure 4 shows.

More generated results for non-humanoid characters and unseen characters can be seen here: https://1drv.ms/v/s!AiJmM4J25MWVzvwreLZLlS_PWkDeHQ?e=rS32C3

Reviewer comment (Experiments):
Driving signal. The driving signals seem to be limited to text in the current form. However, there are motions that cannot be described purely by natural language, such as body gestures, facial expressions, hair movements, etc. This is not necessarily a weakness, as the paper already made the setup clear.

Author response:
(2) **Driving signal.** Thanks for this insightful comment. Whole-body motion generation, such as body gestures and facial expressions, is indeed a valuable research direction that warrants further study.

Reviewer comment (Experiments):
Representation. To generalize to fine-grained motion, such as cloth deformation and hair, the handle-based deformation with a limited number of handles (K=30) seems not enough.

Author response:
(3) **Representation.** Yes, the handle-based deformation strategy currently faces challenges in achieving fine-grained motion control, leaving skeleton-based animation methods dominant in this field. However, the handle-based deformation strategy employed in TapMo offers advantages in efficiently controlling a diverse range of characters in a unified and flexible manner. Our qualitative results illustrate that a limited number of handles (K=30) for generating body animations for these characters is sufficient. Additionally, we conducted tests using 60 handles, the same as SfTP. However, the animation results were inferior to using 30 handles, mainly due to an excess of redundant handles. This abundance adversely affects the Handle Predictor's ability to learn reasonable skinning weights, impacting the overall animation quality.

Reviewer comment (Experiments):
The model is trained on text-motion pairs of human motion but seems to be able to generalize to other categories like quadruped animals. This is interesting. Is the text description transferable over bipeds and quadrupeds? Has the model seen quadruped motion before?

Author response:
(4) **Quadruped animals.** TapMo proves its versatility by successfully animating non-humanoid upright characters like the slime in Figure 1, the potato elf in Figure 4, and the jellyfish, the ghost in Figure 7, enabling them to move in a human-like manner. Additionally, TapMo can drive animals with multiple feet to perform simple actions like walking or running, treating their bodies as human bodies without arms. These quadruped animals and motions haven’t seen by the model in training. These results convincingly demonstrate the remarkable adaptability of our TapMo to characters with diverse shapes and forms. When it comes to more difficult motion of quadrupeds, such as catching prey, their distinct movement patterns significantly differ from those of humans, making it impossible to learn them solely from human motion data. Nevertheless, leveraging the versatility of our Handle-based Mesh Deformation approach, our TapMo system can be extended to animate quadrupeds by training it on quadruped mocap data.

Reviewer comment (Experiments):
The methods use both diffusion-based reconstruction loss and GAN loss. In which case is the GAN loss necessary/complementary to diffusion loss? What happens if GAN loss is removed?

Author response:
(5) **GAN loss.** As indicated by the results presented in Table 2, the inclusion of the GAN loss proves beneficial for our shape-aware motion diffusion model in generalizing to unseen characters. This is evident in the lower "A-L(U)" metric achieved when incorporating the GAN loss. However, the GAN loss will also cause a slight increase in the "A-L(S)" and Handle-FID metrics. Since generalization ability is our focus in dealing with a wide variety of characters, we use the GAN loss in our TapMo.

Reviewer comment (Evaluation):
In Table 1, the TapMo variants have a much higher Diversity metric than top rule methods. Is there an explanation? Additionally, the Multimodal Dist and Multimodality results are not explained in the paper. Explanations on why certain methods perform better/worse would be useful.

Author response:
(6) **Metrics in Table 1.** The value of Diversity metric depends heavily on the pre-trained motion feature extractor. Since the motion representation in our TapMo is differ from that in standard HumanML3D, we adopt the same configurations as theirs and retrain the feature extractor to align with our motion representation. When compared with our real distribution, the Diversity metric of TapMo decreases by 0.452 (16.724 vs. 17.176). This aligns with the trends observed in existing methods, such as T2M and TEMOS compared with their real distribution. The Diversity metric of MDM is most closely aligned with its real distribution (9.559 vs. 9.503).

The analysis method for the Multimodal Dist metric is similar to Diversity. For Multimodal Dist, our TapMo is superior to MDM with 0.687 higher than our real distribution, while MDM is 2.592 higher than its real distribution. T2M demonstrates the best performance on Multimodal Dist with 0.366 higher than its real distribution. As for Multimodality, TapMo performs at a comparable level with MDM and significantly outshines other methods.

Reviewer comment (Theory):
The proposed mesh handle predictor is simple and straightforward, but it is not clear how the proposed method resolves different characters that have different topologies with different semantics. Currently, the manuscript mentions that "each handle is dynamically assigned to vertices with the same semantics across different meshes", but it is not clear how the method will select those handles. Also, it is unclear how the method will choose the number of handles since different topologies tend to have different numbers of handles. It is unclear how the proposed method could achieve training with different numbers of handles.

Author response:
(1) **The proposed Mesh Handle Predictor** is implemented through a GCN-based network capable of handling graph structures with an arbitrary number of vertices and topologies. This enables the processing of different characters with varying numbers of mesh vertices and distinct mesh topologies in a unified manner.

We predefine K=30 handles for all characters and utilize the Handle Predictor to learn and predict a V*K-dimensional skinning weight, assigning the V mesh vertices to K handles. The vertex number V is specialized for different characters. With the exception of the first handle, designated as the root handle, the network autonomously learns the semantics and order of the mesh handles. For instance, in the case of a character lacking legs, as illustrated by the mushroom in Figure 5, vertices will not be allocated to handles associated with leg semantics. In other words, all vertices will have approximately zero skinning weight for the leg handles. 

During training, our Mesh Handle Predictor is trained without handle annotations. It learns the semantics of mesh topologies in a weakly-supervised manner by optimizing the loss in Eq. 5.

Reviewer comment (Novelty):
The proposed Shape-aware Motion Diffusion seems to be simple modifications for existing methods (e.g., MDM), but the current presentation makes it overcomplicated, and difficult for readers to get the key designs that could highly improve the motion generation quality. I am not sure if considering the character shapes is the major factor that improves the quality, and the others could make further improvements.

Author response:
(2) **Shape-aware Motion Diffusion.** The central aim of our Shape-aware Motion Diffusion is to generate motion for a diverse range of skeleton-free characters. The primary challenges lie in the inherent variation in shapes and mesh topologies among different characters, and employing a uniform mesh deformation method can readily result in mesh distortion. Therefore, the consideration of character shapes stands out as the key factor in enhancing the quality of the generated animations.

While the latest diffusion-based motion generation methods have proposed significant enhancements to improve the quality and diversity of generated motion, the focus of our work differs significantly. Our primary objective is not aligned with those methods. Instead, we concentrate on animating a diverse array of non-rigged characters uniformly, placing emphasis on ensuring the quality of animation across this variety.

Reviewer comment (Experiments):
The experiments seem to be insufficient. HumanML3D dataset is the only benchmark that is used to report quantitative comparisons. However, recent methods, such as MotionDiffuse, ReMoDiffuse, and T2M-GPT have not been discussed.

Author response:
(3) **Experiments.** Thanks for pointing out this problem. In the updated version, we have added citations and provided comparisons with these two diffusion-based methods that are relevant to our work.

Reviewer comment (Reproducibility):
Some details are not clearly explained, such as the mesh deformation feature, what exactly is f_ and how it's obtained, and its dimensions, which are not reflected in the main text. From the appendix, it seems to be a 512-dimensional vector. Further explanation from the authors is desired.

Author response:
(1) **Mesh deformation feature.** We thank you for bringing this issue to our attention. The mesh deformation feature is formed by concatenating three-level features within the Mesh Handle Predictor, each with dimensions of 64, 128, and 256 respectively. Subsequently, a linear layer is employed to map the mesh deformation feature to a dimension of 512. This transformed feature is then concatenated with the text token to construct the condition token in diffusion model. The mesh deformation feature encapsulates both the shape and skinning information of the non-rigged character in latent space, providing a comprehensive representation for further shape-aware motion generation.

We have included a comprehensive description of the mesh deformation feature in the revised Appendix (Page 13, Implementation Details).

Reviewer comment (Theory):
How does mesh-specific adaptation affect the vertices? This is not included in the equations.

Author response:
The mesh-specific adaptation refers to a local translation vector associated with the predicted mesh handles. The final local translation of the handles is the cumulative result of the generated local translation and the adaptation. Consequently, the mesh vertices can be deformed through Eq. 3.

Reviewer comment (Reproducibility):
How is the Discriminator implemented?

Author response:
The implementation of the discriminator is inspired by VIBE (CVPR 2020, Kocabas Muhammed et. al.), incorporating a GRU module to extract motion sequence features and an Attention module to fuse these features. The final classification of the motion sequence is achieved through a linear mapping with a sigmoid function. Notably, before feature fusion, we concatenate the features with the mesh deformation feature to enable the discriminator to perceive the mesh shape as well.

Reviewer comment (Reproducibility):
Are the two modules trained jointly or separately?

Author response:
The discriminator and the diffusion model are trained jointly.

Reviewer comment (Experiments):
What are the visualization and qualitative results for Handle-FID? Specifically, what are the two SMPL human models and what do the results look like?

Author response:
(2) **Handle-FID.** The two SMPL human models and their predicted handles utilized in our experiments can be visualized through the following link (https://1drv.ms/f/s!AiJmM4J25MWVzsMKfQ9Fx2L-RXFg3w?e=0aeCBy). It's best to view the OBJ files in MeshLab so that you can observe the skinning weights. We manipulate the beta parameters in the SMPL model to introduce a significant shape difference between the two models. The color of the mesh surface indicates the predicted skinning weights.

Reviewer comment (Experiments):
Are two SMPL human models enough for evaluation? Why don't the authors try MGN ("Multi-garment net: Learning to dress 3d people from images"), which includes clothed human models and can use SMPL pose parameters to drive?

Author response:
The purpose of introducing the Handle-FID is to evaluate the global geometry quality of the animation results. Thus, for ease of implementation without losing universality, we use beta parameters to adjust the shape of the SMPL model instead of using the MGN.

Reviewer comment (Novelty):
The authors are recommended to discuss the skeleton-free papers "Zero-shot Pose Transfer for Unrigged Stylized 3D Characters" and "HMC: Hierarchical Mesh Coarsening for Skeleton-free Motion Retargeting", as the self-supervised shape understanding method in the former could potentially strengthen handle prediction including mesh deformation feature extraction.

Author response:
(3) **Skeleton-free papers.** Thanks for your insightful advice. The ZPT introduces an implicit pose deformation module, which contrasts with our explicit mesh deformation approach. On the other hand, the HMC represents an enhanced version of SfPT, employing a coarse-to-fine transfer of mesh motion to mitigate mesh distortion. However, it's noteworthy that the computational costs associated with HMC are considerably higher than those of SfPT. In our study, we propose the learning of mesh-specific adaptation to address the mesh distortion issue, akin to the fine-grained deformation in HMC but with improved efficiency.

We have incorporated citations and provided a detailed analysis of these two relevant papers in the final version of our paper (Section 2. Related Work. Skeleton-free mesh deformation).

Reviewer comment (Evaluation):
The authors could try using the PMD metric in SfPT and previous works, which allows for direct comparison with ground truth vertices and can judge both motion and mesh quality, and is suitable for SMPL-based models.

Author response:
(4) **PMD metric.** Given that TapMo is a generative model producing diverse motion outputs, calculating the PMD metric using the ground-truth mesh motion of the SMPL model becomes impractical.

Reviewer comment (Theory):
Compared to Eq1 in SfPT, if we ignore the global translation and rotation of the root, the authors added $\tau^{l}_k$ and $h_k$, which seems different from SfPT Eq1. Could the authors explain the purpose of this?

Author response:
(6) **Eq1 in SfPT.** The practical implementation of Eq. 1 in the SfPT code aligns with our approach. Although our formula expression differs from Eq. 1 in SfPT, it is because we choose to separately represent the translation term instead of combining it into the rotation matrix. This decision is made to enhance the clarity of expressing the mesh deformation process.

Reviewer comment (Writing):
For local translations and local rotations, their first dimension is K. Does this include the root (first) handle? Is k=1 representing the root handle in Eq. 3? Or are there K-1 local translations/rotations from k=2 to k=K?

Author response:
(7) **Local translations and local rotations.** The local translations and local rotations have K-1 dimensions in the first dimension, excluding the root handle. We appreciate you for pointing out this problem. We have thoroughly revised the content to rectify the error and enhance the overall clarity of the writing.

Reviewer comment (Writing):
In the first paragraph of the Method section, does "skinning weight s of the handle" refer to the skinning weights of the vertices?

Author response:
(9) **Skinning weights of the vertices.** Yes. Thank you for helping us correct this writing error.

Reviewer comment (Evaluation):
There are still penetrations between foot and ground in the generated animations, which downgrade the animation quality.

Author response:
(1) **Foot-ground penetration.** Thank you for the positive feedback and for bringing attention to the issue of foot-ground penetration in animations—a challenge that is indeed widespread in most text-motion generation models. We appreciate the opportunity to discuss this further. In current kinematic-based motion generation models, addressing this problem typically involves post-processing steps. In contrast, PhysDiff [Ye Yuan et.al. ICCV 2023] employs advanced physical simulation techniques to dynamically adjust foot-ground interactions, resulting in a significant reduction in penetrations. This method represents a novel approach in the realm of text-motion generation models. We are committed to exploring and addressing this challenge in our future research endeavors, striving to find a more effective resolution for skeleton-free animations.

Reviewer comment (Reproducibility):
Please clarify how the adversarial loss is trained in the rebuttal.

Author response:
(2) **The adversarial loss** is one of our special design for the skeleton-free animation pipeline, which helps the network generate reasonable motion adaption to reduce mesh distortion.

Specifically, a mesh motion discriminator is designed to process the mesh vertices of a motion sequence along with the character's shape feature as inputs. It then produces a classification result indicating whether the mesh motion is driven by the predicted handles or the pre-rigged skeleton. The input mesh vertices are automatically sampled on the character's arms using the predicted handles and skinning weights. Recognizing that skeleton-driven character arms can maintain a reasonable articulated structure and shape in a motion sequence, we employ the adversarial loss to penalize abnormal deformations resulting from the handle-based driving approach. 

The adversarial loss is formulated as Eq.8 in the paper. The optimization process involves simultaneously optimizing the diffusion model and the discriminator using adversarial loss and diffusion-based reconstruction loss. In addition, a margin parameter of 0.3 is introduced, dictating that the discriminator is optimized only when the score of the fake sample exceeds the margin parameter, and vice versa.

We have added the detailed training strategy in the revised Appendix (Page 13, Implementation Details).