Paper: Generative Human Motion Stylization in Latent Space (ICLR-2024)

Reviewer comment (Novelty):
In terms of the pipeline itself, there are not much novelties. Every component in the pipeline is straightforward and proposed by previous methods for example the adaptive instance normalization, the cycle consistent reconstuction. Although the proposed homo-style alignment loss is new, it is quite simple.

Author response:
**A1**: We would like hightlight the following technical distinctions of our approach from existing works:
*  **Latent Stylization**: We propose latent stylization which use the latent features from autoencoders instead of raw poses (as in previous works) for stylization. This compact stylization representation contains more discriminative features, and empirically enhances the performance on stylization accuracy, content preservation, and generalization ability on unseen datasets. It also encourages better style space learning as shown in Figure 7.
* **Probabilistic Style Encoding**: Unlike previous works using deterministic codes, we propose probabilistic style encoding, modeling style as a Gaussian distribution for each motion input. During training, we encourage all the style Gaussian distribution to be close to N(**0**, **I**). This lead to a smooth and dense style space, supporting novel and diverse style sampling. Benefiting from the probabilistic style, our method exhibits notable flexibility that previous works didn't have, accommodating diverse stylization, various style inputs, and different training settings as shown in Table 1.

* **Homo-style alignment** and **global motion prediction**.  We introduce *homo-style alignment* to provide extra strong supervision signals of style cues which is essentially important in unsupervised setting, and the *post-global motion prediction* to address practical root motion transfer issues.

These technical designs, while not involving computationally intensive operations, achieve state-of-the-art performance while being the most efficient during use. More details are available in our ablation study (Appendix A.), highlighting the performance improvements brought about by these design choices.

Reviewer comment (Reproducibility):
More details about the "Ours w/o latent" model is needed.

Author response:
**A1**: Thanks for your suggestion on this issue. We agree that this part need further clarification. **Ours w/o latent** employs the identical architecture as our full model, as illustrated in Figure 2 (a), without the steps of pretraining or training the motion encoder $\mathcal{E}$ and decoder $\mathcal{D}$ as autoencoders. Although it maintains the same number of model parameters, it directly learns style transfer on poses, allowing us to assess the impact of our proposed latent stylization. 

We also have revised the content in the section 4.1 of updated manuscript.

Reviewer comment (Presentation):
It seems all the inline references used in the paper are not in parenthesis. I believe the authors miss used \citet when there should be \citep.

Author response:
**A3**: We thank the reviewer for pointing this out. We have corrected the reference format in the submitted revision.

Reviewer comment (Experiments):
It seems the whole model can be trained end-to-end. Why the motion autoencoder is pretrained? What will happend when every components are trained end-to-end?

Author response:
**A4**: While it is true that the entire model can be trained end-to-end, we found it challenging for the model to simultaneously learn a meaningful latent representation and figure out how to transfer style traits in these latent-based representations. During end-to-end training, we observed the loss quickly getting stuck at a high range. Conversely, training the autoencoders and stylization models stage-by-stage essentially decomposes the whole task into two simpler sub-tasks, allowing each model to address its corresponding aspect. This strategy significantly eases the learning process.

We conducted an ablation experiment to study this choice of training strategy, reporting results on Aberman(*) and Xia(#) datasets in supervised motion-based stylization. The experimental results align with our observation, indicating that stylization accuracy is merely around 15% on both datasets in the end-to-end training scenario, compared to the accuracy of 92% achieved by stage-by-stage training.

| Training Strategy | Style Acc*| Style FID*|  Geo Dist*| Style Acc#| Content Acc# | Content FID#| Geo Dist#| 
| ----  |----  | ----  | ----  | ----  |----  | ----  | ----  | 
| Seperately|**0.945**|**0.020**|**0.344**|**0.926**|**0.674**|**0.189**|**0.680**|
|End-to-end|0.125|1.521|0.577|0.174|0.293|1.417|0.700|
---

This table and related discussion has been incoporated in the revised manuscript, as in Table 8.

Reviewer comment (Experiments):
Probably not a “weakness”, but this is a unconvincing design to me that a small 1D convolutional network can predict the global motion (say root position) from local joint motion. I would ask for more details and evidence to support this design. The results shown in Table 7 only suggest relatively marginal influence, especially in the supervised settings. Is it possible to also have more results on other datasets?

Author response:
**A1**: **Technical Details**:
"In this work, our pose representation is a 260-dimensional vector, encompassing rotation, position, and velocity information. The global motion predictor (GMP) focuses on regressing the 2D root velocity on the XY-plane, representing the global motion, based on the remaining 258-dimensional features. All features are Z-normalized. The GMP comprises three 1D convolutional layers with a kernel size of 3, no downsampling, and channel numbers of 512, 256, and 260 at each layer. Training utilizes mean absolute error, with added random Gaussian noise for robustness (scale from 0 to 0.5, probability 0.5).

**More results:** To address your concern, we conducted two experiments to evaluate the GMP. Firstly, we assessed prediction accuracy on three test sets, demonstrating reasonable performance even on unseen datasets (Table 10 in the revised content). We extended the evaluation of GMP to all three datasets (Aberman(*), CMU Mocap(#), and Xia(@)) in motion-based stylization, covering both supervised (S) and unsupervised (U) settings.

|Aberman et al. | CMU Mocap| Xia et al.|
|---|---|---|
|46.2|48.7|57.7|
---

We further evaluate the combined impact of GMP on the final stylization, and extend the original evaluation to all the three datasets, specifically on Aberman(*), CMU Mocap(#) and Xia(@). This experiment is conducted in motion-based stylization, on both supervised (S) and unsupervised setting (U).

|Method| Style Acc*|Foot Skating*| Style Acc#|Foot Skating#|Style Acc@|Foot Skating@|
|---|---|---|---|---|---|---|
|Ours (S)| **0.945** | **0.130**| 0.918| **0.140**|**0.926**|**0.263**|
|Ours w/o GMP (S)|0.942|0.141|**0.920**|0.160|0.882|0.331|
|Ours (U)|**0.840**|**0.102**|**0.828**|**0.099**|**0.860**|**0.179**|
|Ours w/o GMP (U)| 0.817|0.116|0.820|0.122|0.777|0.307|
---

We introduced a foot skating metric to assess foot sliding artifacts, observing that our proposed GMP effectively mitigates this issue. While the root motion's impact on style classification results is limited due to its 2D nature in the 260-D pose vector, it improves stylization accuracy on the Xia dataset by 9%. Emphasizing the primary objective of GMP, we illustrate in our 3rd and 4th supplementary videos how it facilitates adaptive pacing for diverse motion contents and styles, showcasing different stylization outcomes (label-based and motion-based) for the same content.

These tables and discussions have been incoporated in the revised manuscript, as in Appendix A., and table 9 and table 10.

Reviewer comment (Experiments):
Given multiple components and modules are proposed to construct the proposed method, a complete ablation study is expected to validate their effectiveness. The ablation of loss weights shown in Table 6 is probably not enough to convince readers that all proposed components are contributing to the final improvement of the observed benchmarking results.

Author response:
**A2**: We appreciate the reviewer's suggestion and have conducted a comprehensive ablation study on all relevant designs in our model. Below are partial results of motion-based stylization in the supervised setting, and the full comparisons are available in Table 6 of the revised version. Note that symbols * and # represent the Aberman and Xia datasets, respectively.

Methods | Style Acc* |Style FID*|Geo Dist*|Style Acc#|Content Acc#|Content FID#|Geo Dist#
---|---|---|---|---|---|---|---
Ours| **0.945**|**0.020**|**0.344**|**0.926**|**0.674**|**0.189**|**0.680**
w/o latent|0.932 |0.022|0.463|0.851|0.654|0.258|0.707
w/o homo-style|0.883|0.032|0.507|0.851|0.537|0.232|0.760
w/o cycle-recon|0.917|0.021|0.385|0.872|0.627|0.208|0.699

Reviewer comment (Evaluation):
Question: The results are focused on motion generation in the skeleton representations. Can the authors also propose some visualization with SMPL LBS? With the mesh, sometimes it can be more intuitive to estimate the plausibility of the joint angles.

Author response:
**A3**: We would like to clarify that SMPL visualization is outside the research scope of this work. For character animation, we have provided visualization examples using clothed 3D avatars in our submitted videos under the name '2_label_based_stylization.' Or you could easily access through this [anonymous link](https://drive.google.com/file/d/1XrpkLLzWHrGHFtJJuLSUELIXBoENdHEM/view?usp=drive_link). Additionally, fitting SMPL shape and poses is not easy in our cases due to the structural differences between our skeleton and SMPL.

Reviewer comment (Experiments):
How about the performance of the proposed approach on the long motion sequence? Is it possible to maintain the style for the whole motion sequence rather than a short period.

Author response:
**A4**: Our model is designed to stylize motions with arbitrary length, as the content encoder utilizes a 1D CNN window sliding along the temporal dimension. In our submitted videos, you can find many motions of variable length, such as stylized text2motion. Additionally, we have included two animations specifically showcasing long motion stylization in the revised supplementary videos, labeled as "7_additional_results(rebuttal).mp4." Or you could visit through this [anonymous link](https://drive.google.com/file/d/1JcceadXfFwLgDwnlLf0yZ-L_tQ02ZShh/view?usp=drive_link).

Reviewer comment (Theory):
The authors should provide more intuitive explanations for why deterministic encoding is used for the content code while probabilistic encoding is used for the style code.

Author response:
**A1**: The main reason of using probabilistic encodeing for style code is to deriving generative ability for diverse or novel style sampling. While for content code, we instead use deterministic encoding for two reasons. Firstly, We do not expect generative ability from temporal content space (it is also difficulty to have). Secondly, unlike styles, which often represent global features of motions, content features exhibit more locality and determinism. These features correlate with precise details in local contexts, making deterministic encoding more appropriate and alleviating the learning burden.

We agree this could be important as it's part of our model design. We have added the content in sec 3.2.1.

Reviewer comment (Experiments):
The authors need to conduct an ablation study to validate their conclusions regarding this design aspect.

Author response:
**A1**: Thanks for the sugguestion. For the analysis of this selection, we conduct an ablation experiment on the supervised motion-based stylization setting.  We report the performance on Aberman et al(*) dataset and Xia et al (#) dataset. Note Xia dataset is unseen to model, aiming to test the generalization ability of models. We denote probabilistic encoder as P, and deterministic encoding as D. Firstly, we can see that probabilistic encoding of styles (second row) outperforms the deterministic encoding (first row) on both dataset with significant margins. For instance, the content accuracy achieved by the probabilistic style encoding on the Xia et al. dataset is 15% higher than that of the deterministic counterpart. Expanding our analysis to the content space, modeling it in a similar probabilistic manner (third row) yields nuanced performance gains in stylization accuracy. However, we observed a challenge in generalization to the Xia et al. dataset, resulting in a sharp performance drop. Meanwhile, the performance of probabilistic content encoding on *content presevation* (as indicated in Geo Dist, Content Acc, etc.) is also suboptimal, which also agrees with the assumption we made above. We also have added this table and related dicussion on the result in Table 7 in the Appendix. A.

| Content Space   | Style Space | Style Acc*| Style FID*|  Geo Dist*| Style Acc#| Content Acc# | Content FID#| Geo Dist#| 
| ----  | ----  |----  | ----  | ----  | ----  |----  | ----  | ----  | 
| D | D|0.913|0.022|0.509|0.870|0.524|0.249|0.767|
|D|P|0.945|0.020|**0.344**|**0.926**|**0.674**|**0.189**|**0.680**|
|P|P|**0.947**|**0.017**|0.489|0.891|0.417|0.322|0.758|