Paper: Single Motion Diffusion (ICLR-2024)

Reviewer comment (Novelty):
Authors propose a solution that fixes a specialized version of UNet (presumably similar to the one used for image generation) by introducing a local attention mechanism. While this might be a way to repurpose an existing architecture developed for a different task, there exists a wide range of existing architectures (1d convnets / wavenets) which exhibit the same property by construction. It is unclear why the vanilla UNet is considered in the first place.

Author response:
**“Why is the vanilla UNet considered in the first place?”**

**“Why is the UNet the default option here? Wouldn't LSTMs / WaveNet / TCNN would be a better option?”**

We have considered several backbones, focusing on architectures that have demonstrated the highest scores in previous research and have already won ablation studies against alternative models. Our initial choice for a backbone has been a UNet, a selection that aligns with the consensus among imaging diffusion works (Ho et al., 2020; Song et al., 2020a; Dhariwal & Nichol, 2021; Saharia et al., 2022) and has demonstrated effective performance.In addition to the empirical results, Jolicoeur-Martineau et al. (2020) demonstrate that UNets significantly enhance sample quality compared to previous architectures used for denoising score matching. Our second choice of backbone is the Transformer Encoder, used by most motion diffusion works (Kim et al., 2022;Tevet et al., 2023, …). Tevet et al. (2023) conduct ablation studies and show that a Transformer Encoder performs better than a Transformer Decoder and better than a GRU (a GRU is similar to the LSTMs mentioned in R3’s question). In our ablation studies we show that in the case of a single motion input, a UNet performs better than a Transformer.

[Alexia Jolicoeur-Martineau, Rémi Piché-Taillefer, Rémi Tachet des Combes, and Ioan- nis Mitliagkas. Adversarial score matching and improved sampling for image generation. arXiv:2009.05475, 2020.]

Reviewer comment (Writing):
afaik vanilla UNet [Ronneberger'2015] does not have attention layers; do you imply the UNet similar to that used in stable diffusion implementation?

Author response:
**“afaik vanilla UNet [Ronneberger'2015] does not have attention layers, do you imply the unet similar to that used in stable diffusion implementation?“**

 Our "vanilla UNet" is the one employed in early imaging diffusion works, e.g., DDPM (Ho et al., 2020), which includes attention layers. We acknowledge the concern raised by R3 that the term "vanilla UNet" might lead the reader to question whether we are referring to Ronneberger2015's UNet or Ho2020's UNet . We have added a clarification in Section 4 (in blue).

Reviewer comment (Writing):
What exactly is meant by "single input UNet" in 4.?

Author response:
**“What exactly is meant by "single input UNet" in 4?”**

It means a UNet trained on a single input. We revised Section 4 for better clarity. Thank you for bringing it to our attention.

Reviewer comment (Experiments):
Why is the UNet the default option here? Wouldn't LSTMs / WaveNet / TCNN be a better option?

Author response:
**“Why is the vanilla UNet considered in the first place?”**

**“Why is the UNet the default option here? Wouldn't LSTMs / WaveNet / TCNN would be a better option?”**

We have considered several backbones, focusing on architectures that have demonstrated the highest scores in previous research and have already won ablation studies against alternative models. Our initial choice for a backbone has been a UNet, a selection that aligns with the consensus among imaging diffusion works (Ho et al., 2020; Song et al., 2020a; Dhariwal & Nichol, 2021; Saharia et al., 2022) and has demonstrated effective performance.In addition to the empirical results, Jolicoeur-Martineau et al. (2020) demonstrate that UNets significantly enhance sample quality compared to previous architectures used for denoising score matching. Our second choice of backbone is the Transformer Encoder, used by most motion diffusion works (Kim et al., 2022;Tevet et al., 2023, …). Tevet et al. (2023) conduct ablation studies and show that a Transformer Encoder performs better than a Transformer Decoder and better than a GRU (a GRU is similar to the LSTMs mentioned in R3’s question). In our ablation studies we show that in the case of a single motion input, a UNet performs better than a Transformer.

[Alexia Jolicoeur-Martineau, Rémi Piché-Taillefer, Rémi Tachet des Combes, and Ioan- nis Mitliagkas. Adversarial score matching and improved sampling for image generation. arXiv:2009.05475, 2020.]

Reviewer comment (Novelty):
For the Harmonization, the way of using guidance injection is from ILVR [2]. Maybe, could you elaborate on this, and emphasize the contribution of this work?

Author response:
**“could you elaborate on this, and emphasize the contribution of this work?”**

Indeed, the performance of SinMDM is built upon the fundamental concepts of Motion Diffusion Models. However, as Motion Diffusion Models typically rely on  large datasets, adapting them to a single motion required a thoughtful redesign. In Section 2, we note that Wang et al. (2022) and Nikankin et al. (2023) use a UNet with limited depth, a strategy not directly applicable to the motion domain. Unlike images with a regularized 2D spatial structure, motion data consists of non-regularized skeletal joints with a temporal axis and fewer degrees of freedom. To bridge this gap, we propose a novel architecture that combines the strengths of the traditional UNET and recent QNA architectures. This effective adaptation to the motion domain enables the application of advanced techniques such as ILVR, as credited in Section 5.2.

We have revised Section 2 to provide an explanation of the distinction between the imaging and motion domains. Thank you for bringing the need for this clarification to our attention.

Reviewer comment (Evaluation):
As the receptive field is narrowed, how to guarantee temporal smoothness? As shown in Table 3, using a wide receptive field gives better FID.

Author response:
**“As the receptive field is narrowed, how to guarantee temporal smoothness? As shown in Table 3, using a wide receptive field gives better FID.”**

Following this question, we added Appendix F (in blue), presenting a study about motion smoothness. In this study, we describe a metric that measures the smoothness fidelity between the generated motion to the ground truth on which the network has been trained.  We show that using a wider receptive field improves smoothness fidelity but degrades the other metrics, thus reducing motion quality. Finally, we introduce a method to enhance smoothness fidelity while maintaining a narrow receptive field, thereby preserving the quality of the results.

As noticed by R2, using a wide receptive field results in better FID. However, a 'near-perfect' FID indicates overfitting to the input motion, as evidenced by poor scores in diversity metrics.  

Please find a video related to the smoothness study, in the newly uploaded supplementary zip.

Description of the smoothness video:

At the top right of each slide is the input motion.

In the bottom-right corner of each slide are two generated motions. The one on the right has been trained with the smoothness loss (𝜆𝑠𝑚𝑜𝑜𝑡ℎ=1), while the one on the left has been trained without it.

Both motions share the same seed, facilitating a temporal similarity in their sub-motions for a more straightforward comparison. When played at normal speed, the motions provide a general impression, but the details are too fast to discern.

On the left-hand side of each slide, we showcase a slow-motion sequence spanning several frames, highlighting the enhancement achieved in motions trained with the smoothness loss.

The first slide displays an unnatural turn executed by the model trained without the smoothness loss. The second slide depicts a higher-frequency, unnatural leg movement in the model trained without the smoothness loss. The leg changes directions too rapidly, initiates a step, and retracts without completing the step. This leg movement does not align with the motifs of the input motion.

Reviewer comment (Experiments):
Comparison with Ganimator in HumanML3D?

Author response:
**“Comparison with Ganimator in HumanML3D”**

We added a row to Tab. 2, exhibiting the metric scores of Ganimator on the HumanML3D dataset, and demonstrating once again that SinMDM performs better. Note that Ganimator’s scores demonstrate a very high FID, indicating divergence from the input motion. The updated radar plot reveals a relatively small area for Ganimator, suggesting that it faces challenges in generalizing to other datasets.
Running Ganimator on HumanmML3D is not straight forward, as Ganimator expects an entirely different data format. To ensure a fair comparison, we transformed the HumanML3D benchmark into the data structure utilized by Ganimator, enabling Ganimator to leverage its skeleton-aware framework. To evaluate the results using the HumanML3D metrics, we reverted them to the HumanML3D format.

Reviewer comment (Experiments):
As restricted by the source of data, the results of some editing applications are somehow not that impressive. Similar results can be achieved by simple manipulations without resorting to a powerful diffusion model. E.g. the spatial composition looks no more than cutting-and-pasting the upper/lower body motion, and the style transfer application is like temporally synchronising the pace of the steps of the happy/crouched style video to the content.

Author response:
**“Results of some editing applications... Similar results can be achieved by simple manipulations”.**

Many of the applications enabled by SinMDM may also be implemented using designated algorithms. However, these designated algorithms are neither simple nor general, while ours provides a one-stop-shop for all applications, and yields better quality results when compared to simple manipulations.

For spatial composition, using cut-and-paste may yield a motion where the hands and the legs are un-synched, for example, in a walk motion, we expect that when the left hand is at the front, the right hand will also be at the front. In addition, when using cut-and-paste there is no balance compensation when the body leans to a specific direction, for example, when the upper body leans to the right, the pelvis should balance by leaning to the left. The motion generated by cut-and-paste is merely a copy of the upper/lower part of the inputs, hence has no flexibility to adjust them to match each other. On the other hand, the motion generated by SinMDM can be different from the one on which it was trained, thus there can be seamless synchronization between the upper and lower body parts. 
In the context of the style transfer application, while manually synchronizing the pace of steps may yield comparable results, SinMDM achieves this straightforwardly, offering a solution that would otherwise demand a designated, complicated approach, as shown in prior works (Menardais et al., 2004; Lee et al., 2020; Panagiotakis et al., 2013). Temporal synchronization requires the identification of both motions' rhythms, a non-straightforward task by itself. Moreover, stretching one motion according to the rhythm of another is naively done by a  linear interpolation, which can cause motion blur, and may not be linear by nature (say, accelerating a leg motion at the beginning of the step and slowing it as it continues). Our model has no such challenges. It generates the required motion such that it is natural, maintaining the motion motifs of the learned input.

[Stephane Menardais, Richard Kulpa, Franck Multon, and Bruno Arnaldi. Synchronization for dynamic blending of motions. In Proceedings of the 2004 ACM SIGGRAPH/Eurographics symposium on Computer animation, pp. 325–335, 2004.]

[Juyoung Lee, Myungho Lee, Gerard Jounghyun Kim, and Jae-In Hwang. Effects of synchronized leg motion in walk-in-place utilizing deep neural networks for enhanced body ownership and sense of presence in vr. In Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology, pp. 1–10, 2020.]

[Costas Panagiotakis, Andre Holzapfel, Damien Michel, and Antonis A Argyros. Beat synchronous dance animation based on visual analysis of human motion and audio analysis of music tempo. In International symposium on visual computing, pp. 118–127. Springer, 2013.]

Reviewer comment (Experiments):
Are there motion representations other than the adopted one from HumanML3D tested for the proposed method? How do you expect the motion representation could affect the generation performance?

Author response:
**“Are there motion representations other than the adopted one from HumanML3D tested for the proposed method? How do you expect the motion representation could affect the generation performance?”**

SinMDM is experimented with three different representations: (1) The HumanML3D representation, including joint rotations, global locations and velocities; (2) The Mixamo representation, which includes rotations and global location for the root joint only; and (3) The Zoo representation, which requires extra degrees of freedom and includes rotations + xyz relative location for all joints. 

For all datasets, we use rotations of the  6D format, which has been proven by prior work to yield optimal results (See Table 2 in the GANimator paper).

Overall, we showed that our approach is robust with different motion representations.

Reviewer comment (Evaluation):
In what circumstances does the method fail?

Author response:
**“In what circumstances does the method fail?”**

In addition to the limitations mentioned in the conclusions section, we observe a failure case that is common to prior works as well - sub-motions are played in an arbitrary order, and this can be a disadvantage when a sequence of sub-motions needs to be executed in a precise order. For example, while doing the bird dance, one should clap hands immediately after shaking the hips. However, single motion algorithms will place these two sub motions in a temporally arbitrary order.

This can be addressed by enlarging the receptive field (at the cost of lower diversity). We added this limitation to the conclusions section (see in blue).