Paper: Human Motion Diffusion as a Generative Prior (ICLR-2024)

Reviewer comment (Experiments):
The proposed ComMDM is constrained to two persons only. It is hard to scale to multiple persons.

Author response:
Regarding a generalization of ComMDM for more than two people - that’s an interesting point. Although presented for the two-person case, ComMDM can get as input the activations of more than two people and output the correction terms for each. In the absence of such a dataset with meaningful interactions, we were not able to validate the performance of ComMDM in such a setting.

Reviewer comment (Reproducibility):
For the DiffusionBlending, is the interpolation applied to the final motion or the intermediate feature as the CommMDM?

Author response:
Regarding your question about DiffusionBlending - the interpolation is applied to the final output of the model (X0 prediction), similarly to the Classifier-Free-Guidance scheme.

Reviewer comment (Evaluation):
The authors should conduct a user study to quantitatively compare the visual results of TEACH with the proposed method for long sequence generation. Model parameters and inference speed should also be provided for a more comprehensive performance comparison between the two.

Author response:
* Regarding the comparison with TEACH - Following your comment we have conducted a user study as requested and added the results to the appendix (Figures 11, 12). We also added a paragraph regarding the user study in section 4.1 (in blue). As for the user study, we compared our DoubleTake method to TEACH by sampling randomly out of the validation set of BABEL 20 sequences of long motion such that each is ~10 seconds long in total and generated out of 3 different text prompts and durations. Both (DoubleTake vs TEACH) generated the exact same duration of motion when the transitions were marked with purple color and motions with orange. We asked 3 different question per motion sequence: 
(1)  Which animation aligns better with the text?
(2) [Purple Frames only]: Which animation shows a smoother and natural motion transition?
(3) Which animation, overall, demonstrates higher quality?
Testing Overall Quality, Transition quality and Text To Motion alignment. Most of the users (82 to 85%) preferred our method over TEACH. Each survey had 10 different questions and each question was answered by 10 different users; In total, 20 people answered the survey.  
* Following your request we added the number of parameters and runtime of both TEACH and DoubleTake Appendix, section D (in blue).

Reviewer comment (Experiments):
Dual-person motion generation lacks comparative experiments, for example, with InterGen \[1\]. This paper introduces the InterHuman benchmark, a large-scale dataset for dual-person motion, and provides more comparative references in the article.

Author response:
* InterGen follows and extends the two instances’ communication principle presented in ComMDM. As you mentioned, they introduced a new benchmark and used it to evaluate ComMDM as well as their model. Following your comment, we acknowledge their contribution in our related work section (in blue), yet, we will not duplicate their experiment which was already made public.

Reviewer comment (Experiments):
The only concern is on the fine-tuned motion control part, which seems very similar to controlled motion generation. In that case, there is a body of literature on this subject, many of which use diffusion models for controlled motion generation. The authors failed to include these methods and compare against them. Although these methods have different data requirements, they seem to achieve the same goal. The reviewer asks the authors to explain why they did not include these methods in their comparisons, while noting they are still happy with the paper and think the authors could make a case while still including these works.

Author response:
* Regarding motion control prior work. This is a valid point, the works you mentioned are closely related to our fine-tuning task, hence, we added a paragraph presenting and discussing them in the related work (in blue). While related, we find an important difference between the two lines of work that leads to two different evaluation settings: Jiang et al., Du et al. and Castillo et al. are solving the task of reconstructing full body motion given the motion of a VR headset and two hand controllers. As such, they are measured with reconstruction metrics (i.e. MPJPE and jitter). On the other hand, our fine-tuned and DiffusionBlending models are aimed at modeling the distribution of motion given one or more control signals - for example, given the left hand’s motion we aim to model the distribution of the right hand (and the rest of the body), not to reconstruct the original motion. Another example is given a trajectory, a person can walk/dance/run, etc. through it. In addition, we enable combining textual conditions to vary this distribution. Hence our evaluation is distribution-based (i.e. FID and diversity). As a result, we think that comparing the reconstruction works in the distribution setting (and vice versa) will be unfair.

Reviewer comment (Experiments):
Specific related works that were not included for comparison are:
- Jiaxi Jiang, Paul Streli, Huajian Qiu, Andreas Fender, Larissa Laich, Patrick Snape, and Christian Holz. Avatarposer: Articulated full-body pose tracking from sparse motion sensing. ECCV, 2022.
- Du, Y., Kips, R., Pumarola, A., Starke, S., Thabet, A., & Sanakoyeu, A. (2023). Avatars grow legs: Generating smooth human motion from sparse tracking inputs with diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 481-490).
- Castillo, Angela, et al. "BoDiffusion: Diffusing Sparse Observations for Full-Body Human Motion Synthesis."

Author response:
* Regarding motion control prior work. This is a valid point, the works you mentioned are closely related to our fine-tuning task, hence, we added a paragraph presenting and discussing them in the related work (in blue). While related, we find an important difference between the two lines of work that leads to two different evaluation settings: Jiang et al., Du et al. and Castillo et al. are solving the task of reconstructing full body motion given the motion of a VR headset and two hand controllers. As such, they are measured with reconstruction metrics (i.e. MPJPE and jitter). On the other hand, our fine-tuned and DiffusionBlending models are aimed at modeling the distribution of motion given one or more control signals - for example, given the left hand’s motion we aim to model the distribution of the right hand (and the rest of the body), not to reconstruct the original motion. Another example is given a trajectory, a person can walk/dance/run, etc. through it. In addition, we enable combining textual conditions to vary this distribution. Hence our evaluation is distribution-based (i.e. FID and diversity). As a result, we think that comparing the reconstruction works in the distribution setting (and vice versa) will be unfair.

Reviewer comment (Experiments):
For the generation of long sequences, I don't think it makes sense to essentially generate each interval completely independently. A better option would be to use an autoregressive generation method similar to TEACH, but with the smarter option of combining each subsequence. Would it be possible to compare this with a scheme similar to EDGE [1]? Also, the paper admits that a comparison with DiffCollage is not possible due to a lack of publicly available code resources. Their implementation is simple. This may be optional, but would further support the paper.

Author response:
* Regarding the long sequence generation - DoubleTake **does not** generate each interval completely independently. At the first take, the transition parts of the motions are shared between the neighboring intervals as demonstrated in Figure 3. The first take is similar to the EDGE scheme, with a major difference - while EDGE suggests complete overlap, in our case, the overlap is only for the transition part, which is more appropriate to our application.

Reviewer comment (Evaluation):
Regarding multiplayer motion generation, considering that MRT is an old paper, it could be considered to include a discussion and comparison with [1,2,3]. MRT focuses on deterministic prediction, and for the evaluation, is the prediction from ComMDM sampled once? A better comparison would be to use [2] as a baseline, to check the performance of diverse prediction and to measure diversity.

Author response:
* Regarding ComMDM - Thank you for referring us to “Stochastic Multi-Person 3D Motion Forecasting”. We added it to the related work (in blue) and will evaluate ComMDM using the metrics presented in the paper. Since the evaluation code seems missing from their published code base, we contacted the authors and will include this evaluation in our next revision.

Reviewer comment (Evaluation):
The comparison with TEACH is not very fair either. Is it possible to have the same setting with TEACH but using DoubleMDM, e.g. noising the start and end intervals as diffusion inversions and then performing DoubleMDM with additional transition frames? In this case, you can use accuracy measurements instead of current metrics such as FID. This is because current metrics are more of a test of the quality of the movement and whether the movement is consistent with the text. It is also important to determine if the transition is good by measuring smoothness and accuracy.

Author response:
* Regarding the method suggested to compare with TEACH - DoubleTake generates long sequences by generating all the intervals simultaneously with their shared transition. We are afraid that generating transitions in a stand-alone manner will divert too much from our suggested method. Yet, we do suggest an evaluation setting that measures the quality of the motion and the quality of transitions independently. Additionally, in our ablation studies, we evaluate the second take by itself in a setting that quite resembles your suggestion. Since the transitions are generated simultaneously with the motion, and not stand-alone given input motions, we cannot measure its accuracy. Instead, we evaluate the quality and playability of the motion using FID and diversity. To address your concern regarding the fairness of the evaluation we conducted an additional user study comparing to TEACH (in blue) which shows a clear advantage to our DoubleTake considering the quality of motion, transition, and alignment with the input text.