Paper: Realistic Human Motion Generation with Cross-Diffusion Models (ICLR-2024)

Reviewer comment (Theory):
The major weakness of this work is the lack of explanation for why 2D information could help in 3D human motion generation, given 2D motions are merely a projection of 3D motions onto four orthogonal views. Such projection only reduces the 3D information, without introducing new information. Unfortunately, there is no convincing theoretical motivation behind such an operation. Specifically, "... complementary information in both motion representations, capturing intricate human movement details often missed by models relying solely on 3D information", would you explain precisely how the projection helps provide "intricate" details, which are "complementary" to 3D information, given that they come from 3D in the first place?

Author response:
**Q1: How does the projection help provide "intricate" details?**

As we have mentioned in **Response to All Reviewers**, When using only 3D representation, the generation model may struggle to relate text semantics to some body part movements with very small movement variations compared to others, which can lead to overlooking important motion details. This is because the model might focus on more dominant or larger movements within the 3D space, leaving subtle nuances underrepresented. Due to different view projections in 2D data, small body part movements can be magnified in certain projections, making them more noticeable and easier to capture. This helps the text-to-motion generation models to better associate text descriptions with a wider range of human body motion details, including subtle movements that might have been overlooked when relying solely on 3D representation.

To provide a more specific explanation, we have included a complete analysis of 2D and 3D motion in Section 4.4.1, which should help in understanding how the projection helps provide "intricate" details. It is the difference in the distribution of 2D and 3D motion data that inspired us to learn the relation between 2D motion and text.

Reviewer comment (Experiments):
More analysis of 2D information would be helpful. For example, which view from the four (front, left, right, and back) is the most useful? Would a top view be helpful too? I feel the current version creates more questions than it answers.

Author response:
**Q2: More analysis of the 2D information.**

To evaluate the impact of different views, we conducted tests on four settings: (a) only the front view, (b) only the left view, (c) four views including front, left, back, and right, and (d) five views with an additional top view. When multiple views were used, the 3D motion was paired with a random view projection to formulate the loss. The results is shown below:

| Methods | R Precision(top 3) $\uparrow$ | FID $\downarrow$ | MM Dist $\downarrow$ | DIV $\rightarrow$ |
| ----- | ------ | ----- | ----- | ---- |
|1 view(front) | 0.722 | 0.186 | 3.467 | 9.798 |
|1 view(left) | 0.715 | 0.181 | 3.412 | 9.834 |
|4 views | **0.730** | **0.162** | **3.358** | **9.577** |
|5 views | 0.695 | 0.202 | 3.613 | 9.502 |

It is reasonable to assume that four views contain more 2D information and outperform one view.  Furthermore, the front view and left view do not have much distinction in terms of performance. However, the performance actually decreased with the additional top view. This could be due to the fact that the 2D information becomes more difficult to classify without the condition of the camera view. We did not pass along the camera information because it is difficult to estimate the camera view of in-the-wild videos, and injecting camera information could disrupt the structure of the text-to-2D model, making it difficult for the text-to-3D model to follow. In conclusion, the number of views serves as a practical hyperparameter that can be adjusted through enumeration experiments.

We have provided a complete analysis in Appendix A, which we encourage you to refer to. Additionally, we were expecting that adding a top view would improve the performance, but the results showed the opposite. This has prompted us to think that more views do not necessarily lead to better results. A large gap between 2D information may complicate the easy-to-learn text-to-2D relationship. Therefore, the number of views serves as a hyperparameter in our CrossDiff model.

Reviewer comment (Reproducibility):
Considering the losses, is the framework aware of the input/target view (front, left, right, and back)? Specifically, for each 3D motion, how is the four 2D projection paired in the training? Some more details would be helpful.

Author response:
**Q3: Is the framework aware of the input/target view?**

No, we did not pass along the camera information. During training, each epoch, the 3D motion is paired with a random 2D projection. Our design is to extend our model to learn in-the-wild 2D pose while camera information is difficult to obtain. For further illustration, please refer to Q2 of the response to reviewer W5FS.

Reviewer comment (Novelty):
I wonder if the $x_{2D}$ -> $\hat{x}_{3D}$ motion generation is linked to 3D pose estimation via lifting (such as [A] and many follow-up works)?

Author response:
**Q4: If the $x_{2D}\rightarrow\hat{x}_{3D}$ motion generation is linked to 3D pose estimation via lifting?**

Yes, we also find that there is a similarity between $x_{2D}\rightarrow\hat{x}_{3D}$ motion generation and 2D motion lifting. [A] proves that 3D pose estimation via lifting is more accurate than the end-to-end method. The insight behind it might be that 3D motion is more difficult to extract features than 2D motion. So directly estimating 2D motion and lifting it works like specifying a feature that is easy to learn. Our method also considers learning the relation between 2D motion and text is easier, and we think this is helpful for data learning.

Reviewer comment (Presentation):
A video in the supp would be helpful, as "high-quality" motion generation has been mentioned in the manuscript.

Author response:
**Q5: A video in the supp would be helpful.**

As you requested, we have presented a video in the supplementary material to compare our results with other methods.

Reviewer comment (Evaluation):
What is the concrete conclusion we could draw from Figure 6a)? Mixing sampling performs better in R Precision with a large $\alpha$, but consistently outperformed by the standard sampling in terms of FID? It is very common to have conflicting trends with different metrics, but some more elaboration will be helpful.

Author response:
**Q6: What is the concrete conclusion we could draw from Figure 6(a)(the number of old PDF)?**

The figure clearly indicates that enhancing R precision is achieved by increasing 3D sampling steps, whereas optimal FID performance necessitates a balanced combination of 2D and 3D sampling. This finding emphasizes the critical need for custom-tailored sampling strategies to meet specific performance criteria effectively. For a more intuitive understanding, we suggest refering the supplementary material, which includes a video providing visual comparisons.

Reviewer comment (Presentation):
Minor: Figure 6 will benefit from some reformatting. Currently, the figure is too small, while large margins waste a lot of space.

Author response:
**Q7: Reformatting of Figure 6(the number in old PDF) and root-decoupled diffusion.**
Thank you for pointing out the problem. We have moved both parts to the Appendix.

Reviewer comment (Novelty):
Minor: it would be hard to consider the root-decoupled diffusion as a significant novelty.

Author response:
**Q7: Reformatting of Figure 6(the number in old PDF) and root-decoupled diffusion.**
Thank you for pointing out the problem. We have moved both parts to the Appendix.

Reviewer comment (Experiments):
The authors claim an essential advantage of the proposed method as “to utilize 2D motion data without necessitating 3D motion ground truth during training, enabling the generation of 3D motion.”. However, through the experiments discussed in Sec 4.3, before training with 2D-only data, the model has been pretrained on the complete 3D motion dataset only. Therefore, the claim seems misleading to me. It makes good sense that when 3D data is available, by projecting the 3D to 2D representation, we can learn a joint representation space for both 2D and 3D space. With a language encoder, the motion space and the language space are connected, thus further making text-to-motion generation. By fine-tuning on new 2D-only data, the model learns new samples aligned under the 2D representation, thus extending the text-to-motion generation diversities. However, this practice can hardly be claimed as “using 2D motion data without 3D GT during training” in my opinion.

Author response:
**Q1: The claim of “using 2D motion data without 3D GT during training”.**

We acknowledge that the original description may have been misleading, and we want to emphasize that we have no intention of exaggerating the quality or significance of our work. We have revised the statement to "with a pretrained model, our approach accommodates using in-the-wild 2D motion data without 3D motion ground truth during training to generate 3D motion." We hope this clarification makes our attribution clear and removes any doubts.

Reviewer comment (Theory):
The major claim of the paper, that 2D human motion somehow contains more intricate motion than 3D human motion, is unconvincing as the 2D motion is strictly less “informative” than the 3D motion.

Author response:
**Q1: The analysis of 2D and 3D motion.**

As we have mentioned in **Response to All Reviewers**, when using only 3D representation, the generation model may struggle to relate text semantics to some body part movements with very small movement variations compared to others, which can lead to overlooking important motion details. This is because the model might focus on more dominant or larger movements within the 3D space, leaving subtle nuances underrepresented. Due to different view projections in 2D data, small body part movements can be magnified in certain projections, making them more noticeable and easier to capture. This helps the text-to-motion generation models to better associate text descriptions with a wider range of human body motion details, including subtle movements that might have been overlooked when relying solely on 3D representation.

We have provided a complete analysis of 2D and 3D motion in Section 4.4.1, which should help in understanding why 2D motion contains more details. We propose the concept that 2D motion can help in 3D motion learning, regardless of the model used.

Reviewer comment (Writing):
The purpose of the learnable token embeddings is not explained.

Author response:
**Q3: What is the purpose of the learnable token embeddings?**

We do not have any special purpose for learnable token embeddings. Our model is simple and nearly the same as the standard transformer model [1] when viewed in only one domain pipeline. As the original transformer decoder is inputted with the output embedding, we regard learnable token embeddings as a simple method to learn the output feature.

Reviewer comment (Writing):
It is not explained why the text embedding is added separately to the 3D and 2D encoder and not “jointly” in the “Shared Weights Encoder”.

Author response:
**Q4: Why is the text embedding added separately to the 3D and 2D encoder and not “jointly” in the “Shared Weights Encoder”?**

As mentioned in our response to Q1, we find that 2D motion contains different attention to the different joints. Therefore, we want to establish a direct connection between the 2D motion and text prompt and enforce the model to learn the connection between the different details and the text. If the text embedding is added to the "Shared Weights Encoder," the whole model is forced to learn the connection between 3D and 2D motion first and is only able to extract the feature of motion, which may result in the same output as extracting the feature from 3D motion alone. This would cause the remarkable details in the 2D domain to be fused and lose their value.

Reviewer comment (Presentation):
Figure 6 is too small.

Author response:
In response to your feedback, we have moved the related experiments from Figure 6 (the number in the old PDF) to the Appendix to make room for other important experiments in the main body. Once again, we appreciate your valuable comments and are open to any further questions or suggestions you may have.

Reviewer comment (Novelty):
Some significant references are missed in this paper, such as ReMoDiffuse\[1\] and Fg-T2M\[2\].

Author response:
We would like to express our gratitude for your insightful comments and constructive feedback. We have carefully considered each concern and provide our responses below:

**Q1: Add significant references.**

We acknowledge our oversight and have taken steps to address it. Specifically, we have added the relevant work to both Section 2.1 and Section 4.

Reviewer comment (Experiments):
Some architecture designs are not sufficiently evaluated. For example, why the authors choose to share the intermediate transformer. Quantitative results are required here.

Author response:
We would like to express our gratitude for your insightful comments and constructive feedback. We have carefully considered each concern and provide our responses below:

**Q2: Effect of the shared-weights encoder.**

In response to the reviewer's concern regarding the efficacy of the shared-weight encoder, we have conducted a evaluation, the results of which are presented as below. 

| Methods | R Precision(top 3) $\uparrow$ | FID $\downarrow$ | MM Dist $\downarrow$ | DIV $\rightarrow$ |
| ----- | ------ | ----- | ----- | ---- |
| w/o shared-weights encoder | 0.714 | 0.187 | 3.496 | 9.488 |
| w/ shared-weights encoder | **0.730** | **0.162** | **3.358** | **9.577** |

These empirical findings demonstrate that incorporating a shared-weight encoder significantly boost the the performance across several key metrics, including R Precision, Precision, FID, and DIV. This enhancement is not merely incremental but substantial, as evidenced by the quantitative results presented. 
Inspired by [1], we found that when learning with data from two modalities, extracting separate and fused feature layer-by-layer is more efficient. The shared-weights encoder serves as a fusing module, while the 3D/2D motion decoder acts as a separate decoder module. The goal is to ensure that the decoder layers follow the same extraction patterns as the shared-weight layers, rather than simply gaining deeper embeddings.

For an in-depth analysis and discussion of these outcomes, we kindly direct the reviewer's attention to Section 4.4.3 of our revised manuscript, where these aspects are elaborated upon in detail. We hope this additional data addresses the concerns raised and further substantiates the efficacy of our proposed method.

Reviewer comment (Evaluation):
The authors should provide user studies to quantitatively evaluate the visual quality.

Author response:
We would like to express our gratitude for your insightful comments and constructive feedback. We have carefully considered each concern and provide our responses below:

**Q3: Add the User study.**

Thank you for the suggestion. We conducted a user study on motion performance, in which participants were asked two questions
to assess the vitality and diversity of the motions. The first is "Which motion is more realistic and contains more details?" and the second is "Which generations are more diverse?". The results are presented as below.

| Question | Ours vs MDM | Ours vs MLD | Ours vs T2M-GPT |
| ----- | ----- | ----- | ----- |
| q1 | 81% | 66% | 56% |
| q2 | 60% | 58% | 69% |

Our method produces more realistic and vivid generations while maintaining the diversity of the diffusion model. For visualization results, please refer to the supplementary material.

Thank you once again for your valuable advice on the experimental settings. We greatly appreciate your feedback and are open to any further questions or suggestions you may have.

[1] Xu et al. Building bridges between encoders in vision-language representation learning.