Paper: OmniControl: Control Any Joint at Any Time for Human Motion Generation (ICLR-2024)

Reviewer comment (Novelty):
Previous works guiding the position of the pelvis could potentially be intuitively adapted to joint positions with proper coordinate transformations, which undermines the novelty of this work, making it seem more of an incremental step rather than a solution to fundamental problems in the field.

Author response:
**A1**: Do you mean that, if we know the coordinate transformation between the pelvis and other joints, we may be able to control them together using previous methods?

In general, we know in advance neither the position of the pelvis nor the coordinate transformations between the pelvis and other joints that we’d to control, as discussed in the introduction (paragraph 2) and Section 3.1 (human pose representations). Thus adapting the previous pelvis-control-only methods to other joints is not feasible.

If it’s not what you mean, we would greatly appreciate it if you can clarify the question.

Reviewer comment (Experiments):
Despite the novelty of controlling all joints at any frame, the paper lacks a discussion on efficient collection of the spatial guidance trajectory, particularly considering the fact that the joint positions are no more relative to the pelvis. This aspect is crucial for practical applications in industries such as 3D animation. Moreover, the paper does not discuss the model's tolerance for inherently unnatural guidance (*e.g.*, manually drawn trajectories or those from different datasets).

Author response:
**A2**: We envision there may be two ways to collect the spatial guidance trajectories. First, for practical applications in industries such as 3D animation, the user (designer) may provide such guidance trajectories as part of the application development. Second, the spatial guidance may be from the interactions and constraints of the scene/object. For example, the height of the ceiling or the position of the chair (and thus the position of the controlled joint), as we show in the last column of [Figure 1](https://github.com/OmniControl24/OmniControl24.github.io/blob/main/static/images/figure1.jpg) in the paper. In both cases, the spatial guidance trajectories can be efficiently collected.
The discussion has been added to **Appendix A.11** of the revised paper.

**A3**: In all the figures and supplementary video, the input control signals are manually drawn (e.g. the circle, straight, and sine lines in [Figure 1](https://github.com/OmniControl24/OmniControl24.github.io/blob/main/static/images/figure1.jpg) and [supplementary video](https://youtu.be/29M-Shs1Orw?t=27)). To better resolve your concern, we added some results with extremely unnatural spatial control signals in the updated supplementary video [(from 4:41 to 5:23)](https://youtu.be/29M-Shs1Orw?t=284) and penultimate section of the [website](https://omnicontrol24.github.io/). These results show that our model has a good tolerance for unnatural trajectories (teleportation, spiral forward).

Reviewer comment (Presentation):
There are some formatting errors (*e.g.*, the misuse of `\citep` instead of `\citet` in some citations).

Author response:
**A4**: Thank you for pointing this out. We have revised these misuses in the citations.

Reviewer comment (Experiments):
In most scenarios, the text input and the spatial guidance seem redundant. Can the model effectively comprehend and follow instructions that appear only in one modality? (*e.g.*, providing spatial guidance only on the foot or pelvis, while the hand activity is only described via the text input?)

Author response:
**A5**: In some cases especially when the human is interacting with the scene or object, the spatial control signals are usually provided as global locations of joints of interest in keyframes **as they are hard to convey in the textual prompt**. For instance, in the low-ceiling scenario shown in the last column of [Figure 1](https://github.com/OmniControl24/OmniControl24.github.io/blob/main/static/images/figure1.jpg) in the paper, it is infeasible to specify the spatial constraints in the textual prompt precisely. Even if we do, there is no guarantee that the model will faithfully incorporate them into the human motion generation, which will lead to unsatisfactory user experiences. Therefore, it is more natural and intuitive to provide spatial control signals as a separate input. At the same time, the textual description is helpful for conveying high-level semantic guidance, such as “playing the violin”. Therefore, the text prompt and spatial guidance are complementary to each other instead of redundant.

**A6**: Yes it can. In the example in the top left of [Figure 1](https://github.com/OmniControl24/OmniControl24.github.io/blob/main/static/images/figure1.jpg), the spatial control signal is applied to the pelvis (walk along the circle), while the text is used to describe the hand activity (play the violin). 
As can be seen, our model can effectively comprehend and follow instructions on separate joints (pelvis and hand). We provide the results with only a text prompt or only a trajectory as the condition in the supplementary video [(From 5:23 to 5:45)](https://youtu.be/29M-Shs1Orw?t=325) and the last section of the anonymous [website](https://omnicontrol24.github.io/). The results show that our model can follow the instructions that appear only in one modality.

We’d like to note that since the text input and the control signals describe the motion of different joints, they are complementary instead of redundant.

Reviewer comment (Writing):
The paper writing is not fluent enough and needs polishing to be easier to follow.

Author response:
**A5**: We have revised the paper to improve the reading flow. We would greatly appreciate it if more insights could be shared on how to further improve it.

Reviewer comment (Evaluation):
Given the carefully designed modules, the time efficiency for training is important to evaluate the significance of the proposed method. However, this part is missing in the paper.

Author response:
**A1**: Thank you for the suggestion to add the training time. It takes **29 hours** to train our model on a single NVIDIA RTX A5000 GPU with batch size of 64 and 250,000 iterations in total.  
As a reference, MDM, PriorDM, and GMD take 72, 11.5, and 39 hours, respectively, according to what is reported in their papers.  
MDM and PriorMDM are trained on a single NVIDIA GeForce RTX 2080 Ti GPU (slower than the A5000 GPU).  
GMD is trained on a single NVIDIA RTX 3090 (roughly comparable with the A5000 GPU).  
Our training efficiency is better than GMD. PriorMDM performs better in training efficiency because it shares the same architecture as MDM and thus can directly finetune the pre-trained weight of MDM (the pre-training time was not included in the reported timing).  
We have added this information in **Table 4 in Appendix A.2** of the revised version.

Reviewer comment (Experiments):
Some important baselines are missing in the experiment sections, such as [1,2]. Adding the full set of published baselines on the benchmarks of HumanML3D and KIT-ML will change the position of the proposed methods highly. Can the authors elaborate more about the comparison with the baselines? Or maybe there is any reason that these baselines are not proper to compare with?

Author response:
**A2**: Thank you for sharing these two papers. These baselines [1, 2] are indeed not feasible to compare with because they **cannot** integrate spatial control signals into text-based human motion generation, which is the main focus of our paper.  
The baselines we compared in our paper are all the methods that can integrate the spatial control signals (to our best knowledge). We have compared with  
1. MDM and PriorMDM: they demonstrate that the inpainting-based methods can be integrated into their pipeline to achieve pelvis-controlling with dense signal in their paper. They can only control the pelvis with the global control signal.  
2. GMD: it focuses on exactly the same problem as ours but can only control the pelvis.  

*[1]: "T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations” CVPR 2023*  
*[2]: “Generating Diverse and Natural 3D Human Motions from Text”, CVPR 2022*

Reviewer comment (Writing):
Some minor writing issues, such as duplicated typo: input -> "input".

Author response:
**A3**: We believe you meant “imput -> input”. We have revised it to avoid any confusion.

Reviewer comment (Evaluation):
In the third column of Figure 1, the authors show that the method can support a combination of control signals from different joints. However, the paper lacks quantitative analysis to further examine its performance.

Author response:
**A1**: Thank you for the suggestion! We have added such quantitative results of the combination of control signals from different joints, which are inserted into **the last row of Table 1**. We have also added more discussions in **Appendix A.4**. 
We also post the results here:
|              | FID ↓ | R Precision (top 3) ↑ | Diversity-9.503→ | Foot sliding ratio↓ | Traj. err. ↓ (50 cm) | Loc. err. ↓ (50 cm) | Avg. err. ↓ |
|--------------|----------------|----------------------------------|------------------|---------------------|----------------------|---------------------|-------------|
| Single Joint | 0.310          | 0.693                            | 9.502            | 0.0608              | 0.0617               | 0.0107              | 0.0404      |
| **Combination**  | 0.624          | 0.672                            | 9.016            | 0.0874              | 0.2147               | 0.0265              | 0.0766      |

There are 57 possible combinations for six types of joints. Since running an evaluation for each of them is costly, it's impractical to evaluate all the combinations. Instead, we randomly sample one possible combination for each motion sequence for evaluation. 
The performance is lower compared to the single-joint control (not an apple-to-apple comparison though as the ground-truths are different). Nevertheless, the results show that controlling multiple joints is harder than a single one due to the increased degrees of freedom.