Paper: NeRM: Learning Neural Representations for High-Framerate Human Motion Synthesis (ICLR-2024)

Reviewer comment (Experiments):
The paper could benefit from a more extensive discussion on the generalizability of NeRM to different datasets and motion types. It would be valuable to explore the performance of NeRM on diverse motion datasets and evaluate its ability to handle complex and varied motion patterns.

Author response:
We thank the reviewer for the thoughtful comment. We will add a quadruped motion dataset [Ref4] that contains 30 minutes of dog motions. Just a kindly note that, we have covered almost all the mainstream datasets that applied in current human motion generation (text-to-motion, action-to-motion, unconditional generation) tasks, and achieved promising results on them, which can validate the effectiveness of our approach. However, we really appreciate the reviewer’s enlightment of including more interesting cross-domain datasets. We are now experimenting on [Ref4], which takes much time since it refers to a different skeleton representation and we need to conduct further research over pre-processing and training stuff. We will include the results in the final version.

Reviewer comment (Novelty):
Some more motion synthesis literatures can be included in this paper, such as:  
[a] A unified 3d human motion synthesis model via conditional variational auto-encoder  
[b] Towards diverse and natural scene-aware 3d human motion synthesis.

Author response:
We thank the reviewer for the suggestion of the two great works [Ref5, Ref6] as references. We carefully include them into the Introduction (Sec. 1 of main paper) with detailed descriptions in Sec. A of Appendix.

Reviewer comment (Evaluation):
It would be beneficial to include more detailed explanations and insights into the proposed clip-FID metric for evaluating the quality of high-framerate generative details. How does clip-FID preserve target framerates without downsampling and how does it capture local details and artifacts such as foot sliding?

Author response:
We thank the reviewer for the suggestion. Following the reviewer's comment, we add a figure (Figure 2 in Sec. C.2 of Appendix) and related description (in Sec. C.2 of Appendix) to better explain the proposed clip-FID metric. Specifically, the figure exhibits the comparison of standard FID and clip-FID, where FID evaluates global structure **(the coarse information of the entire motion is preserved)**, but downsamples all human motions to a common 20fps, ignoring high-framerate details; clip-FID takes motion clips extracted/cropped from motions **(the detailed information of the motion segment is preserved)**, keeping the realism of details at high framerate without downsampling. Note that we randomly sample clips (a short segment of motion) of size $m$ from real motions at framerate $s$ and center $v$ to generate the corresponding clip. 

Due to downsampling, the pre-processed low-framerate data in standard FID inevitably misses the expressive details in motions (e.g., how does the foot move when lifting it during walking) or extreme poses (e.g., in the process of extending and retracting the arm, the farthest reaches of hand may be discarded due to down-sampling, especially if the movement is fast). On the other hand, as each frame of the original motion is not discarded in clip-FID, local details and artifacts such as foot sliding are captured.

Reviewer comment (Novelty):
Question: Eq 2 is confusing to me. Normalizing the clip length from the standard seconds to the relative length has been commonly adopted. In Eq 2, given both the relative time position and the frame rate, the input information to the generation function f_\theta is exactly the same as what previous works input. Could the authors elaborate more about their differences here?

Author response:
We thank the reviewer for this great reminder, and elaborate more about the differences between previous works (including motion-based INRs and image-based INRs) input and ours in Eq 2. We revise the related content in Sec. 3.1 in main paper. First, we sincerely apologize that the input to Eq 2 needs to be revised as $(t_{v,s}^{i},z_{i},s^{i})$. Below we provide detailed explanation. 

**(1) The input to motion-based INRs vs. ours.** Previous motion-based INRs only consider the simple setting: all $n$ training motions $\textbf{X}=[x^{1},x^{2},\cdots,x^{n}]$ in a dataset have the same framerate $s$ and duration $l$, leading to the fixed number of human poses (i.e., $T=s\cdot l$) in each motion. In this case, they regard a motion $ x^{i}=[x_{1}^{i},x_{2}^{i},\cdots,x_{T}^{i}]\in\mathbb{R}^{d\times T}$ as a function of temporal coordinates $t=[t_{1},t_{2},\cdots,t_{T}]$, i.e., $f_{\theta}: (t_{j},z_{i})\mapsto x_{j}^{i}$ for any $j=1,2,\cdots,T$ and $i=1,2,\cdots,n$. To normalize the coordinates into a bounded domain $[-1,1]$, their normalized temporal coordinate can be represented by $t=[-1,\frac{-T+2}{T},\frac{-T+4}{T},\cdots,\frac{T-2}{T},1]$. **The interval of each two normalized coordinate is** $\frac{2}{T}$. **Note that all motions in the dataset have the same normalized temporal coordinate** $t$. However, in our setting, all training motions $\textbf{X}=[x^{1},x^{2},\cdots,x^{n}]$ in a dataset have different framerate $s^{i}$ and duration $l^{i}$, resulting in varied number of human poses (i.e., $T^{i}=s^{i}\cdot l^{i}$). This task is quite challenging as the network architecture can only accept mini-batch data of a fixed size. Thus, we randomly choose a motion clip of a fixed size $m$ for each motion, obtaining $[x_{clip}^{1},x_{clip}^{2},\cdots,x_{clip}^{n}]$ with each motion clip $x_{clip}^{i}\in\mathbb{R}^{d\times m}$. Now, we have a fixed-size motion clips like existing motion-based INRs, but we cannot adopt similar coordinate normalization because each motion clip comes from different position of the original motion. Therefore, we propose the following method. Suppose that the maximum duration is $l_{max}$ for all motions,  we normalize the temporal coordinates of the $i$-th motion as $t_{s}^{i}=[-1,\frac{-s^{i}\cdot l_{max}+2}{s^{i}\cdot l_{max}}, \frac{-s^{i}\cdot l_{max}+4}{s^{i}\cdot l_{max}},\cdots,\frac{s^{i}\cdot l_{max}-2}{s^{i}\cdot l_{max}},1]$. **The interval of each two normalized coordinate is** $\frac{2}{s^{i}\cdot l_{max}}$. **Note that all motions in the dataset have the different normalized temporal coordinate** $t$, **depending on the framerate** $s^{i}$. Our extracted motion clip also depends on the center $v$. Finally, if we want to extract the first $m$ poses as a motion clip from the $i$-th motion with framerate $s^{i}$ and duration $l^{i}$, the normalized coordinates can be represented as $t_{v,s}^{i}=[-1,\frac{-s^{i}\cdot l_{max}+2}{s^{i}\cdot l_{max}}, \frac{-s^{i}\cdot l_{max}+4}{s^{i}\cdot l_{max}},\cdots,\frac{-s^{i}\cdot l_{max}+2\cdot (m-1)}{s^{i}\cdot l_{max}}]$. Due to different $s^{i}$ in each motion, we also input it as a condition. The input of Eq 2 becomes $(t_{v,s}^{i},z_{i},s^{i})$, which is totally different from previous works.

**(2) The input to image-based INRs vs. ours.** Previous image-based INRs regard an image as a function of horizontal and vertical coordinate $(p_{x},p_{y})$ and maps to RGB values. The images in the dataset have the fixed resolution $r\times r$. Therefore, their normalized spatial coordinate can be represented by $p_{x}=[-1,\frac{-r+2}{r},\frac{-r+4}{r},\cdots,\frac{r-2}{r},1]$ and $p_{y}=[-1,\frac{-r+2}{r},\frac{-r+4}{r},\cdots,\frac{r-2}{r},1]$, leading to a bounded domain $[-1,1]\times [-1,1]$. **The interval of each two normalized coordinate is** $\frac{2}{r}$. If each image in the dataset has its own resolution $r^{i}$, the interval can be formulated as $\frac{2}{r^{i}}$. Images are continuous in 2D space, and motion sequences are continuous in the temporal dimension, but we can not simply handle this difference by ignoring one dimension. The **essential difference between image and motion** lies in that, image size (pixel number) is determined only by resolution, while motion size (frame number) is determined by both framerate and duration. This leads to the different normalized coordinate setting: $\frac{2}{r^{i}}$ for image while $\frac{2}{s^{i}\cdot l_{max}}$ for motion. (Note that resolution in image can be analogous to framerate in motion.)

Reviewer comment (Theory):
In my understanding the claim that “NeRM can generate motions with arbitrary framerates s and durations l by setting appropriate temporal coordinates.” may be inaccurate. Technically, by the design of the method, the claim can be partially correct. But given the training data captured under certain frame rates, without considering the motion speed and sensitiveness of capture sensor etc, the model is unlikely to learn motion patterns in frame rates exceeding the highest frame rate contained in the training data.

Author response:
We agree with the reviewer that the model is unlikely to learn motion patterns in framerates exceeding the highest framerate contained in the training data. Just a kindly reminder, actually, we have claimed such limitation in the last lines of Sec. 3.2 in the original paper (Due to space limitation, we move it to Sec. B.3 in Appendix). We carefully improve our expression to make it clearer according to the reviewer's suggestion and highlight it with blue. We also involve it as a limitation in Sec. F of Appendix.

Reviewer comment (Experiments):
Question: is there any reason that a close baseline Nemf (He et al 2022) is not included in the benchmarking comparisons?

Author response:
We appreciate the reviewer for the valuable question. Our NeRM is designed for motion generation with external conditions like text prompts and action labels, so the baselines we choose are also under the same setting. Differently, NeMF focuses on using internal conditions (like keyframes that drawn from the motion itself) for generation, which is orthogonal to our setting. Meanwhile, NeMF can only realize single-motion (deterministic) generation based on its conditions, and cannot support diverse (probabilistic) generation like ours.

However, considering that both NeRM and NeMF have the compatibility of handling unconditional motion generation, we refer to the results (on AMASS dataset) in original NeMF paper, and compare it with ours as below. It illustrates that our NeRM outperforms NeMF in terms of both FID and diversity. This can be ascribed to flexible multi-framerate training and powerful generative ability of the diffusion model. Please note that NeMF is based on VAE framework and can only support fixed-framerate training. We include them in Sec. 4.3 of main paper.

| Method | FID&darr;  | Diversity&uarr; |
| ------ | ------------------------------------------------- | ------------------------------------------------------ |
| NeMF   | 2.07                  | 2.77      |
| MDM    | 8.84                  | 6.43     |
| MLD    | 1.40                  | **8.57**      |
| NeRM   | **1.07**                  | 6.91     |

Reviewer comment (Experiments):
The method is constructed with multiple components and requires settings of many introduced hyper-parameters, such as the normalizing of time positions, the codebook-coordinate attention and etc. The authors may need to provide corresponding ablation studies to support the effectiveness of these modules and the help readers understand the resources of performance gains more clearly.

Author response:
We thank the reviewer for this great suggestion. Following your comment, we will add the ablation study of normalizing of time positions in Sec. E.3 of Appendix. Ablation studies on the time position encodings, codebook-coordinate attention and some other components have been involved in Table 2, Sec. E.3 of Appendix. Please let us know if you have further requirements for ablation of our components.

Reviewer comment (Experiments):
How can we determine if the feet sliding shown in Figure 4 is a result of training with a low framerate?

Author response:
We thank the reviewer for this great reminder. We provide clearer explanation for qualitative visualization in Figure 4 of main paper. As baseline cannot directly generate high-framerate motions, we let it first generate conventional 20 fps (blue), and **conduct interpolation towards target 100 fps** (purple, pink, yellow, green). Floating feet in these 4 colors indicate the sliding phenomenon caused by interpolation. However, for our NeRM, we can **directly** generate 100 fps (orange+purple, pink, yellow, green) with reasonable visual effect. Note that we did not draw every frames in 100 fps as it may appears too crowded, so instead, we only draw frames at 20 fps in orange to present overall generation, with two enlarged views of 100 fps details in the figure. Similarly, Figure 5 of Sec. E.6 in Appendix is also the same. In baseline, the blue is baseline 20 fps, and other colors are interpolation towards target framerate. In NeRM, all frames are directly generated, and we draw at 20 fps as orange, with high-framerate details in other colors.
Additionally, these figures, along with our subsection _High-framerate motion generation_, have proved that low-framerate generation with interpolation is not sufficient to generate high-quality high-framerate motions. This could also be a response for the above W1.

Reviewer comment (Theory):
Does the use of variational INR, i.e., normalizing the latent code $\mathbf{z}_i$ to a normal distribution with KL loss, affect the preservation of high-framerate details? Such normalization typically leads to a smoother representation space?

Author response:
We appreciate the thoughtful comment. To some extent, the use of variational INR introduces a slight degradation in motion reconstruction during our first stage. However, **it is quite important for the second stage (diffusion stage)** due to a smoother representation space (explained in Q2.2). Please refer to Table 3 and related analysis in Sec. E.3 of Appendix. We also highlight this part in blue for emphasis. The table illustrates that variational INR can better preserve high-framerate details compared to non-variational INR, as evident from the clip-FID scores. Additionally, we observe that the reconstruction errors or non-variational INRs is smaller than variational ones, as non-variational INRs may overfit the motion sequence by powerful neural network. These findings suggest that the variational approach effectively regularizes the latent space and improves the sampling of new motions.

Yes, such normalization does lead to a smoother representation space. In this context, each latent code **$z_{i}$** is optimized with an over-parameterized decoder $f_{\theta}$ to reconstruct a single sample. This makes the distribution of latent codes complex, and results in a representation space where semantic similarity does not correspond to a simple distance measure. Consequently, interpolations between representations in this space may lack meaningful interpretations. To avoid such a complex representation space, we normalize the latent code **$z_{i}$** to a normal distribution using KL loss. This regularization of the latent distribution ensures a more meaningful and interpretable latent space. By increasing the weight of the KL loss, the network tends to map different input samples to similar regions in the latent space. This constraint encourages the model to learn a smoother representation space where similar samples are closer to each other in the latent space.

Reviewer comment (Experiments):
As for the generation part, the part of the input for INR is z, it seems that the model heavily depends on the quality of the latent representation z.

Author response:
We agree with the reviewer that the latent representation **$z$** is a crucial variable in our NeRM. It acts as a bridge between the variational INR model $f_{\theta}$ and the diffusion model $\epsilon_{\phi}$. This is similar to latent representation in Stable Diffusion [Ref1]. In our context, the quality of latent representation can be influenced by the dimension and the variational distribution $\mathcal{N}(\mu,\Sigma)$ of **$z$**. Therefore, we add ablation studies to investigate the impact of different dimensions $d$ of **$z$** and weight $\lambda_{KL}$ on the KL-loss for motion reconstruction. When the dimension $d$ is set to 256, the motion reconstruction loss becomes significantly small, and higher dimension can slightly improve the reconstruction ability. For $\lambda_{KL}$, a higher weight results in a smoother latent space but increases the reconstruction error. Following Stable Diffusion, we set $\lambda_{KL}$ as a small value (i.e. 1e-4). Overall, we find that the performance of our method is not very sensitive to both parameters in the wide range. The details concerning the above are added in Sec. E.3 of Appendix.

| $d$ | 128 | 256 | 512 |
| --- | --- | --- | --- |
| MRE | 0.0459 | 0.0412 | **0.0407** |

| $\lambda_{KL}$ | 1e-5 | 1e-4 | 1e-3 |
| --- | --- | --- | --- |
| MRE | **0.0410** | 0.0412 | 0.0459 |

Reviewer comment (Novelty):
The idea of using time INR has been use in NeMF: Neural Motion Fields for Kinematic Animation, which may limit the novelty contribution of this paper.

Author response:
We respectfully disagree with the reviewer that the idea of using time INR in our method is not novel due to the presence of NeMF. Admittedly, both NeMF and our method are based on time INR. However, they are quite different with respect to the following main aspects.
**(1) High-framerate motion generation.** The core idea of our NeRM is to generate high-framerate human motions using available yet limited motion datasets with textual annotations. These valuable datasets, however, consists of multi-framerate data, ranging from 20fps to 250fps. We therefore meticulously design an INR framework to take full advantage of such varied-size data. On the other hand, NeMF adopts a VAE framework that requires explicit modeling of raw motions for input to the VAE encoder, leading to the inability to support training at arbitrary framerate. While NeMF can roughly achieve high-framerate generation by adjusting the temporal coordinates, the quality of the generated motions is significantly poor, resembling common motion interpolation techniques.
**(2) Conditional generation.** Although both of NeMF and NeRM support unconditional generation, our research is orthogonal to NeMF in terms of conditional generation. Particularly, NeMF supports generation with internal conditions (such as keyframes, which belong to motion itself); while NeRM is designed for generation with external condions (such as text prompts and action labels). Another difference is that the deterministic optimization process makes NeMF only generate one motion, limiting probabilistic motion synthesis; while our NeRM supports diverse motion generation.
**(3) Generative framework.** NeMF is trained in the form of VAE, so it has the limitation of not always giving satisfactory results when sampling the latent space. Instead, our NeRM integrates the diffusion model into the neural motion representation, achieving impressive performance of motion generation. 

Note that the limitation (2) and (3) of NeMF can be found in Sec. 5.1 of NeMF original paper [Ref2].

Reviewer comment (Reproducibility):
Two-stage training gives good performance. It would be good to give more details on training the auto-encoding of the latent code.

Author response:
We thank the reviewer for the suggestion. It is required to emphasize that unlike a traditional **auto-encoder** whose latent code is produced by the encoder, we draw inspiration from DeepSDF [Ref3] in 3D shapes and use an **auto-decoder** to learn the latent code without an encoder. This encoder-less design avoids explicit modeling of raw motions, making arbitrary-framerate training feasible. To construct distributions where one can sample a representation to generate a new high-quality motion, we gain insight from Variational Auto-Encoder (VAE) that utilizes an encoder to infer a distribution from which representations of motions can be sampled, and employs a decoder to reconstruct the data from the representation. _As our design does not contain an encoder,_ we model each motion by sampling from a learnable distribution with optimized parameters, i.e., mean $\mu$ and covariance $\Sigma$, leading to learnable parameters ($\mu_{i}$, $\Sigma_{i}$) for $i$-th motion in training dataset. This is our variational INR framework. When training, we sample a latent representation **$z_{i}$** from a distribution $\mathcal{N}(\mu_{i},\Sigma_{i})$. We then combine **$z_{i}$** with temporal coordinate encodings $\gamma(t)$, and input them into a shared MLP decoder $f_{\theta}$. The parameters $\mu_{i}$, $\Sigma_{i}$ and $\theta$ are optimized simultaneously by minimizing the reconstruction loss between the generated and ground truth motion. The details concerning the above are added in Sec. B.2 of Appendix.

Reviewer comment (Novelty):
In order to avoid retraining of INR, some other methods choose to use latent code to modulate the weights of INR. Is there any insight for the choice in the paper?

Author response:
Indeed, both (a) _modulating features (in our paper)_ and (b) _modulating weights of INR (the reviewer mentioned)_ belong to **Generalizable INRs**, which aim to learn common representations of a MLP across instances and adapt to unseen data instances. The simple and widely-used (such as SDF, NeRF) _feature-modulation_ concatenates the latent code into the features of MLP as the input condition and shares whole MLP weights across data instances; while the _weight-modulation_ learns to update the part or whole MLP weights for high performance with unstable and expensive training. Feature-modulation is a simple formulation we adopted for its significantly computational efficiency, and our experimental results show that the feature-modulation is sufficient to achieve good reconstruction of human motions. More advanced designs such as weight-modulation models for supporting arbitrary-framerate training can be left for future work.

Reviewer comment (Experiments):
How do you determine the number of codes in the codebook?

Author response:
We directly use the pre-trained codebook reported in [Ref4], which is trained on the HumanML3D dataset (the same dataset as ours). [Ref4] provides an ablation on the number of codes in Sec. 4 of their supplemental material, and suggests that the performance of 512 codes is slightly better than 1,024 codes while 256 codes are not sufficient for reconstruction. We follow this work and determine the 512 codes in the codebook.