Paper: Efficient Meshy Neural Fields for Animatable Human Avatars (ICLR-2024)

Reviewer comment (Presentation):
Better illustrations are needed. Many illustrative figures in this manuscript lack proper notation and are hard to align with the text. It would make the reader understand the technical details better if more details were included.

Author response:
**[Q1]** Illustration of the model.

We have revised the figure accordingly, including annotations to help clarify our design. The revised version has been updated.

Reviewer comment (Novelty):
Novelty? The proposed method combines existing methods [1] and [2] (and many other papers in the field). Although one of the major differences is the efficiency in rendering, the same improvement can also be theoretically achieved by training a SDF and extracting a water-tight mesh for rendering afterward. It would be great to have a more thorough discussion on the merit of the current pipeline as well as other potential advantages it brings.

Author response:
**[Q2]** Novelty.

[1] is designed for static object reconstruction. Our method extends it to modeling dynamic humans. One insight is that, given the mesh extraction framework from [1], we can directly learn the forward LBS skinning without iterative and (potentially) ambiguous root-finding [3].
While [2] uses learned backward skinning which is insufficient for animation and faithful reconstruction of the motion (We include a figure in the updated supp. to demonstrate the problems).
To the best of our knowledge, this work is the first to explore the pipeline of differentiable textured mesh + motions (forward LBS skinning + non-rigids) and demonstrate its merits like fast training, quick inference, mesh representation, and template-free property.

**[Q3]** Extracting mesh for NeRF-based methods.

Every time a new pose is given, the NeRF-based methods have to predict new density and color fields for mesh extraction, which is inefficient. One can only extract the canonical mesh, yet the skinning weights are either inaccessible for backward-skinning ones or maybe (partially) wrong after discretization, and the non-rigid motions may be lost or degraded due to discretization. Besides, the threshold and resolution of the marching cube algorithm are tricky to tweak for both good shape and appearance quality. To summarize, the post-processing of NeRFs leads to either quality degradation or loss of features (like non-rigid).

Additionally, we can deem our method as a mesh-, or quantization-aware framework for modeling non-rigid. Moreover, the learned motion dynamics not only help inference rendering but also facilitate the training as a result of better inter-frame correspondences, leading to better quality.

Reviewer comment (Experiments):
Comparison with other efficient frameworks? On the efficiency side, the joint optimization of mesh and SDF is interesting. However, there are also many other sparse structures used for speeding up the rendering efficiency like using layered mesh representation [] and sparse volumes [3, 4]. It would enhance the quality of the manuscript if some comparisons or discussions were added.

Author response:
**[Q4]** More comparisons.

InstantAvatar [3]  is designed for learning a neural field from a fixed-posed human-rotating monocular video or a synthetic monocular video without loose clothes, and it is unknown that it can handle non-rigid dynamics with large poses from multi-view datasets. We also do not find any qualitative results from InstantAvatar for extensive comparison. Furthermore, our rendering speed (100 FPS) is faster than InstantAvatar (15 FPS).

As for [4] and [5], they are very exciting industrial-standard research works that push the frontier of human digitization. However, the evaluation setting of [4] is unclear and [4] unfortunately does not conduct extensive comparisons of the benchmarks. The code of [5] is unavailable.

Ultimately, we have referred to the related literature and uploaded the newer version of the pdf.

Reviewer comment (Experiments):
I view the comparisons as not being extensive. For example, this concurrent work seems to solve the same problem [1], and they compare to other baselines which appear better than the ones used in this method. Additionally, there exist a number of methods which use the SMPL mesh for skinning weights driving motion [2][3] for which the quality appears quite good. Why are these approaches not compared to in this work, as it appears that these methods are able to generate better results qualitatively? If this method is focusing on human body avatars, why learn general skinning weights with a skeleton instead of using those from a known human body model, such as SMPL.

Author response:
**[Q1]** More Comparison.

Our method focuses on human body avatars, which is different from the aim of Articulated Point NeRF [1]. So we do not compare the general dynamic NeRF baselines in Articulated Point NeRF.

Both Neuman [2] and InstantAvatar [3] focus on monocular video reconstruction, and do not compare themself to the standard multi-view video benchmarks like H36M and ZJU-Mocap. So directly comparing with their methods is unfair. 

Neuman [2] is designed for learning human NeRF and scene NeRF from a single-view video, and it only learns a frame-dependent error-correction network for observation space. So quantitative comparison is unfair. We give a qualitative comparison of ZJU-Mocap in the updated supplementary. Our results preserve more details than Neuman.

InstantAvatar [3]  is designed for learning a neural field from a fixed-posed human-rotating video or a synthetic video without loose clothes, and it is unknown that it can handle non-rigid dynamics with large poses from multi-view datasets. We also do not find any qualitative results from InstantAvatar for extensive comparison. Furthermore, our rendering speed (100 FPS) is faster than InstantAvatar (15 FPS).


Ultimately, we have referred to the related literature and uploaded the newer version of the pdf.

Reviewer comment (Evaluation):
For the efficient training, I'm not sure why the representation actually trains faster? It seems to use the entire image as opposed to individual rays as in volume rendering, and marching to find the surface also requires a number of samples of the SDF representation. This results in each iteration having more rays, and just as many samples per ray, so I don't understand why the method actually trains faster.

Author response:
**[Q3]** Efficiency breakdown.


The efficiency of the method comes from two-fold: Both geometry and rendering. We provide a runtime breakdown and analyze the efficiency.
The runtime breakdown:
   
| Function                 | Time (ms)               |
|---------------------|---------------------|
| Extract Mesh (w/ geo. NF query, w/ non-rigid NF query)      | 11.39273 |
| Extract Mesh (w/o geo. NF query, w/ non-rigid NF query)       | 3.08209  |
| Render Mesh (w/ texture NF query, w/ env light query)   | 7.00726  |
| └──  (texture NF query)      | 4.08976  |

- "NF" means Neural Fields.
- Extract Mesh: For NF query in mesh extraction, we query both the canonical SDF field and the non-rigid motion field.
    -  In inference time, we only query the neural field once to get the canonical mesh and use the same mesh for the latter rendering. The motion field will be queried by the number of vertex of the mesh times.
    -  In training time, we need to query the NF for every optimization step to update the SDF and non-rigid fields.
    -  The insight is that, compared with NeRFs, our gradients only flow through the iso-surface, which is drastically less than NeRF, which where the gradients flow over the whole space.
- Render Mesh: 
    - For rendering an NxN resolution image, we only query **O(NxN)** times for the texture field thanks to the rasterization. While NeRF-based method needs to query **O(NxNxM)**, where M is the number of points per pixel (ray). The usage of tiny-cuda-nn, a highly optimized package for neural fields, further improves the speeds.
    - It involves both texture querying and env-light map querying.

Reviewer comment (Presentation):
It is difficult to read the main maniscript without supplementary materials. For example, crucial information such as Nerf structure should be included in the main maniscript.

Author response:
**[Q1]** Illustration of the model.

We have revised the figure accordingly, including annotations to help clarify our design. The revised version has been updated.

Reviewer comment (Experiments):
This work would be further strengthened if more in-the-wild results were provided.

Author response:
**[Q2]** In-the-wild data.

The challenge of the in-the-wild data are 1) inaccurate pose tracking and parsing, and 2) inaccurate global coordinate system.

We use datasets that are in the controlled environment, as this paper focuses on building a pipeline that enjoys fast training, quick inference, mesh representation, template-free (which can be extended for a variety of objects and avoid proprietary models), and feedforward LBS for training. 

Recently, [1] proposed an in-the-wild dataset. We will explore the in-the-wild scenario in the future.

Reviewer comment (Experiments):
As the framework takes into account environmental lights and the non-rigid model, how does this method perform on in-the-wild data?

Author response:
**[Q2]** In-the-wild data.

The challenge of the in-the-wild data are 1) inaccurate pose tracking and parsing, and 2) inaccurate global coordinate system.

We use datasets that are in the controlled environment, as this paper focuses on building a pipeline that enjoys fast training, quick inference, mesh representation, template-free (which can be extended for a variety of objects and avoid proprietary models), and feedforward LBS for training. 

Recently, [1] proposed an in-the-wild dataset. We will explore the in-the-wild scenario in the future.

Reviewer comment (Experiments):
Can this method be used for avatar modeling with soft cloth (e.g. wearing a dress)?

Author response:
**[Q3]** Soft clothes.

Yes, our method can be applied to soft clothes thanks to the non-rigid modeling and the template-free property. In our supp. video, there are avatars with loose clothes in clip 01:06 - 01:24 (or "ema.supp.representation_visualization.mp4" in supp). The mesh visualization offers a better view. Admittedly, modeling clothes dynamics like dress is a hard research problem requiring further research.

Reviewer comment (Evaluation):
What is the geometric precision of this method?

Author response:
**[Q4]** Geometry quality.

In supplementary materials, Sec. H, Mesh visualization, we qualitatively visualize the canonical meshes. Note that the number of faces for each mesh is quite small. Though increasing the resolution of tetrahedra grids may improve the details of both geometry and materials, we do not conduct this experiment for it is orthogonal to our technical contributions.

Reviewer comment (Experiments):
The EMA method employs neural networks to encode canonical geometry, materials, and motion models. In practical applications, the complexity of these neural networks may affect the inference speed. Have the authors conducted experiments in this regard, or attempted to use simpler or more efficient neural network architectures to balance the inference speed and reconstruction quality?

Author response:
**[Q1]** Efficiency wrt. NNs for geometry, appearance, and motions as they may affect inference speed.

We follow the design of NN of the previous arts. Since they are fast in our pipeline, we do not further tweak or improve the architecture of the fields. Besides, during inference, the mesh is only extracted once. So there is no NF overhead for mesh extraction. For light and appearance, we only query O(NxN) thanks to the rasterization technique.

Reviewer comment (Evaluation):
The EMA method learns a fixed environment light and uses a Physically-Based Rendering (PBR) material model. In practical applications, would this lighting and material modeling approach potentially limit the inference speed to some extent? Is it possible to use simpler lighting and material models to further improve efficiency?

Author response:
**[Q2]** Efficiency wrt. lighting and PBR.

It is possible to use simpler ones like Spherical harmonic lighting [1], but it is not necessary as the computation overhead is, again, negligible. In addition, PBR and Env light are industrial standards in Computer Graphics, e.g., they are used in video games to achieve real-time and photo-realistic rendering [2].

Reviewer comment (Evaluation):
The EMA method employs pose-dependent non-rigid offsets to compensate for non-rigid cloth dynamics. Does this modeling approach increase computational complexity and impact inference speed? Are there any other more efficient methods to handle non-rigid cloth dynamics? Judging from the video results, it seems that EMA might also be limited in terms of non-rigid cloth dynamic modeling?

Author response:
**[Q3]** Efficiency wrt. modeling dynamics

We model dynamics for better reconstruction and animation. Since we only move the coordinates of the extracted mesh, the time is negligible. To the best of our knowledge, there is no literature providing even efficient approach to modeling the clothes dynamics. In our supp. video, there are non-rigid offsets in clip 01:06 - 01:24 (or "ema.supp.representation_visualization.mp4" in supp). The mesh visualization offers a better view. Admittedly, modeling clothes dynamics is a hard research problem requiring further research.

Reviewer comment (Evaluation):
In the experimental part, it can be seen that in the case of fast training (10 minutes), EMA has a satisfactory result (better than ARAH). However, under the same hour, the advantage seems not obvious. Additionally, when the pose tracking quality is poor, EMA is also greatly affected.

Author response:
**[Q5]** Compare with ARAH.

Although our method only achieves comparative performance with ARAH, the qualitative results of our method are better in Fig. 1 and Fig. 2, e.g., the rendering of ARAH is blurrier than ours. This indicates the drawback of current reference-based metrics.

Reviewer comment (Novelty):
Are there other methods that also use mixed representation training methods? The improvements brought by this method have not been well explained and proven in experiments.

Author response:
**[Q6]** Other mixed representation.

We refer to and discuss other mixed representation methods in the related works. The majority of them are volume-based, while our method is mesh-based. Compared with previous methods, the training speed is largely increased, the rendering is real-time, and the outputs are triangular meshes that are fully compatible with the industrial graphics pipeline. Here, the insight is that equipped with a differentiable meshes extractor and renderer, not only the geometry and appearance can be learned, but the motions, both rigid and non-rigid ones, can be learned in a feed-forward manner, which avoids proprietary human templates.