Paper: AniHead: Efficient and Animatable 3D Head Avatars Generation (ICLR-2024)

Reviewer comment (Evaluation):
All generations have the same skin color, e.g. Mark Zuckerberg and Morgen Freeman in Figure 8.

Author response:
__A1__: Besides texture resolution, reviewers are also concerned about the skin color generated by our method. 
This issue is partly caused by the lighting settings in rendering software, such as Blender. We have updated the relevant images in the main paper and the appendix to better highlight the skin color variations.

Reviewer comment (Evaluation):
Why do the generations have very limited skin color variations?

Author response:
__A1__: Besides texture resolution, reviewers are also concerned about the skin color generated by our method. 
This issue is partly caused by the lighting settings in rendering software, such as Blender. We have updated the relevant images in the main paper and the appendix to better highlight the skin color variations.

Reviewer comment (Experiments):
The authors mention that they use SDS optimization to obtain 600 samples while selecting only 50 samples for training to “ensure a balance of gender, ethnicity, and age”. Why selection is needed, and why not just balancing the input text prompts? Also, 50 training samples sound very limited. Why not use more samples?

Author response:
__A2__: We use only 50 samples because, generally, LoRA training requires just a few dozen samples. Besides, due to the inductive bias of SD, the generated data by SDS is not very fair in terms of gender, ethnicity, and age. Therefore we perform a simple selection from 600 sample to form our training set. Moreover, we empirically find that using more sample for LoRA finetuning not only leads to longer finetuning time, but also results in overfitting problem. The finetuned model will concentrate on generating normal-looking humans, while having poor generation of contents that are not part of the dataset and require large areas of color blocks, such as The Joker and Batman.

Reviewer comment (Experiments):
How does the proposed method compare with the SDS optimization pipeline which is used for training data generation?

Author response:
__A3__: We have updated the results in our appendix. As shown in Fig.10 in the appendix, the data generated by SDS does contain some blurry samples, but the final generated results are clearer, thus showing the effectiveness and necessity of our method.

Reviewer comment (Novelty):
The novelty of the proposed method appears slightly limited. Similar to DreamFace, pretrained CLIP and LDM models are used in the avatar generation with independent geometry (shape) and texture generators. The idea of using a mean texture token is novel, but similar ideas have been explored in the form of pre-defined identity token in DreamBooth, and domain-specific prompt tuning in DreamFace.

Author response:
__A1__: Thank you. We would like to stress that using pretrained foundation models like CLIP and LDM is common in the current vision tasks due to their strong capability. We have not listed such techniques as our contribution. Instead, as mentioned in the general response, our novel pipeline for 3D head avatar generation along with the data-free training strategy and the usage of mean texture token for representing general human facial details are the key contribuion of our paper.

Reviewer comment (Novelty):
Introduction point 2 mentions challenges in animation due to implicit representations as a limitation of existing text-to-3D methods. However, DreamFace uses the ICT-FaceKit face model that can be integrated with existing animation pipelines, so the benefit obtained from using the FLAME model in the current work is not clear.

Author response:
__A3__: Thank you. The advantage of FLAME introduced in our paper is in comparison to latent representation methods like Nerf and DMTet. With this linear shape parametric model, we can easily control the shape using 100-dimensional parameters and easily integrate with a CG pipeline to achieve downstream tasks, such as video-driven animation.

Both our FLAME model and DreamFace's ICT-FaceKit belong to the 3DMM parametric model category and share the advantages mentioned above. The differences lie in:

(1) The texture coverage of ICT-FaceKit does not include the entire face, covering only the ears and the frontal face without eyes, while our FLAME model's texture covers the entire head, which can be seen in the videos in our supplementary material.

(2) Our usage of the 3DMM model also differs from DreamFace. DreamFace samples shape parameters from a multivariate normal distribution N(0,1) to obtain one million candidates, followed by 300 carving steps. In contrast, we design the AniHead model pipeline to directly learn shape parameters, inferring the shape parameter vector directly from the prompt. This not only results in higher shape-text matching (as shown in Tab.1 and Fig.3) but also reduces inference time (ours can generate shape within seconds).

Reviewer comment (Experiments):
Although the paper claims a reduction in inference time, there are doubts about the generalization ability of the method in generating arbitrary high-fidelity avatars of varying age, skin colours, etc., given that the training data consists of manually selected 50 training samples generated by SDS optimization.

Author response:
__A7__: Thank you. We would like to kindly point out that 'data-free' refers to the fact that manual data collection is not required. In the SDS generation process, the only preparation needed is providing a prompt, such as "a portrait of {name} with neck and shoulder, without hair", which can be easily created. Moreover, the generalization ability has been shown in results in Fig.4. Given limited training data generated by SDS, these results further prove the effectiveness of our proposed pipeline.

We would like to clarify that the Stable Diffusion model is freeze during the SDS optimization process, because SDS only needs to leverage the prior information from Stable Diffusion.

Reviewer comment (Writing):
Unclear writing on Page 2 — the phrase “need for a cumbersome two-stage generation process” is unclear; what two-stage generation process?

Author response:
__A5__: Thanks for pointing out. We have updated these problems in the new version.

Reviewer comment (Writing):
Unclear writing on Page 2 — the phrase “meticulously crafted to encapsulate essential human head texture information” is unclear; how is this achieved?

Author response:
__A5__: Thanks for pointing out. We have updated these problems in the new version.

Reviewer comment (Writing):
Unclear writing on Page 3 — the phrase “we further propose other specific design to generate high-quality animatable 3D head avatars” is unclear; what designs are meant?

Author response:
__A5__: Thanks for pointing out. We have updated these problems in the new version.

Reviewer comment (Writing):
The sentence “the renewed text prompts can contribute to fine-grained personalized characteristics with high fidelity of identity” doesn’t make sense.

Author response:
__A5__: Thanks for pointing out. We have updated these problems in the new version.

Reviewer comment (Writing):
The phrase “common texture-wise features shared by human beings” is unclear.

Author response:
__A5__: Thanks for pointing out. We have updated these problems in the new version.

Reviewer comment (Writing):
Epsilon is not defined in Equation 1.

Author response:
__A5__: Thanks for pointing out. We have updated these problems in the new version.

Reviewer comment (Writing):
Typo — “our propose generalized shape” on Page 5.

Author response:
__A5__: Thanks for pointing out. We have updated these problems in the new version.

Reviewer comment (Writing):
Typo — in Equation 2, \phi() needs to be replaced by e_\phi().

Author response:
__A5__: Thanks for pointing out. We have updated these problems in the new version.

Reviewer comment (Novelty):
Missing citation — “Existing methodologies [??] typically leverage SDS”.

Author response:
__A5__: Thanks for pointing out. We have updated these problems in the new version.

Reviewer comment (Novelty):
Missing citation — “remarkable strides achieved in diffusion-based text-to-3D models [??]”.

Author response:
__A5__: Thanks for pointing out. We have updated these problems in the new version.

Reviewer comment (Writing):
Missing citation — “While these [??] SDS-based approaches”.

Author response:
__A5__: Thanks for pointing out. We have updated these problems in the new version.

Reviewer comment (Writing):
Missing citation — “Leveraging readily available off-the-shelf models [??]”.

Author response:
__A5__: Thanks for pointing out. We have updated these problems in the new version.

Reviewer comment (Novelty):
Missing citation — “[Articulated Diffusion]”.

Author response:
__A5__: Thanks for pointing out. We have updated these problems in the new version.

Reviewer comment (Experiments):
Experimental results issue — 3D view (other than frontal) should have been included, similar to existing works such as DreamFace. In the absence of a supplementary video it is hard to assess the qualitative results.

Author response:
__A6__: We have provided the required videos in the supplementary material. The results are consistent with the original results in our paper, supporting the efficacy of our method.

Reviewer comment (Reproducibility):
The description “data-free” strategy appears ambiguous as it also mentions that SDS is used to generate training data. More clarity is needed on the training strategy in Section 3.3. Is a pretrained stable diffusion model being finetuned for the training data preparation? What kind of candidate text prompts are used for the geometry and UV texture generation (few examples)? How is it ensured that the “training data” generated using SDS is sufficiently accurate for generalized performance at inference time?

Author response:
__A7__: Thank you. We would like to kindly point out that 'data-free' refers to the fact that manual data collection is not required. In the SDS generation process, the only preparation needed is providing a prompt, such as "a portrait of {name} with neck and shoulder, without hair", which can be easily created. Moreover, the generalization ability has been shown in results in Fig.4. Given limited training data generated by SDS, these results further prove the effectiveness of our proposed pipeline.

We would like to clarify that the Stable Diffusion model is freeze during the SDS optimization process, because SDS only needs to leverage the prior information from Stable Diffusion.

Reviewer comment (Reproducibility):
Which pre-trained LDM models are used for finetuning?

Author response:
__A8__: We use the pretrained LDM checkpoint stable-diffusion-2-1-base from stabilityai, and we found that stable-diffusion-v1-5 from runwayml also works well. Therefore, the choice of LDM base model checkpoint does not significantly impact the performance of our method.

Reviewer comment (Reproducibility):
The significance of the mean-texture token is not clear from the results. How is the mean-texture token prompt obtained at test time?

Author response:
__A9__: We would like to clarify that the mean-texture token is not a trainable parameter in our work. It is represented by a rare token, specifically 'shs'. During inference, simply adding this token to the beginning of the text input is enough for the model to understand the corresponding semantic meaning. As for the effectiveness of the mean texture token, we have showcased in Fig.6 that this unlearnable token can better represent human facial texture than natural language.
We also demonstrate in Fig.11 in the appendix that, with the mean texture token, the UV geometry of delicate parts may be misaligned, such as the nose and eye regions shown in Fig.11. In contrast, our design leads to better UV map positioning. As we have analyzed, the mean texture token strengthens the UV relation map.

Reviewer comment (Evaluation):
Is the mean-texture token sufficient to finetune a pretrained LDM (trained on diverse images) to the specific task of the Face UV texture generation? How is it ensured that the generated texture is consistent with face geometry? In the absence of UV texture ground truth to finetune pretrained LDM for texture, how is the accuracy ensured at inference time?

Author response:
__A10__: Indeed, in our training process, we do not use ground truth datasets. Instead, we treat the optimization results from SDS optimization as ground truth, given the strong capabilities of SDS. Moreover, this approach allows us to achieve data-free training. Our experimental results in Fig.10 demonstrate that even if the training data contains some blurry examples, it does not negatively impact the final generated results. The generated images exhibit clear skin textures and patterns.
Whereas some of the SDS generated data are blurry in details, the generated results are clear and can easily be generated to out-of-domain examples such as 'The Joker'.

Reviewer comment (Experiments):
The paper mentions “we set this parameter to a relatively low value and obtain more realistic, real-life outcomes.” Is there any ablation done on the guidance scale parameter to justify this statement?

Author response:
__A11__: We have updated Fig.7 in our appendix to demonstrate that relatively low CFG value (7.5) leads to more realistic, real-life outcomes.