Paper: AvatarStudio: High-fidelity and Animatable 3D Avatar Creation from Text (ICLR-2024)

Reviewer comment (Experiments):
The text prompts are also simple.

Author response:
Thank you for your feedback regarding the simplicity of our text prompts. We agree that testing our method with more complex prompts is essential for demonstrating its robustness and effectiveness. We have experimented with more complex prompts for human avatar creation, as shown on our [project page](https://avatarstudio3d.github.io/) (video grid with caption "Avatar creation with more complicated prompts"). Our method has shown promising results, effectively aligning the generated avatar with the detailed descriptions in the prompts. This demonstrates the robustness and effectiveness of our method even with complex prompts.

Reviewer comment (Evaluation):
The time taking to create the avatar is 2.5 hours which is too much of time.

Author response:
Thank you for your feedback regarding the optimization time of our method. We acknowledge that the time taken to create an avatar is a significant factor in the practicality of our approach. However, we would like to highlight the optimization time for our method is comparable to that of concurrent works in the field of text-to-3D generation. For example, methods such as Magic3D, AvatarVerse and ProlificDreamer require around 2-5 GPU hours with a single A100 GPU; while DreamAvatar and DreamWaltz require 2 hours and 3 hours on a single 2080Ti GPU and 3090 GPU, respectively. We have made efforts to reduce the optimization time of our method. By reducing the training steps (from 10,000 to 5,000 for the coarse stage, and from 3,000 to 2,000 for the fine stage), we have managed to decrease the optimization time from 2.5 hours to around 1 hour. The results, as shown on our [project page](https://avatarstudio3d.github.io/) (video grid with caption "Generations with fewer optimization steps"), are still comparable, demonstrating the effectiveness of our method and its potential to achieve more efficient results. In the future, we plan to explore other methods to further reduce optimization time, such as using more efficient Gaussian Splatting representation and stronger human prior and guidance.

Reviewer comment (Experiments):
While the ablation shows favorable results, it is not clear why the coarse and fine stages are separately required.

Author response:
Thank you for your question regarding the necessity of our coarse-to-fine stages. The rationale behind this design choice is to address the limitations of each individual stage and to leverage their strengths in a complementary manner. The coarse NeRF stage struggles with rendering high-resolution images due to the significant computational cost of volume rendering, which limits its ability to learn detailed generation quality. On the other hand, the DMTet-only stage, while capable of achieving high-resolution rendering owing to its explicit mesh representation, is difficult to optimize from scratch and can result in geometry collapse (as shown in Fig. 6(a) of our ablation study). To address these issues, we levereage a coarse-to-fine pipeline. In the coarse stage, we use NeRF to train a coarse but reasonable result. This result is then used as an initialization to further optimize the articulated DMTet representation in the fine stage, achieving higher quality texture and geometry generation as well as avatar animation. We have provided more results from NeRF-only and DMTet-only methods on our [project page](https://avatarstudio3d.github.io/) (video grid with caption "Comparison between different 3D representations"). As can be seen, the NeRF-only generation results are relatively coarse and noisy geometry with floating artifacts can be observed, while the DMTet-only ones are not plausible. This further illustrates the necessity of our design choice.

Reviewer comment (Evaluation):
The skinning is borrowed from SMPL which is coarse. The nearest vertices based weight can create a deformation problem. How that is not impacting?

Author response:
Thank you for your question regarding the impact of using SMPL for skinning and the potential deformation problems due to the nearest vertices based weight.
1. In this work, we use a relatively simple animation technique, i.e., SMPL-guided deformation. Despite its simplicity, it has proven effective for avatar animation, even for some avatars wearing loose clothing, as shown in the results ("A ballerina") on our [project page](https://avatarstudio3d.github.io/) (video grid with caption "More animation results"). We hypothesis that there are two main reasons why the potential deformation problems are not significantly impacting our results. First, we use a pre-trained 2D diffusion model to provide SDS guidance, which can correct areas where the SMPL deformation is inaccurate, leading to better animation results. Second, we use a strong DensePose conditioned guidance, which provides strong pose information, helping the model learn what the avatar should look like in different poses. These priors can help the model achieve plausible results on top of the coarse SMPL-guided animation.
2. Additionally, we have developed a non-rigid deformation module to learn residual deformation to compensate for areas where SMPL deformation is inaccurate. We found that adding such deformation makes the animation results more robust, especially for avatars with complex clothing, as shown on our [project page](https://avatarstudio3d.github.io/) (video grid with caption "More animation results-AvatarStudio with non-rigid deformation"). 
Moving forward, we plan to explore more effective driving techniques and more robust driving signals to further improve the quality of our animations. We appreciate your insightful question and look forward to any further queries you may have.

Reviewer comment (Evaluation):
It is not clear why densepose is important than skeleton. Is it because skeleton is noisy? It is not clear from Fig 6(b) why the control net is behaving well with densepose. The argument is not strong. Can't there be other cases where densepose is failing?

Author response:
Thank you for your question regarding the importance of DensePose over the skeleton in our method.
1. Skeleton-based guidance is relatively sparse. the same skeleton-represented pose can potentially map to multiple real human poses, resulting in inaccuracies in the generated 3D avatars. The result in Fig 6 (b) demonstrates this. Moreover, this sparsity can also lead to ambiguity in distinguishing between frontal and back views, a phenomenon often referred to as the "Janus problem". On the other hand, DensePose provides a more detailed and accurate description of a person's pose. Its dense nature allows for a more precise mapping between the guidance and the actual human pose, thereby alleviating the aforementioned Janus problem and enhancing pose control accuracy. We have provided more results on our [project page](https://avatarstudio3d.github.io/) (video grid with caption "Comparison between different SDS guidances") to demonstrate this. Results based on skeleton guidance exhibit the Janus problem, while our proposed DensePose guidance method effectively mitigates this issue, yielding correct results.


2. However, we acknowledge that DensePose is not perfect. We have found that it is not precise enough for controlling avatar's hand creations, yielding artifacts. For example, in the supplementary materials for the "business woman" prompt, two wallets appear on the hands. A promising solution is to employ OpenPose hand model as guidance, which contains 21 finger joint keypoints, to model the fine-grained details of hands.

Reviewer comment (Writing):
How "p" is used in equation 4?

Author response:
Thank you for your question regarding the use of "p" in Equation 4. In our method, "p" represents the parameters of the SMPL model. It is used to generate the DensePose control signal, denoted as $I_{cond}$. In Equation 4, $I_{cond}$ serves as the control signal provided to the DensePose ControlNet. This is used to calculate the SDS loss for gradient backpropagation and model parameters update. We acknowledge that this could have been clearer in our original manuscript and will clarify this point in our revision.

Reviewer comment (Experiments):
What is the comparison between CFG and CFG rescale trick?

Author response:
Thank you for your question regarding the comparison between the CFG and the CFG rescale trick. Current text-to-3D works use a large CFG scale when optimizing avatar representation with SDS to achieve reasonable results. However, a large CFG scale can produce severe color saturation, making the generated avatars look unreal. To alleviate the color saturation issue, we apply the CFG rescale trick from [1] to rescale x_t back to the original standard deviation w.r.t. a rescale strength $\phi$. Please refer to [1] for more details. This method effectively reduces color saturation and produces a more natural appearance in the generated avatar as shown in Fig.6(c). We will add a clearer introduction to the CFG rescale trick in revision.

[1]  Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and
sample steps are flawed. arXiv, 2023b.

Reviewer comment (Experiments):
Why Avatarverse has produced red belt instead of black belt in second row of Fig 3? Has the comparison been done with proper negative prompt too?

Author response:
Thanks for your comments. In the manuscript, we directly use the results from AvatarVerse paper for comparison. For a fair comparison, we also removed the negative prompt in our method. Despite removing the negative prompt, our results remained consistent, i.e., the color of the belt aligns with the text prompt. As for AvatarVerse's generation of a red belt given a black belt prompt, we hypothesize that this could be due to the training data bias in their ControlNet, leading to the generation of a red belt instead of a black one.

Reviewer comment (Writing):
It is not clear how the method is handling the janus problem.

Author response:
Thank you for your question regarding how our method handles the Janus problem. The Janus problem, a common issue in text-to-3D generative models, arises due to the lack of multi-view supervision. This problem often results in the model regenerating content described by the text prompt in different views, leading to multi-view inconsistency. To alleviate this issue, we render the SMPL mesh into DensePose conditions to guide the 3D human generation (see Fig. 6 (b)). This approach allows us to obtain precise view correspondence across views and between the 2D view and the 3D space. In contrast, skeleton conditions, which only have 18 keypoints (see Fig. 6 (b)), can only provide sparse correspondence between different views and from 2D view to 3D space. This sparsity often leads to difficulties in distinguishing between the front and back views of the generated human, resulting in problems such as multiple faces and unclear facial features in 3D avatar generation. We provide more comparison results on our [project page](https://avatarstudio3d.github.io/) (video grid with caption "Comparison between different SDS guidances"). For instance, in the case of the prompt "Albus Dumbledore", when we provide a back-view skeleton map projected from SMPL mesh as the conditional image and perform text-to-image generation, a frontal face appears in the generated image. This is because the keypoints reside in the SMPL mesh, and it is difficult to determine whether they are occluded, and thus guide the text-to-3D model to yield incorrect back-view image. In contrast, the DensePose control signals provide a more detailed and accurate description of a person's pose and view and thus guide the model to generate reasonable 3D avatar results, effectively mitigating the Janus problem. This has been demonstrated in the results on our project page. We hope this explanation addresses your question.

Reviewer comment (Novelty):
Generating 3D models in a coarse-to-fine manner with two representations is not a new idea. In fact, it has already been proposed in ProlificDreamer [Wang et al, 2023], which also leverages NeRF for initialization and uses DMTet for further refinement. Therefore, I don't think it can be regarded as a technical contribution.

Author response:
Thank you for your valuable feedback. We acknowledge that the concept of generating 3D models in a coarse-to-fine manner is not new, and we did not intend to claim this as our novelty. In the revision, we will tune down our claim to be more in line with your comments. However, we believe that our method introduces significant advancements and unique contributions in the context of animatable human avatar generation, which we would like to elaborate on:
1. While previous methods have explored the coarse-to-fine pipeline for 3D generation, they mainly focus on static objects, which are not animatable. In contrast, our method extends this coarse-to-fine paradigm to animatable human avatar generation, which is a more challenging task. This is a significant step forward as it allows for more dynamic 3D modeling. In our refinement stage, unlike ProlificDreamer that uses DMTet for image quality enhancement on static objects, we propose an articulated DMTet representation. This novel representation, driven by SMPL-guided articulation, enables the creation of animatable human avatars, a feature not present in previous text-to-3D works.
2. Having established this novel representation, we found that it alone was not sufficient to achieve optimal performance for animatable 3D avatar generation. Therefore, we introduced several improvements to enhance the efficacy of our approach. Specifically, we propose the use of DensePose-conditioned ControlNet to guide the human generation process. This leverages the DensePose human prior as multi-view supervision, providing a more accurate and dense description of a person's pose, thereby enhancing pose accuracy and alleviating the Janus problem. To further improve the performance of our method, we incorporated a series of techniques into our framework, such as CFG rescale, part-aware super-resolution, coarse-to-fine optimization, and dual space training. These techniques, when combined, effectively enhance our method and enable us to achieve plausible results.

Reviewer comment (Novelty):
Jointly optimizing the textured avatar mesh in both deformed and canonical spaces is also not new and has been proposed in DreamAvatar, which also proposes to perform SDS supervision in both canonical space and posed space.

Author response:
Thanks for pointing out this issue. We acknowledge that the concept of performing SDS supervision in both spaces has been previously proposed in works such as DreamAvatar. We will add references in the revision. Also, dual-space training is not intended to be claimed as a novelty of this work. Rather, we adopted it as a technique to improve the accuracy of animation. Our main contributions lie elsewhere, specifically in the development of an articulated DMTet representation for animatable human avatar generation, the introduction of DensePose-conditional ControlNet guidance, and the incorporation of techniques such as CFG rescale, part-aware SR, dual space training, and coarse-to-fine optimization. These methods, when combined, have allowed us to significantly advance the state of the art in animatable 3D avatar generation. We hope this response addresses your concerns and we appreciate your feedback as it helps us improve the clarity of our work.

Reviewer comment (Novelty):
Overall, although this paper demonstrates good results, its technical novelty is not strong enough to me. I feel that the proposed method is more like a combination of existing techniques and tricks.

Author response:
Thank you for your feedback. We understand your perspective regarding the novelty of our work. While it's true that our method incorporates existing techniques, the process of combining them effectively to achieve high-quality results is a non-trivial task. We would like to highlight the following key points to clarify our contributions: 1) we propose a novel articulated DMTet representation for the challenging 3D animatable human avatar creation. This novel representation, driven by SMPL-guided articulation, enables the creation of high-fidelity and animatable human avatars, a feature not present in previous text-to-3D works. 2) Optimizing the articulated 3D representation is not sufficient to achieve significant performance for animatable 3D avatar creation due to the lack of effective 2D pretrained guidance. To address this, we propose using a DensePose-conditional ControlNet to guide the human generation process, which provides a more accurate and dense description of a person's pose, thereby enhancing pose accuracy and alleviating the Janus problem. 3) To further improve the performance of our method, we incorporated a series of techniques into our framework, such as CFG rescale, part-aware super-resolution, coarse-to-fine optimization, and dual space training. 4) Our method achieves promising avatar animation and stylized avatar creation. Thus, we believe integrating all these techniques and demonstrating their effectiveness in animatable human avatar creation constitutes a valuable contribution, as also highlighted by **Reviewer L83u**.