Paper: SweetDreamer: Aligning Geometric Priors in 2D diffusion for Consistent Text-to-3D (ICLR-2024)

Reviewer comment (Evaluation):
The quantitative results are not satisfying. The evaluation scores reported in Table 1 only show the performance of inconsistency. Did the author compare with other methods by use of other metrics (e.g., R-Precision in DreamFusion)?

Author response:
A： We thank the reviewer for this insightful suggestion.  For more discussion, please refer to the feedback of Reviewer nptq. During the rebuttal, we obtained the numerical results of appearance-related metrics. We report the R-precision as your suggestion. The results are presented below and in Section A.7 and Table 3 in the revised paper.

Reviewer comment (Experiments):
A simple baseline is missing. Have the authors considered directly replacing the original SDS loss with the aligned geometric SDS loss and calculated the inconsistencies in Table 1?

Author response:
A: Thanks for pointing this out. When generating the 3D results shown in the submission, AGP has actually dominated the geometry modeling in both NeRF- and DMTet-based pipelines, as we set a large weight  (5x \lambda_ori) for \lambda_align at the beginning of the optimization. As demonstrated in Figure 9 of the revised version, the overall shape generated by AGP-only is almost the same as the AGP + DeepFolyed IF SDS, except for minor details. As a result, the qualitative results are the same as the Ours NeRF-based IF. Please refer to Figure 9 in the revision paper for further visual results at each stage in the generation pipeline.

Reviewer comment (Reproducibility):
How long does the fine-tuning for a stable diffusion model take?

Author response:
A: Since we only use coarse geometries, i.e., low-resolution 64x64 CCM maps, to fine-tune the SD, the training process is highly efficient. The whole fine-tuning process takes about one day on 8 A100 GPUs for reasonably good results, and the final model used in the submission was trained for two days.
We added more details about finetune in the revision A.2.

Reviewer comment (Experiments):
There are no ablation studies in this paper. For example, the author claims that the SDS loss from the Canonical Coordinates Map can avoid overfitting the 3D training data. However, there are no experiments to support this idea. Moreover, all the design choices in this paper including how to inject the camera information into the diffusion model are not ablated.

Author response:
**Q: Ablation studies and the generalization ability of the proposed method.**

A: Please refer to the “More ablation studies” and “Technical novelty” in the common response for the ablation studies and generalization ability,
respectively.

Reviewer comment (Novelty):
Introducing a view-dependent SDS loss to tackle the multi-face issue has been proven in Zero-123 and Magic-123, which use a view-aware SDS loss to generate a 3D object. SweetDreamer uses a Canonical Coordinates Map SDS loss rather than RGB SDS loss, so the novelty and technical contribution is limited.

Author response:
**Q: Novelty.**

A: Please refer to the “Technical novelty” in the common response.

Reviewer comment (Experiments):
SweetDreamer uses Objaverse data to fine-tune the Stable Diffusion model to predict the Canonical Coordinates Map. However, defining a canonical coordinate system uniformly across diverse categories is challenging. The authors' choice to whittle down the Objaverse dataset from 800k to 270k, in effect, results in significant data attrition. When compared with techniques that fine-tune based on multi-view images, this could constrain the method's generalizability.

Author response:
**Q: Filtering the data and not using all may constrain the generalization ability.**

A: Our filtering process mainly filters out noisy 3D data, such as low-poly shapes, point clouds, unbounded scenes, etc, using some simple and heuristic rules. In addition, we also filtered out some obviously wrongly oriented samples in certain categories (e.g., characters) based on the orientation of their Axis-Aligned Bounding Box (we missed this filtering in the submission and have revised accordingly). While we agree that the data can potential affect the fine-tuning process, we also believe that inherently the key is the diversity and quality of the data rather than simply the absolute amount of data. Moreover, we only use the coarse geometries in the dataset for aligning the diffusion priors at the coarse geometry level, such “less is more” way of using the data has demonstrated its efficacy in alleviating the multi-view inconsistency issues without compromising significantly the generalizability of powerful diffusion priors in terms of highly varied appearance and geometric details, as evidenced by extensive successful results in the submission.

In addition, recent works have also found that 2D diffusion models are already very powerful, and the results of lightweight fine-tuning using a small number of high-quality 3D data may even be better than training with a larger amount of data (such as Wonder3D and Instant3D). This is consistent with our experimental experience, and we have added more discussions on this in the revision (See Appendix A.5). 

Last, we have also added a discussion on the generalizability of our method in Appendix A.4. More results, that showcase the ability to generalize to highly varied objects unseen in the dataset, can also be found on this anonymous website. http://3dsweetdreamer.github.io/nerf-based-gallery_0.html.

Reviewer comment (Experiments):
The experiments have some weaknesses. Only 80 results from 80 text prompts are generated using each method. The results can be noisy based on the small amount of data.

Author response:
**Q: Limited experiments.**

A: Almost all existing text-to-3D methods, including ours, share a common limitation that the optimization-based generation process could take hours for obtaining a single 3D result. For example, DreamFusion takes 1.5 hours on 4 TPUs, Magic3D 40 minutes on 8 GPUs, Fantasia3D 1 hour on 4 GPUs, and ProlificDreamer approximately 8 GPU hours. Due to this very reason, while DreamFusion managed to present results obtained using 398 prompts, the number of results presented in most of these works is indeed limited, such as 40 prompts in MVDream and a dozen in Fantasia3D and ProlificDreamer. 

Yet, in our submission, we have presented around 100 results obtained using various arbitrary prompts in the video and supplementary, and compare to baseline methods on 80 prompts. Therefore, we would like to emphasize that our experimental results are sufficient and should not be considered as a weakness, as these prompts are several times more than those used for evaluating previous methods. During the rebuttal, we have increased the number of text prompts up to 150 (see anonymous links http://3dsweetdreamer.github.io/nerf-based-gallery_0.html and https://3dsweetdreamer.github.io/dmtet-based-gallery_0.html) but are indeed limited by the computing resources for more, we are happy to add more results using more prompts to the revision in the next stage.

Reviewer comment (Evaluation):
The definition of "3D consistency" in the quantitative experiment is vague ("e.g., multiple heads, hands, or legs"). For example, when the generated object is blurred, or does not have the concept of "head, hand, leg" (e.g., building), or some part of the generated object is twisted but not duplicated or missing, how can we determine the 3D consistency is correct or not?

Author response:
**Q: Criteria for determining 3D consistency in quantitative evaluations.**

A: We agree that the definition of the 3D consistency can be vague to describe, and currently there is a lack of objective metrics that can be computed automatically to determine if a 3D object is multi-view inconsistent or not. This is why we rely on human users for quantitative evaluation of the 3D consistency of the generated results. Specifically, the human participants involved in Table 1 of the paper are researchers from academic laboratories, with highly relevant research backgrounds such as Computer Vision, Computer Graphics and Visual Computing, while the users involved in obtaining the preference rate charts are from loosely related backgrounds such as CS and EE (a more noisy evaluation process, yet our method consistently outperforms other baselines). Furthermore, we have included the user study interface in the appendix of the revised paper for clarification.

Reviewer comment (Evaluation):
It is better to also report the appearance or texture quality measurements, even though this paper mainly focuses on improving the 3D consistency. It can show whether (and if yes, to what extent) the proposed method will influence the appearance or texture quality of the generated objects.

Author response:
**Q: Report the appearance quality.**

A: We thank the reviewer for this insightful suggestion. As also mentioned in the common response, our method can preserve the high realism of the 2D results of the powerful text-to-image foundation model, as we only align the geometric priors at the coarse geometry level. This is particularly attractive, as the 3D dataset (e.g., Objaverse) in its current form falls short of covering samples as diverse as the 2D text-image dataset, particularly in terms of the appearance. During the rebuttal, we have obtained the numerical results of appearance-related metrics. As suggested by Reviewer UgSj, we use the R-precision, which is also used in related work. The results are presented below, and in Section A.7 and Table 3 in the revised paper. It is obvious that our method achieves the best performance on this metric over competing methods.
| Method           | CLIP B/32 R@1 | CLIP B/32 R@5 | CLIP B/16 R@1 | CLIP B/16 R@5 |
|------------------|---------------|---------------|---------------|---------------|
| Magic3D          | 60.1          | 71.5          | 62.7          | 77.7          |
| TextMesh         | 51.7          | 65.1          | 55.1          | 78.4          |
| SJC              | 40.2          | 51.2          | 52.5          | 62.5          |
| DreamFusion      | 59.7          | 70.2          | 61.6          | 74.3          |
| Ours (NeRF-based)| **77.5**      | **84.9**      | **88.7**      | **92.3**      |

Reviewer comment (Experiments):
The method is based on the assumption that all objects within the same category adhere to a canonical orientation in the training data. However, Objaverse does not align the zero poses of the objects. It is interesting to see whether only using a smaller subset, where objects have a canonical orientation, can lead to better results or not.

Author response:
**Q: Whether using a small amount of high-quality data for fine-tuning will improve the performance.**

A:  In the submission, our model is fine-tuned using 3D objects obtained by filtering the Objaverse via some simple and heuristic rules. In addition, we also filtered out some samples in certain categories (e.g., humans) simply based on the orientation of their Axis-Aligned Bounding Box (we missed this filter in the submission and have revised accordingly). After these filterings, most of the data have been canonically aligned (around 80%, estimated based on statistical random samplings), and the rest wrongly oriented objects have only a minor impact on the performance of our model, as manifested by extensive successful results in the submission.

In addition, as suggested, we have also trained our model on a carefully and manually selected dataset, which consists of 20,000 objects. The resulting model is still able to generate highly varied 3D results, with detailed appearances (See A.5 and Figure 10 of the revised paper). While we have not observed significant improvement on this setting, we believe that using more well-structured data could potentially benefit the 3D generation and will perform a more thorough study in the next stage.

Reviewer comment (Reproducibility):
"Ours (NeRF-based full) using DeeFloyd IF first and then Stable Diffusion". How is it implemented in detail? What is the method (Ours (NeRF-based)) used in user study (NeRF-based IF or NeRF-based full)?

Author response:
**Q: More implementation details.**

A: Ours (NeRF-based full) uses the DeepFloyd IF model first and then using SD. While DeepFloyd IF can be a more powerful model for text-to-image, it can only perform at the 64 x 64 resolution on our side due to its use of much more computational resources. For a fair comparison, Ours (NeRF-based) in the user study is conducted using only DeepFloyd IF, which is the same as other competing methods.

Reviewer comment (Experiments):
How are $\lambda^{ori}$ and $\lambda^{align}$ chosen?

Author response:
**Q: Hyperparameters.**

A: All results presented in the paper were generated using the same hyperparameters, without the need for adjusting individually for each generation. More specifically, at the beginning of the generation we guide the generation to focus more on the geometry, so the $\lambda^{align}$ is set to a larger value (5x $\lambda^{ori}$) and then decreased to 0.5x for higher-quality appearance.

Reviewer comment (Experiments):
The user study interface can be shown in the appendix (How did the interface show users to "consider only the 3D consistency"? How are the users chosen? Do they have the concept of 3D consistency?)

Author response:
**Q: More details about the interface of the user study and the criteria for user selection.**

A: We added more details and the figure of the interface for user study in the appendix. Please kindly see the feedback in the revision.

Reviewer comment (Experiments):
My another question is: if objaverse is used to finetune SD for CCM generation, will the generalization ability be decreased? As objaverse is lacking diversity especially on complicated scenarios. More discussions are encouraged.

Author response:
Since we only finetune and constrain the geometric prior at a coarse resolution, the powerful generalization ability of the image diffusion models is largely preserved. The appearance modeling totally relies on the original diffusion model trained on billions of high-quality images. During the geometric modeling process, we were surprised to find that the finetune model still has strong generalization ability and can generate prompts such as Einstein wearing a gray suit and riding a bicycle. This is difficult to achieve with other existing methods. We have added more discussion on the generalizability in Appendix A.4.