Paper: Text-to-3D with Classifier Score Distillation (ICLR-2024)

Reviewer comment (Reproducibility):
In our reimplementation, the Janus problem is very serious.

Author response:
**[W2] Janus problem**

1. We would like to clarify that the core concepts and theoretical insights of our paper are not directly associated with the Janus problem. The fundamental challenge of the Janus problem stems not from the loss function, but from the limited 3D-aware capabilities inherent in the diffusion model. Our approach can be seamlessly integrated with techniques specifically designed to address the Janus problem, such as using the multi-view-diffusion model.
2. Here, we conduct a set of experiments using the pre-trained multi-view diffusion model from the concurrent work MVDream [1], combined with our CSD loss. We provide the qualitative results at this **[anonymous link](https://drive.google.com/drive/folders/1duCZB2KFozMGFOgQmdQGGiCwiZooEsOr?usp=sharing)**. The results illustrate that our loss function can generate realistic results, free from the Janus problem, indicating that this issue is more attributable to the diffusion model's 3D-awareness rather than our loss. **This set of experiments further strengthens the versatility and generalizability of our analysis.** We also provide the code at the same link in **W1** for your reference and we hope it can help the clarification.

Reference:

[1] MVDream: Multi-view Diffusion for 3D Generation. Shi et al., 2023

Reviewer comment (Experiments):
Lack of quantitative comparison with SOTA approaches such as Prolific Dreamer, Fantasia3D.

Author response:
**[W3] Quantitative comparisons with ProlificDreamer and Fantasia3D**

1. We would like to clarify that the reason that we omitted quantitative comparisons with ProlificDreamer and Fantasia 3D is that it is very time-consuming to evaluate ProlificDreamer based on public implementations (8h per prompt on an A800 GPU), and Fantasia3D requires manual tuning configuration parameters for each prompt (see their [note ("What do you want?")](https://github.com/Gorilla-Lab-SCUT/Fantasia3D#what-do-you-want) in their official implementation).  It is also worth noting that in their original paper, ProlificDreamer evaluated the performance on 15 prompts with a user study, and Fantasia3D evaluated the performance on 10 prompts with a user study.

2. Here, within the short period of rebuttal time and our constrained computational resources,  we randomly select 20 prompts and conduct experiments and comparisons with CSD, ProlificDreamer (using ThreeStudio), and Fantasia3D (using official code). The table below presents the results, showing that our method is generally comparable to Fantasia3D and ProlificDreamer. Moreover, our approach tends to perform better in comparison with other approaches under a more strict evaluation criterion like the CLIP score that directly measures the exact match between text prompts and generated images.  However, it's important to note that metrics like the CLIP score and R-precision primarily measure textual-image congruence and may not reflect 3D and texture quality. For a more comprehensive analysis, please refer to the detailed discussion below.

   |                |         |                |                     |
   | -------------- | :-----: | :------------: | :-----------------: |
   |                | **CSD** | **Fantasia3D** | **ProlificDreamer** |
   | **CLIP Score** |  81.6   |      77.8      |        80.2         |
   | **CLIP R@1**   |  96.2   |      93.8      |        96.2         |
   | **CLIP R@5**   |  98.8   |      100       |         100         |
   | **CLIP R@10**  |   100   |      100       |         100         |
   |                |         |                |                     |

3. **[Analysis of CLIP score for 3D evaluation]** It's important to note that in evaluating 3D synthesized results, the CLIP score may not fully reflect the quality for several reasons: (1) As an image-level metric, the CLIP score assesses each projected image independently, thus lacking comprehensiveness for 3D results; (2) **The CLIP score might not align with human perceptual judgments, which are crucial in evaluating generative results**; (3) This score evaluates the match between synthesis and text prompts using its learned feature space, which might not correlate directly with quality;  and (4) The CLIP score has biases toward its training data distribution, potentially differing from 3D projected images.  Consequently, many methods, like [1,2,3], choose to only conduct user studies for performance evaluation. In the final version, we will include user study results for these new experiments (note that we have previously conducted a user study comparing our results with DreamFusion and Magic3D, as detailed in our paper Table 1). Please see our qualitative results at the provided  [anonymous link](https://drive.google.com/drive/folders/1T6TuRrEH54CS4f84zcmf9sbOBeQ4OSMb?usp=sharing). 


Reference:

[1] MVDream: Multi-view Diffusion for 3D Generation. Shi et al., 2023

[2] ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation. Wang et al., 2023

[3] Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation. Chen et al. 2023

Reviewer comment (Reproducibility):
What negative prompts did you use?

Author response:
**[Q1, Q2] Negative prompts**

1. We use "oversaturated color, ugly, tiling, low quality, noisy" as our negative prompts. If negative prompts are not used, some generated results may show oversaturated colors or a higher occurrence of floaters. For demonstration, videos are provided at the following **[anonymous link](https://drive.google.com/drive/folders/1gvRq-Tr7dtTSBX-D3YRk184TEdIW1b_j?usp=sharing)**.
2. It should be emphasized that our paper's focus lies in providing a new understanding of score distillation and establishing the link between classifier scores and negative prompts rather than contributing to the technique of negative prompts itself. With this new understanding and guidance, we envision that future research could explore a range of techniques to improve quality, not limited to negative prompts. This includes developing adaptive negative classifier scores, for example, we examine the connection between VSD and classifier scores in equation (12).

Reviewer comment (Experiments):
How well does the CSD loss perform without the help of negative prompts, i.e., with the Eq. (7) only?

Author response:
**[Q1, Q2] Negative prompts**

1. We use "oversaturated color, ugly, tiling, low quality, noisy" as our negative prompts. If negative prompts are not used, some generated results may show oversaturated colors or a higher occurrence of floaters. For demonstration, videos are provided at the following **[anonymous link](https://drive.google.com/drive/folders/1gvRq-Tr7dtTSBX-D3YRk184TEdIW1b_j?usp=sharing)**.
2. It should be emphasized that our paper's focus lies in providing a new understanding of score distillation and establishing the link between classifier scores and negative prompts rather than contributing to the technique of negative prompts itself. With this new understanding and guidance, we envision that future research could explore a range of techniques to improve quality, not limited to negative prompts. This includes developing adaptive negative classifier scores, for example, we examine the connection between VSD and classifier scores in equation (12).

Reviewer comment (Experiments):
Which 81 text prompts did you choose to compute CLIP R-precision, and why don’t you compute all the text prompts (415 text prompts) provided in the DreamFusion repo?

Author response:
**[Q3] Using all 415 prompts for calculating CLIP R-precision**

Thank you for your suggestion. For calculating the R-precision, we utilized the same set of prompts as those used in the generation process, in accordance with Cap3D [4]. These 81 prompts are detailed in our supplementary materials, with corresponding video filenames indicating the specific prompts employed. Note that our comparison between CSD and other methods is fair, since they are all based on same prompts.

To further address your concern, we additionally calculated the R-precision using a larger set of 415 prompts for the caption set. As shown in the table below, the results are consistent with our original paper.

|                |         |                 |             |
| -------------- | :-----: | :-------------: | :---------: |
|                | **CSD** | **DreamFusion** | **Magic3D** |
| **CLIP Score** |  78.6   |      67.5       |    74.9     |
| **CLIP R@1**   |  64.5   |      55.6       |    53.1     |
| **CLIP R@5**   |  84.9   |      77.5       |    78.7     |
| **CLIP R@10**  |  89.2   |      84.3       |    85.8     |
|                |         |                 |             |

Reference:

[4] Scalable 3D Captioning with Pretrained Models. Luo et al., 2023

Reviewer comment (Experiments):
According to implementation details in section 5.1, this work uses two different pre-trained text-to-image models (DeepFloyd-IF stage-I model and Stable Diffusion 2.1). Is there any reason or ablation study for this design choice?

Author response:
**[W1] Reason for DeepFloyd-IF stage-I & Fairness of comparison**

1. The Stable Diffusion model is a high-resolution generative model (512x512), while the DeepFloyd-IF stage-I is a low-resolution model (64x64). Since our first stage focuses on generating rough results, we use DeepFloyd-IF stage-I.
2. We did our best to ensure fair comparisons with other methods by faithfully re-implementing their approaches based on their design principles or directly using their best available results as follows: 1) DreamFusion uses the superior **Imagen** model, which is not open-sourced. Fortunately, it publishes all results and thus, we directly compared our results with the best results from its official website. 2) The original Magic3D paper used **eDiff-I** in its first stage, which is also not open-sourced. Besides, Magic3D has not released its code or more results, so we re-implement its method upon the ThreeStudio. As the first stage of Magic3D also focuses on generating a coarse shape, we used DeepFloyd-IF stage-I in its first stage, the same as our setting. 3) ProlificDreamer needs high-resolution NeRF training in its first stage,  which requires high-resolution diffusion models. Thus, we followed their setting by using Stable Diffusion in the first stage. 4) Fantasia3D disentangles the generation into geometry and texture, in which both stages need high-resolution diffusion models to guarantee high-quality results. Thus, we followed their original paper by using Stable Diffusion.

Reviewer comment (Experiments):
Some baselines (like ProlificDreamer) only use the pre-trained text-to-image model of Stable Diffusion, so it is somewhat not fair to compare this work with other baselines using different pre-trained models.

Author response:
**[W1] Reason for DeepFloyd-IF stage-I & Fairness of comparison**

1. The Stable Diffusion model is a high-resolution generative model (512x512), while the DeepFloyd-IF stage-I is a low-resolution model (64x64). Since our first stage focuses on generating rough results, we use DeepFloyd-IF stage-I.
2. We did our best to ensure fair comparisons with other methods by faithfully re-implementing their approaches based on their design principles or directly using their best available results as follows: 1) DreamFusion uses the superior **Imagen** model, which is not open-sourced. Fortunately, it publishes all results and thus, we directly compared our results with the best results from its official website. 2) The original Magic3D paper used **eDiff-I** in its first stage, which is also not open-sourced. Besides, Magic3D has not released its code or more results, so we re-implement its method upon the ThreeStudio. As the first stage of Magic3D also focuses on generating a coarse shape, we used DeepFloyd-IF stage-I in its first stage, the same as our setting. 3) ProlificDreamer needs high-resolution NeRF training in its first stage,  which requires high-resolution diffusion models. Thus, we followed their setting by using Stable Diffusion in the first stage. 4) Fantasia3D disentangles the generation into geometry and texture, in which both stages need high-resolution diffusion models to guarantee high-quality results. Thus, we followed their original paper by using Stable Diffusion.

Reviewer comment (Experiments):
The evaluation of text-guided 3D generation is performed over 81 diverse text prompts from the website of DreamFusion, but the website contains lots of results (more than 81 prompts). How were the 81 diverse text prompts chosen? Any screening criteria behind this selection?

Author response:
**[W2-1] Criteria of text prompts**

1. Thank you for this great question. Our criteria for selecting prompts followed three principles:
   - **Sufficient Coverage and Diversity**: For example, some categories in the prompt set include: 1). **Animal Characters**: e.g., "a koala wearing a party hat"; 2). **Daily Life Objects**: e.g., "a shiny red stand mixer"; 3). **Transportation and Structures**: e.g., "a Panther De Ville car";  4). **Fantastical and Whimsical Themes**: e.g., "a humanoid banana doing homework"; and 5). **Nature and Scenery**: e.g., "a bumblebee sitting on a pink flower".
   - **Avoidance of Overly Similar Prompts**: There are many very similar prompts on the official DreamFusion website. We filtered these out to avoid such situations and achieve greater diversity.
   - **Inclusion of Uniquely Creative Prompts**: We retained prompts that are highly imaginative and creative, such as "a humanoid banana doing homework".
2. It's noteworthy that there is no established gold standard for evaluation prompts, and existing methods often select a subset of prompts for easier evaluation. For example, ProlificDreamer assessed their method using 15 prompts in a user study, while Fantasia3D did so with 10 prompts. Therefore, in our original paper, we adhered to this practice, choosing 81 prompts to guarantee coverage, diversity, and creativity, and to avoid redundancy in our representative evaluation prompts.
3. Given the constraints in computational resources, the brief rebuttal period, and the demands of conducting other experiments, generating results for 415 prompts exceeds our current capacity within the rebuttal timeframe. However, we have included results from additional randomly selected prompts, as illustrated in the comparison in **W3-1**. We pledge to release our results for all prompts once the experiments are completed.

Reviewer comment (Evaluation):
This evaluation only uses CLIP ViT-B/32 to extract text and image features, while DreamFusion uses three models (CLIP B/32, CLIP B/16, CLIP L/14) to measure CLIP R-Precision. Following DreamFusion, it would be better to report more results using more CLIP models.

Author response:
**[W2-2]  More CLIP R-precision**

We report more CLIP scores and CLIP R-precision here using different pre-trained models. Please refer to **W3-1** for an analysis of the CLIP score. 

|                           |         |                 |             |
| ------------------------- | :-----: | :-------------: | :---------: |
|                           | **CSD** | **DreamFusion** | **Magic3D** |
| **CLIP Score (ViT-L/14)** |  69.1   |      63.6       |    65.4     |
| CLIP R@1 (ViT-L/14)       |  96.0   |      88.3       |    84.9     |
| CLIP R@5 (ViT-L/14)       |  99.1   |      97.8       |    97.5     |
| CLIP R@10 (ViT-L/14)      |  99.7   |      98.8       |    99.4     |
| **CLIP Score (ViT-B/32)** |  78.6   |      67.5       |    74.9     |
| CLIP R@1 (ViT-B/32)       |  81.8   |      73.1       |    74.1     |
| CLIP R@5 (ViT-B/32)       |  94.8   |      90.7       |    91.7     |
| CLIP R@10 (ViT-B/32)      |  96.3   |      97.2       |    96.6     |
| **CLIP Score (ViT-B/16)** |  81.1   |      73.0       |    77.8     |
| CLIP R@1 (ViT-B/16)       |  94.1   |      88.3       |    84.3     |
| CLIP R@5 (ViT-B/16)       |  98.5   |      97.8       |    96.0     |
| CLIP R@10 (ViT-B/16)      |  98.8   |      99.1       |    97.5     |
|                           |         |                 |             |

Reviewer comment (Experiments):
For text-guided 3D generation, Table 2 only includes two baselines, while other strong baselines (Fantasia3D and ProlificDreamer) are missing.

Author response:
**[W3-1] Comparisons with ProlificDreamer and Fantasia3D:**

1. We would like to clarify that the reason that we omitted quantitative comparisons with ProlificDreamer and Fantasia 3D is that it is very time-consuming to evaluate ProlificDreamer based on public implementations (8h per prompt on an A800 GPU), and Fantasia3D requires manual tuning configuration parameters for each prompt (see their [note ("What do you want?")](https://github.com/Gorilla-Lab-SCUT/Fantasia3D#what-do-you-want) in their official implementation). It is also worth noting that in their original paper, ProlificDreamer evaluated the performance on 15 prompts with a user study, and Fantasia3D evaluated the performance on 10 prompts with a user study.

2. Here, within the short period of rebuttal time and our constrained computational resources, we randomly select 20 prompts and conduct experiments and comparisons with CSD, ProlificDreamer (using ThreeStudio), and Fantasia3D (using official code). The table below presents the results, showing that our method is generally comparable to Fantasia3D and ProlificDreamer. Moreover, our approach tends to perform better in comparison with other approaches under a more strict evaluation criterion like the CLIP score that directly measures the exact match between text prompts and generated images. However, it's important to note that metrics like the CLIP score and R-precision primarily measure textual-image congruence and may not reflect 3D and texture quality. For a more comprehensive analysis, please refer to the detailed discussion below.

   |                           |         |                |                     |
   | ------------------------- | :-----: | :------------: | :-----------------: |
   |                           | **CSD** | **Fantasia3D** | **ProlificDreamer** |
   | **CLIP Score (ViT-L/14)** |  71.8   |      65.2      |        72.7         |
   | CLIP R@1 (ViT-L/14)       |   100   |      93.8      |         100         |
   | CLIP R@5 (ViT-L/14)       |   100   |      100       |         100         |
   | CLIP R@10 (ViT-L/14)      |   100   |      100       |         100         |
   | **CLIP Score (ViT-B/32)** |  81.6   |      77.8      |        80.2         |
   | CLIP R@1 (ViT-B/32)       |  96.2   |      93.8      |        96.2         |
   | CLIP R@5 (ViT-B/32)       |  98.8   |      100       |         100         |
   | CLIP R@10 (ViT-B/32)      |   100   |      100       |         100         |
   | **CLIP Score (ViT-B/16)** |  83.1   |      78.6      |        83.2         |
   | CLIP R@1 (ViT-B/16)       |  97.5   |      96.2      |         100         |
   | CLIP R@5 (ViT-B/16)       |   100   |      100       |         100         |
   | CLIP R@10 (ViT-B/16)      |   100   |      100       |         100         |
   |                           |         |                |                     |

3. **[Analysis of CLIP score for 3D evaluation]** It's important to note that in evaluating 3D synthesized results, the CLIP score may not fully reflect the quality for several reasons: (1) As an image-level metric, the CLIP score assesses each projected image independently, thus lacking comprehensiveness for 3D results; (2) **The CLIP score might not align with human perceptual judgments, which are crucial in evaluating generative results**; (3) This score evaluates the match between synthesis and text prompts using its learned feature space, which might not correlate directly with quality;  and (4) The CLIP score has biases toward its training data distribution, potentially differing from 3D projected images.  Consequently, many methods, like [1,2,3], choose to only conduct user studies for performance evaluation. In the final version, we will include user study results for these new experiments (note that we have previously conducted a user study comparing our results with DreamFusion and Magic3D, as detailed in our paper Table 1). Please see our qualitative results at the provided  [anonymous link](https://drive.google.com/drive/folders/1T6TuRrEH54CS4f84zcmf9sbOBeQ4OSMb?usp=sharing).

Reference:

[1] MVDream: Multi-view Diffusion for 3D Generation. Shi et al., 2023

[2] ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation. Wang et al., 2023

[3] Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation. Chen et al. 2023

Reviewer comment (Reproducibility):
Section 5.2 only mentions the computational cost of ProlificDreamer and this work. It would be better to list the computational cost of each run.

Author response:
**[W3-2] Computational cost for each run:**

1). For DreamFusion, we used their official website's results, employing 4 TPUv4 chips for 15,000 iterations over 1.5 hours (6 TPUv4 hours). 2). For Magic3D, we used a single A800 GPU, with a batch size of 4 for 10,000 NeRF iterations (70 minutes) and a batch size of 1 for 10,000 iterations in the second stage (20 minutes), totaling 1.5 GPU hours. 3). For Fantasia3D, following their official code, we used 8 GPUs, each stage taking about 10 minutes, for a total of 2.7 GPU hours. 4). For ProlificDreamer, we use their original paper's settings: 25,000 VSD iterations for NeRF (3.5 hours), 15,000 SDS iterations for geometry (30 minutes), and 30,000 VSD iterations for texture (4 hours), totaling 8 GPU hours. 5). For our experiments, both stages were optimized for 10,000 iterations, taking 35 and 25 minutes respectively, amounting to 1 GPU hour.

Reviewer comment (Evaluation):
For text-guided texture synthesis, a strong baseline (Tango: Lei J, Zhang Y, Jia K., Advances in Neural Information Processing Systems, 2022) is missing for performance comparison. Moreover, only a user study is performed for this task, and it would be useful to see more quantitative comparison using the CLIP score or CLIP R-Precision.

Author response:
**[W3-3] Comparisons with Tango & quantitative evaluations:**

Thank you for your reminder. We have cited this paper in our revised paper (appendix A.1) and here we present the comparison results with it. We show the qualitative comparison results at this **[anonymous link](https://drive.google.com/drive/folders/1dANkz6X56wFKb-1jVakCsFimo_SuBVnD?usp=sharing)**.

Regarding the task of texture synthesis, as mentioned in **W3-1**, evaluations based on CLIP may not be accurate. Most current works rely on user studies. Here, we provide the CLIP score for your reference.  Although Fantasia3D performs the best on two backbones, there is not a single method that wins in all metrics, and our method performs reasonably well on all evaluated settings.  **More importantly, our user study in Table 1 in our main paper demonstrates a clear win of our approach in terms of texture synthesis when compared to Fantasia 3D, ProlificDreamer, Magic3D, and TEXTure.**

|                     |                           |                           |                           |
| ------------------- | :-----------------------: | :-----------------------: | :-----------------------: |
|                     | **CLIP Score (ViT-L/14)** | **CLIP Score (ViT-B/32)** | **CLIP Score (ViT-B/16)** |
| **CSD**             |           64.6            |           72.7            |           74.5            |
| **Tango**           |           61.6            |           72.3            |           74.0            |
| **TEXTure**         |           61.4            |           72.0            |           72.6            |
| **ProlificDreamer** |           63.8            |           73.1            |           75.0            |
| **Fantasia3D**      |           63.2            |           74.2            |           75.3            |
| **Magic3D**         |           61.9            |           71.1            |           74.2            |
|                     |                           |                           |                           |

Reviewer comment (Experiments):
There is an interest in seeing more results by plugging the proposed CSD into more baselines (like DreamFusion and ProlificDreamer).

Author response:
**[W4] CSD on more baselines:**

1. Please note that for DreamFusion, ProlificDreamer, and our work, the principal differences and unique contributions are centered on the three distinct losses: SDS, VSD, and CSD. Therefore, integrating CSD into these methods essentially amounts to employing the CSD approach, albeit with some variations in the training process. For more details, kindly refer to equations 6, 8, and 12 in our paper. In fact, our loss is model-agnostic or independent of 3D representation, as evidenced by our approach in different stages. For instance, in the first stage, we utilize the DeepFloyd-IF stage-I model with NeRF representation, and in the second stage, we apply Stable Diffusion with DMTet representation. This showcases the versatility of our loss across various models and representations.
2. To show more results in more baselines, we integrated the proposed CSD into the pre-trained multi-view diffusion model from the recent concurrent work MVDream [1]. We provide the qualitative results at this [anonymous link](https://drive.google.com/drive/folders/1duCZB2KFozMGFOgQmdQGGiCwiZooEsOr?usp=sharing). The results indicate that our loss can be seamlessly integrated into this new framework, producing realistic contents and high-quality textures.

Reference:

[1] MVDream: Multi-view Diffusion for 3D Generation. Shi et al., 2023

Reviewer comment (Experiments):
If I understand correctly, empirically, this is not much different than using SDS with a large weight for CFG. It would be helpful to show comparisons to SDS with very large CFG weights. See questions for more details.

Author response:
**[W2, Q3] CFG in baseline methods & Experiment with larger CFG for SDS**

1. **For the CFG scale of other methods, we followed the values stated in their respective papers**. For ProlificDreamer, we set the CFG scale to 7.5 in both the first and third stages, where VSD is used, and increased it to 100 in the second stage, where SDS is employed for refining the geometry. For Fantasia3D, we consistently applied a CFG scale of 100 for both stages. In the case of Magic3D, we used a CFG scale of 100 for the second stage. However, for the first stage of Magic3D, since we do not have access to their unreleased eDiff-I model, we utilized the DeepFloyd-IF stage-I model with a CFG scale of 20, following the default implementation in ThreeStudio.

2. Although SDS with a large CFG value approaching infinity may approximate the effects of using classifier-free guidance alone (i.e., CSD), this raises significant optimization issues, making it impractical. Within the SDS framework, which combines generative and classifier priors, hyperparameter tuning with a large CFG scale becomes complex and often requires extensive trial and error to adjust learning rates and geometric constraint loss weights. For instance, an excessively large CFG scale can result in problems such as numerical overflow in gradient calculations due to substantial guidance weights. Moreover, this can cause a "gradient domination" effect, diminishing the impact of other geometric regularization terms.
3. To empirically demonstrate the effect of using higher CFG values with SDS loss, we have conducted experiments using the first stage of Magic3D. Note that the first stage of Magic3D is essentially the same as DreamFusion, with the difference being that DreamFusion employs Mip-NeRF representation and the more powerful Imagen model that we do not have access to. We use SDS with CFG scales of 20, 100, 200, 500, and 1000. However, simply increasing guidance weights does not guarantee satisfactory results, as confirmed in our results (refer to results in  **[anonymous link](https://drive.google.com/drive/folders/1qV2K2eaKibcpuOYkUEm3j7NdnK0wWzr5?usp=sharing)**): higher CFG values can lead to more floaters, and at extremely high CFG values, there's a risk of results collapsing.

Reviewer comment (Presentation):
Figure 2a: It might be more clear to show both norms on the same scale. At first glance it can be confusing if you don’t notice the different scales.

Author response:
**[W3, W4] Revise of Figure 2**

Thanks for your suggestion, we have revised the Figure 2 for better demonstration.

Reviewer comment (Experiments):
Figure 2b: Consider including CSD here. It would be interested to see higher values for w as well since DreamFusion uses w=100.

Author response:
**[W3, W4] Revise of Figure 2**

Thanks for your suggestion, we have revised the Figure 2 for better demonstration.

Reviewer comment (Evaluation):
Similarly, how is the diversity of generations using CSD for a given prompt? I would guess that there is less diversity than SDS since higher CFG weight typically reduces diversity.

Author response:
**[W1, Q2] Diversity**

1. Empirically, we did **not** observe lower diversity using CSD compared to SDS. We think that improving diversity through reducing CFG  might not be a viable choice for the SDS-based method as a lower CFG value may also lead to optimization failure, which is verified by our analysis in Figure 2 of our paper. In this work, our focus is to uncover the component— the classifier-free guidance— that makes SDS work.

2. To improve diversity, we believe a better approach might be to consider more diverse prompts. We can enhance the user-provided prompts, ensuring they meet the user’s content requirements while introducing a variety of variations. This can be achieved automatically using large language models (LLMs). As an illustration, here we employed ChatGPT to enhance the given prompt 'a DSLR photo of a bear dressed in medieval armor', where we asked it to generate different variations while retaining the original meaning, resulting in the following prompts:

   - “A DSLR photo of a panda bear in stylized Eastern medieval armor, reflecting Asian historical designs.”
   - “A DSLR photo of a polar bear in ornate, silver-colored medieval armor, contrasting with its white fur.”
   - “A DSLR photo of a sun bear wearing lightweight leather armor, reminiscent of medieval archers, with a quiver of arrows.”
   - “A DSLR photo of a panda bear in ancient Chinese warrior armor, embellished with dragon motifs and vibrant colors.“

   We then applied our method (NeRF stage) to these prompts, and we provided video results at the following **[anonymous link](https://drive.google.com/drive/folders/1q49ztkSIx9USFzeH1sFPSAVS54tDcCcR?usp=sharing)**. This approach successfully enhances diversity while not suffering from optimization issues.

Reviewer comment (Experiments):
What are the CFG weights used in the experiments section for the SDS on the baseline methods? It is specified that the DreamFusion results were obtained from its website implying a CFG of 100, but what about for the others? The default value in ThreeStudio appears to be 100 for methods using stable diffusion and 7.5 for Prolific Dreamer. Is that what was used for the experiments? If so, it might be helpful to add experiments showing existing SDS methods with very large CFG weights (i.e. 200, 500, 1000, etc.) and see how that compares to CSD.

Author response:
**[W2, Q3] CFG in baseline methods & Experiment with larger CFG for SDS**

1. **For the CFG scale of other methods, we followed the values stated in their respective papers**. For ProlificDreamer, we set the CFG scale to 7.5 in both the first and third stages, where VSD is used, and increased it to 100 in the second stage, where SDS is employed for refining the geometry. For Fantasia3D, we consistently applied a CFG scale of 100 for both stages. In the case of Magic3D, we used a CFG scale of 100 for the second stage. However, for the first stage of Magic3D, since we do not have access to their unreleased eDiff-I model, we utilized the DeepFloyd-IF stage-I model with a CFG scale of 20, following the default implementation in ThreeStudio.

2. Although SDS with a large CFG value approaching infinity may approximate the effects of using classifier-free guidance alone (i.e., CSD), this raises significant optimization issues, making it impractical. Within the SDS framework, which combines generative and classifier priors, hyperparameter tuning with a large CFG scale becomes complex and often requires extensive trial and error to adjust learning rates and geometric constraint loss weights. For instance, an excessively large CFG scale can result in problems such as numerical overflow in gradient calculations due to substantial guidance weights. Moreover, this can cause a "gradient domination" effect, diminishing the impact of other geometric regularization terms.
3. To empirically demonstrate the effect of using higher CFG values with SDS loss, we have conducted experiments using the first stage of Magic3D. Note that the first stage of Magic3D is essentially the same as DreamFusion, with the difference being that DreamFusion employs Mip-NeRF representation and the more powerful Imagen model that we do not have access to. We use SDS with CFG scales of 20, 100, 200, 500, and 1000. However, simply increasing guidance weights does not guarantee satisfactory results, as confirmed in our results (refer to results in  **[anonymous link](https://drive.google.com/drive/folders/1qV2K2eaKibcpuOYkUEm3j7NdnK0wWzr5?usp=sharing)**): higher CFG values can lead to more floaters, and at extremely high CFG values, there's a risk of results collapsing.

Reviewer comment (Experiments):
In general, I do not see a crucial weakness of this paper as it illustrates a simple method that improves the current text-to-3D generation. I believe providing detailed hyperparameter ablation study will make the paper more informative.

Author response:
**[W1] Hyper-parameter ablation study**

Thanks for your suggestion. In our experiment, we found that employing appropriate sparsity and orientation losses during the NeRF optimization is crucial for geometry. Here, we show four distinct experiments to demonstrate this: A). Without any geometric constraint loss. B). Constrained only by sparsity loss. C). Constrained only by orientation loss. D). (i.e., CSD) Employing both constraints simultaneously.

We have provided video comparison results at this **[anonymous link](https://drive.google.com/drive/folders/1IU583pgt_2hipKA9W2Y7jnt8I_EzUubf?usp=drive_link)**. As can be observed, it's only when both constraints are applied together that we can ensure smoother normals and better 3D consistency (i.e., less Janus problem), as well as guarantee the overall success of the optimization process.

Reviewer comment (Theory):
See Strengths; how the image-conditioned noise subtraction of InstructPix2Pix diffusion model in Collaborative Score Distillation paper can be related to classifier score distillation? Can Collaborative score distillation can be improved with classifier score distillation like approach?

Author response:
**[S2, Q1] Relation to Collaborative Score Distillation**

Thank you for your insightful comment and the recommended work, which we have now cited in our revised paper. We find it to be a very interesting work! Interestingly, its formulation also replaces subtracting noise with subtracting a score computed from a reference image. This bears a resemblance to our discussion on DDS. We believe that substituting this aspect with CSD could potentially further enhance the quality.

Since the code for this paper has not been released yet, it is a bit challenging for us to faithfully reproduce this work during this short rebuttal period.  We will continue to pay close attention to this work and its release. Thank you for your insightful comments!

To demonstrate the generality and adaptivity of our CSD formulation, we conducted additional experiments on the recently released multi-view diffusion model [1]. We provide the qualitative results at this **[anonymous link](https://drive.google.com/drive/folders/1duCZB2KFozMGFOgQmdQGGiCwiZooEsOr?usp=sharing)**. The results demonstrate that our loss can be seamlessly integrated into this new framework, producing realistic 3D content.

Reference:

[1] MVDream: Multi-view Diffusion for 3D Generation. Shi et al., 2023