Paper: DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior (ICLR-2024)

Reviewer comment (Experiments):
The running time of methods. This method involves several stages and looks quite complicated. I am curious about how long does it take to finish the whole process? And how does it compare to other baselines?

Author response:
**Q1: Runtime of the method.**

A: The training time comparison against recently published works is shown below.


| Method          | Training Time |
| --------------- | ------------- |
| DreamCraft3D    | 2.3h          |
| Magic123        | 2.8h          |
| Magic3D         | 1.5h          |
| ProlificDreamer | 10h           |
| DreamFusion     | 0.7h          |

Below we report the detailed runtime profile. These numbers are measured on 30 objects. The training is conducted on 8 A100 GPUs and the inference on 512 $\times$ 512 resolution is measured using a single A100 GPU. We did not fully optimize the training time, and we expect significant speedup with further engineering optimization. 

| Running Type                       | Time  |
| ---------------------------------- | ----- |
| Training Time (total)              | 2.3h  |
| Training Time (Geometry Sculpting) | 1.5h  |
| Training Time (Texture)            | 0.8h  |
| Inference Time                     | 0.05s |

Reviewer comment (Reproducibility):
It would be more clear if the difference between the proposed bootstrapped diffusion prior and LORA updates in ProlificDreamer is elaborated, since they both update the diffusion during the distillation. What's the intuitive difference and the actual difference in implementation?

Author response:
**Q2: The difference between the proposed bootstrapped diffusion prior and the LoRA updates in ProlificDreamer.**

A: Thank you for raising the point. Below we clarify their difference.

ProlificDreamer regards the 3D solution corresponding to the text as a distribution. Thus, ProlificDreamer aims to learn a distribution $\mu$ to match the distribution characterized by the pretrained diffusion model. Mathematically, this is achieved by minimizing the score function of the two distributions. The score function of the pretrained model is represented by its noise prediction $\epsilon_{\text{pretrain}}(x_t,t,y)$. On the other hand, the score function of the optimized 3D scene is estimated by the LoRA model, *i.e.*,$\epsilon_\\phi\left(x_t, t, c, y, \\{x\\}\right)$, using the multi-view image renderings $\\{x\\}$. Hence, the VSD loss used in ProlificDreamer is:
$$
\nabla_\theta \mathcal{L}_{\mathrm{VSD}}(\theta) \triangleq \mathbb{E}\_{t, \epsilon, c}\left[\omega(t)(\epsilon\_{\mathrm{pretrain}}\left(x_t; t, y\right)-\underbrace{\epsilon\_\phi\left(x_t; t, c, y, \\{x\\}\right)}\_{\text {LoRA }}) \frac{\partial g(\theta, c)}{\partial \theta}\right].
$$

The proposed bootstrapped score distillation differs in the motivation fundamentally. We claim that a fixed 2D diffusion prior is not 3D aware and cannot provide sufficient guidance that ensures multi-view consistency. Instead, we propose to learn from a moving target that also updates based on the 3D scene being optimized. In the implementation, we render the multi-view images of the scene and augment their quality (*i.e.*, adding the Gaussian noise and then denoising them) using a diffusion model, forming high-quality augmented renderings denoted as $\\{\tilde{x}\\}$. These augmented images are used to compute the LoRA model that serves as the training target (the first term in the following formula): 
$$
\nabla_\theta \mathcal{L}_{\mathrm{BSD}}(\theta) \triangleq \mathbb{E}\_{t, \epsilon, c}\left[\omega(t)(\underbrace{\epsilon\_{\mathrm{DreamBooth}}\left(x_t; t, c, y,  \\{\tilde{x}\\}_k\right)}\_{\text{BSD}}-\underbrace{\epsilon\_\phi\left(x_t; t, c, y, \\{x\\}\right)}\_{\text {LoRA }}) \frac{\partial g(\theta, c)}{\partial \theta}\right]
$$
We iterate the above process and apply attenuated augmentation strength to derive augmented views, $\\{\tilde{x}\\}_k$, where $k$ denotes the iteration steps. As training proceeds, both the diffusion prior and the 3D scene get improved, which bootstraps the 3D optimization. We have better formulated the above process during revision.

Reviewer comment (Evaluation):
I understand it is hard to evaluate generation tasks, but more details about metrics used in Table 1 would be convincing. All CLIP, Contextual, PSNR and LPIPS are image-level evaluation, and it is not clear how to convert them to 3D level. PSNR and LPIPS are measured on reference images, which is understandable. But how are the CLIP and Contextual calculated? How many views are rendered from each scene for evaluation?

Author response:
**Q3: Further explanation of the evaluation protocol.**

A: Thanks for the question. We compute the CLIP score to evaluate the semantic similarity between the image renderings at novel views and the reference image. A higher CLIP score indicates better semantic consistency around different views. The contextual loss is computed to measure the style similarity between the novel views and the reference image. The contextual metric can be regarded as an improved LPIPS without requiring pixel-wise alignment. A higher contextual metric means improved style consistency around the generated 3D asset. We use 120 rendered views to compute both the CLIP and the contextual score. To better reveal the subjective quality, we also conducted a user study which gathered 480 responses from a total of 32 participants. We have included these details in the paper revision.

Reviewer comment (Reproducibility):
Multiple loss or gradient terms are introduced in Section 4.1, but the relative weights of these terms are not described.

Author response:
A: Thanks for figuring it out. We have included this implementation detail in Section 4.2 during revision.

Reviewer comment (Reproducibility):
Do all examples shown in the paper share the same hyperparameter configuration?

Author response:
A: Thanks for raising the point. Most cases in our paper (roughly 90%) share the same hyper-parameters. However, some adjustments might be needed. When the reference is captured at an oblique angle, we may change the evaluation angle for the view-conditioned diffusion model. Additionally, we might need to properly set the camera FOV to suit the object size in the reference image.

Reviewer comment (Evaluation):
**Claims** : The authors claim that personalizing the diffusion model based on the multiview renderings helps improve texture, however, this seems counter intuitive, since the initial texture generated from the 3D representation are expected to be worse than the high quality texture. Providing additional insights about this would be helpful.

Author response:
**Q2: How does the personalized diffusion model trained on initial texture boost the texture quality?**

A: Thanks for raising the point. Indeed, the texture from the previous stage has limited quality and suffers from blurriness. Therefore, **we use a pretrained diffusion model to enhance their quality**.

We incorporate Gaussian noise into multi-view images, allowing a diffusion model to reconstruct high-quality visuals. Despite affecting view consistency, this process yields impressively realistic images. The enhanced multi-view images introduce scene aspects to form a personalized diffusion model, which significantly improves texture quality.

We iterate the above process several times and each step leads to improved 3D texture. We progressively decrease the strength to augment the multi-view rendering to better retain their view consistency. As we iterate the above process, we bootstrap the texture quality for the 3D instance being optimized.

Reviewer comment (Writing):
**Writing**: The manuscript contains certain syntactic and language errors (such as some of the nits indicated below) and would benefit from a thorough proof reading pass of both the main paper and the appendix.

Author response:
**Q3: Writing.**

A: Thanks for your comment. We have conducted careful proofreading and fixed some typos in the revision.

Reviewer comment (Experiments):
**Ablations**: Qualitative ablations are provided for some of the components. But adding some quantitative ablations that show change in quality would be helpful. Particularly, metrics for 3D consistency for number of BSD steps, texture quality with and without BSD and effect of progressive training, timestep annealing and choice of diffusion model used for SDS (SD vs DeepFloyd).

Author response:
**Q4: More ablation studies.**

A: Per your suggestion, we provide additional qualitative and quantitative ablation studies for two stages respectively as follows. The techniques of time annealing and progressive training improve the view consistency relative to the reference image, as measured by the CLIP and contextual score. We have included these results in the appendix.  

|               | CLIP$\uparrow$  | Contextual $\downarrow$ | PSNR $\uparrow$  | LPIPS $\downarrow$ |
| ------------- | ----- | ---------- | ------ | ------ |
| Default setting | **0.745** | **1.936**  | **20.836** | 0.031  |
| w/o $\mathcal{L}_{\textnormal{RGB}}$ | 0.619 | 2.212 | 10.562 | 0.095  |
| w/o $\mathcal{L}_{\textnormal{mask}}$| 0.736 | 1.949 | 19.307 | 0.057  |
| w/o 3D prior  | 0.679 | 2.105 | 20.252 | 0.092 |
| w/o time annealing | 0.737 | 1.943 | 20.120 | 0.035 |
| w/o progressive training | 0.721 | 1.962 | 20.317 | **0.029** |
| Stable diffusion | 0.669 | 2.154      | 18.182 | 0.099  |

|               | CLIP$\uparrow$  | Contextual $\downarrow$ | PSNR $\uparrow$  | LPIPS $\downarrow$ |
| ------------- | ----- | ---------- | ------ | ------ |
| SDS           | 0.775 | 1.894      | 25.258 | 0.036  |
| VSD           | 0.792 | 1.833      | 31.426 | 0.005  |
| One-round BSD | 0.863 | 1.647      | 30.431 | 0.008  |
| Two-round BSD | **0.896** | **1.579**      | **31.801** | **0.005**  |

Reviewer comment (Evaluation):
**Training and inference time costs**: A comparison of training and inference time cost and memory footprint of the proposed approach compared to the baselines is also important and would provide some insights about the inference time trade off and memory required to achieve this quality.

Author response:
**Q5: Training and inference time costs.**

A: The training time comparison against recently published works is shown below.


| Method          | Training Time |
| --------------- | ------------- |
| DreamCraft3D    | 2.3h          |
| Magic123        | 2.8h          |
| Magic3D         | 1.5h          |
| ProlificDreamer | 10h           |
| DreamFusion     | 0.7h          |

We report the detailed runtime profile below. These numbers are measured on 30 objects. The training is conducted on 8 A100 GPUs and the inference on 512 $\times$ 512 resolution is measured using a single A100 GPU. We did not fully optimize the training time and we expect significant speedup with further engineering optimization. 

| Running Type                       | Time  |
| ---------------------------------- | ----- |
| Training Time (total)              | 2.3h  |
| Training Time (Geometry Sculpting) | 1.5h  |
| Training Time (Texture)            | 0.8h  |
| Inference Time                     | 0.05s |

Reviewer comment (Experiments):
**Question**: What is the effect of time step annealing? and progressive viewpoint training?

Author response:
**Q4: More ablation studies.**

A: Per your suggestion, we provide additional qualitative and quantitative ablation studies for two stages respectively as follows. The techniques of time annealing and progressive training improve the view consistency relative to the reference image, as measured by the CLIP and contextual score. We have included these results in the appendix.  

|               | CLIP$\uparrow$  | Contextual $\downarrow$ | PSNR $\uparrow$  | LPIPS $\downarrow$ |
| ------------- | ----- | ---------- | ------ | ------ |
| Default setting | **0.745** | **1.936**  | **20.836** | 0.031  |
| w/o $\mathcal{L}_{\textnormal{RGB}}$ | 0.619 | 2.212 | 10.562 | 0.095  |
| w/o $\mathcal{L}_{\textnormal{mask}}$| 0.736 | 1.949 | 19.307 | 0.057  |
| w/o 3D prior  | 0.679 | 2.105 | 20.252 | 0.092 |
| w/o time annealing | 0.737 | 1.943 | 20.120 | 0.035 |
| w/o progressive training | 0.721 | 1.962 | 20.317 | **0.029** |
| Stable diffusion | 0.669 | 2.154      | 18.182 | 0.099  |

|               | CLIP$\uparrow$  | Contextual $\downarrow$ | PSNR $\uparrow$  | LPIPS $\downarrow$ |
| ------------- | ----- | ---------- | ------ | ------ |
| SDS           | 0.775 | 1.894      | 25.258 | 0.036  |
| VSD           | 0.792 | 1.833      | 31.426 | 0.005  |
| One-round BSD | 0.863 | 1.647      | 30.431 | 0.008  |
| Two-round BSD | **0.896** | **1.579**      | **31.801** | **0.005**  |

Reviewer comment (Presentation):
**Clarity of Approach Section**: The introduction of multiple stages and new sub-modules (loss functions, optimization stages) makes the presentation in the approach section less clear. It would be beneficial to include an overview section with a clear definition of the total loss and a pseudo-algorithm summarizing the steps in different stages.

Author response:
A: Thanks for the comment. As suggested, we summarize the loss functions in the two stages in Eq.7 and Eq.9, respectively. The algorithm for the bootstrapped score distillation is detailed in the appendix. Furthermore, we have supplemented more implementation details. We hope these changes have improved the clarity of our method.

Reviewer comment (Experiments):
**Technical Contribution Clarification**: The major technical contributions are not clearly defined. It's essential to clarify which submodules or loss functions represent the most significant changes and contribute the most to the final performance. This information is crucial for a deeper understanding of the work. Overall, the proposed framework is a bit complicated and contains many modules and stages.

Author response:
A: Our method addresses 3D generation through a two-tiered process: a geometry sculpting stage and a texture boosting stage.

In the geometry sculpting stage, we strive to construct a coarse but 3D-coherent geometry. This is achieved through a 3D-aware diffusion prior, augmented by several effective training strategies. This stage aims to create robust and coherent geometry.

For the texture boosting stage, we introduce a novel bootstrapped score distillation loss. This approach enables the generation of detailed and visually stunning textures, contributing to a more realistic and high-quality 3D output.

Hence, the two stages focus on different aspects: the former prioritizes geometry while the latter adds meticulous texture. To explain the relative importance of different techniques, we conduct a more thorough ablation study, both qualitatively and quantitatively, in Figure 11, Table 2 and Table 3, as shown in the appendix. For your convenience, we provide quantitative results for the BSD loss as follows:  

|                                       | CLIP$\uparrow$ | Contextual $\downarrow$ | PSNR $\uparrow$ | LPIPS $\downarrow$ |
| ------------------------------------- | -------------- | ----------------------- | --------------- | ------------------ |
| Default setting                       | **0.745**      | **1.936**               | **20.836**      | 0.031              |
| w/o $\mathcal{L}_{\textnormal{RGB}}$  | 0.619          | 2.212                   | 10.562          | 0.095              |
| w/o $\mathcal{L}_{\textnormal{mask}}$ | 0.736          | 1.949                   | 19.307          | 0.057              |
| w/o 3D prior                          | 0.679          | 2.105                   | 20.252          | 0.092              |
| w/o progressive training              | 0.721          | 1.962                   | 20.317          | **0.029**          |
| w/o time annealing                    | 0.737          | 1.943                   | 20.120          | 0.035              |
| Stable diffusion                      | 0.669          | 2.154                   | 18.182          | 0.099              |

|               | CLIP$\uparrow$  | Contextual $\downarrow$ | PSNR $\uparrow$  | LPIPS $\downarrow$ |
| ------------- | ----- | ---------- | ------ | ------ |
| SDS           | 0.775 | 1.894      | 25.258 | 0.036  |
| VSD           | 0.792 | 1.833      | 31.426 | 0.005  |
| One-round BSD | 0.863 | 1.647      | 30.431 | 0.008  |
| Two-round BSD | **0.896** | **1.579**      | **31.801** | **0.005**  |

Reviewer comment (Experiments):
**Incomplete Validation**: Some modules are not thoroughly validated in the experiments. For instance, the importance and effects of the two losses in Eq.4, the L_RGB, and L_mask losses should be elaborated upon. It's also important to explain the advantages of using NeuS compared to NeRF-variants and provide validation results. Additionally, details on the implementation of "Progressive view training" should be provided.

Author response:
A: We add both quantitative and qualitative ablation studies of loss functions ($\mathcal{L}\_{\textnormal{RGB}}$ and $\mathcal{L}_{\textnormal{mask}}$), training strategies (timestep annealing and progressive training), and model choice (DeepFloyd vs. Stable Diffusion) in the revision (See Appendix A.4), showing the effectiveness of individual techniques.

We choose to optimize NeuS instead of NeRF in the geometry sculpting stage because NeRF learns a volume field which is hard to yield a high-quality surface for the following stage, whereas NeuS parameterizes the scene with neural SDF representation, yielding a high-quality surface for the following stage. Please refer to Figure 12 in the Appendix for visual examination.

For progressive training, we linearly increase the sampling range of camera positions with elevation angle ($\phi_\textnormal{cam}$) from $0^{\circ}$ to  $[-10^{\circ}, 45^{\circ}]$, and azimuth angle ($\theta_\textnormal{cam}$) from $0^{\circ}$ to $[-180^{\circ}, 180^{\circ}]$. The progress viewing training takes 200 iterations. We have included this implementation detail in the Appendix following your suggestion.

Reviewer comment (Reproducibility):
**Prompt Engineering Clarity**: The paper needs to address discrepancies in prompt engineering. As shown in Fig.2, the prompt for generating the reference image is "an astronaut in a sand beach," while for dreambooth, it's "an [v] astronaut." These differences need clarification to ensure consistency and reproducibility.

Author response:
A: We are thankful for your careful examination. We use the prompt "a [V] astronaut" rather than "an astronaut in a sand beach" because we focus on hallucinating the 3D for the foreground object exclusively. The unique identifier “[v]” is used to characterize subject-specific attributes such as lighting, color, pose, etc.

Reviewer comment (Presentation):
**Figure Order**: The order of figures is confusing. Fig.4 is mentioned later in the text than Fig.5/6 but appears earlier. Consider reordering them for coherence.

Author response:
A: We have improved the order of figures and added text descriptions of the corresponding stages in Figure 6, as per your suggestion.

Reviewer comment (Presentation):
**Fig.6 Text Descriptions**: Fig.6 is missing text descriptions of the corresponding stages. It's unclear which figure corresponds to what stage, making it difficult to follow.

Author response:
A: We have improved the order of figures and added text descriptions of the corresponding stages in Figure 6, as per your suggestion.

Reviewer comment (Evaluation):
**Training and Inference Time**: Could you provide insights into the training time for the entire pipeline and the time required for each individual stage? Additionally, how long does it take to generate a new 3D model from scratch, and could you provide a breakdown of the time allocation for this process?

Author response:
A: We compare the training time  against recent leading methods:


| Method          | Training Time |
| --------------- | ------------- |
| DreamCraft3D    | 2.3h          |
| Magic123        | 2.8h          |
| Magic3D         | 1.5h          |
| ProlificDreamer | 10h           |
| DreamFusion     | 0.7h          |

The profile of training and inference time is reported in the following table. We report the average runtime on 30 objects. The 3D asset is optimized on 8 A100 GPUs and the inference is done using a single GPU. We render at 512 $\times$ 512 resolution during inference. We haven't carefully optimized the optimization speed we consider efficient 3D generation as future work.

| Running Type                       | Time  |
| ---------------------------------- | ----- |
| Training Time (total)              | 2.3h  |
| Training Time (geometry sculpting) | 1.5h  |
| Training Time (texture boosting)            | 0.8h  |
| Inference Time                     | 0.05s |

Reviewer comment (Experiments):
**Robustness of the System**: Given the numerous modules and training stages, how robust is the system? Are the same set of parameters applicable to different methods, and if not, what are the characteristics of failure cases? Have you observed any convergence issues in any of the stages?

Author response:
A: This is a good point! Most cases in our paper (roughly 90%) share the same hyper-parameters. However, some adjustments might be needed. When the reference is captured from an oblique angle, we may change the evaluation angle for the view-conditioned diffusion model. Additionally, we might need to properly set the camera FOV to suit the object size in the reference image.

Reviewer comment (Experiments):
**Choice of Deepfloyd IF**: What is the rationale for not using SD in the first stage and opting for Deepfloyd IF instead? Could you elaborate on the advantages of Deepfloyd IF in this context?

Author response:
A: We employ the DeepFloyd IF model as it provides enhanced view-awareness and reduces the likelihood of encountering the “Janus (multi-face) problem”. This is because the DeepFloyd IF model has a stronger language encoder (T5-XXL) that better encodes the pose information denoted in the text prompt. Moreover, the DeepFloyd base model offers gradient on a coarse resolution (64x64), which promotes a coherent 3D geometry rather than introducing high-frequency signals that distract the geometry sculpting. The ablation of this design choice is shown in Table 3 in the revised version.