Paper: Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation (ICLR-2024)

Reviewer comment (Novelty):
Missing references on image-to-3D generation (CVPR'23 papers, both released on arxiv before the original release of this submission). [1] proposes textual inversion + dreambooth method for improving semantic consistency, similar to Imagic. [2] uses textual inversion for semantic consistency. Would be nice if these approaches were discussed in the related works section, as they are relevant to the semantic consistency component in this work.

Author response:
## **W1. Missing references on image-to-3D generation.**

Thank you for your constructive comment. Following your advice, we have added both approaches with discussions in the Related Work section of our revised paper, and we would be thankful if you could view it. We are very grateful for your helpful advice in strengthening our paper.

Reviewer comment (Novelty):
What are the major differences between the proposed method and the above [1][2] on improving semantic consistency in SDS?

Author response:
## **Q1. Major differences between the proposed method and previous works on improving semantic consistency in SDS.**

Thank you for pointing this out. We explain the main difference between the semantic consistency used in NeuralLift -360 [a] and RealFusion [b] in two aspects. 

### 1. Regarding differences in the main objective between [a,b] and ours.

Both NeuralLift and Realfusion aim to leverage Stable Diffusion's generative capability to modify text-to-3D generative methodologies into **one-shot 3D reconstruction methods**, where they aim for pixel-level accurate alignment and reconstruction of the input image into a full-fledged 3D model. To this end, they apply direct reconstruction loss as well as score distillation to generate 3D models aligned to accurately input images. 

On the other hand, the main objective of our work is fundamentally a **text-to-3D** generation. In our work, we do utilize an image as a stepping stone for point cloud generated and text embedding optimization, but we do not aim for perfect image-to-3D reconstruction like the above papers through these features: instead, our model solely aims for stronger robustness in geometric and semantic consistency of the generated models - for quality enhancement. In this manner, the two works and our 3DFuse tackle a fundamentally different problem in 3D generation.

### 2. Regarding differences in technical components.

- NeuralLift-360:  This model uses a monocular depth prediction model coupled with pixel-level reconstruction loss from the image to facilitate the generation of **geometry around the image viewpoint**.
- RealFusion: This method mentions in its Appendix that the incorporation of mono-depth prediction was not effective in facilitating image-to-3D generation: however, this work aims to make the generated scene semantically consistent with a given image through textual inversion, while also using **reconstruction loss to fit the 3D scene** to the image given.
- 3DFuse (Ours): Our method leverages a generated point cloud to ensure geometric consistency globally across all viewpoints of the scene. While point cloud is coarse and sparse in comparison to mono-depth predictions, this problem is well addressed by our Sparse Depth Injector.  In this process, to ensure semantic consistency we use semantic coding (LoRA + inversion), this time ensuring semantic consistency across all viewpoints of the scene.

We are grateful to the reviewer for your construction question, and we believe this discussion will help strengthen our paper greatly. We will add an additional section in the Appendix to discuss the above comparison in detail. Thank you for your time and effort in reviewing our paper.

&nbsp;

### Reference

[a] Xu, Dejia, et al. ”NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360° Views”, CVPR 2023

[b] Melas-Kyriazi, Luke, et al. “RealFusion: 360° Reconstruction of Any Object from a Single Image”, CVPR 2023

Reviewer comment (Experiments):
3DFuse relies heavily on off-the-shelf point cloud reconstruction or generation methods like Point-E, to obtain coarse point clouds. These point clouds subsequently serve as conditional information in the SDS optimization process. However, sparse-depth renderings are low-quality and may have many artifacts, resulting in degenerated and ambiguous results as shown in Fig.4.

Author response:
## **W1. Regarding the quality of point cloud obtained from off-the-shelf point cloud models.**

Thank you for pointing this out. We emphasize that one of the main contributions of our paper, **“Sparse Depth Injector (Section 4.3)”,** specifically focuses on addressing the problem of unstable artifacts and errors within the generated 3D point clouds you have mentioned. As we have stated in the *"*Training the sparse depth injector*"* section of our paper, our depth injector **"successfully infers dense and robust structural information without needing any auxiliary network for depth completion"** due to the generative capability of the diffusion model. 

We demonstrate this capability in detail in Figure 3 of our paper, where our model equipped with the Sparse Depth Injector implicitly infers the overall geometry from the sparse and erroneous point cloud depth map given and **robustly generates a realistic, artifact-free image in accordance with the general shape of the given sparse depth map.** In comparison, the same sparse point cloud map given to previous depth-conditioning diffusion models [a, b] results in **disfigured, erroneous images**, suffering from the exact problem that you have mentioned. 

We find that this ability to infer robust geometric information from sparse point clouds naturally extends to its **robust handling of erroneous point clouds and artifacts** given to it as a condition. Though there may be errors and artifacts in the predicted point cloud, our diffusion model implicitly ignores them in the process of generating realistic images, effectively having an effect of filtering out such point cloud errors when applied for 3D generation. This effect is clearly demonstrated in Figure 11 and Figure 15 of our paper, where the conditioning point cloud of the cup evidently has **erroneously missing regions near the top and the bottom**, but our model generates a whole, robust 3D cup nonetheless (Fig 11), and Figure 15 in our Appendix shows that our model is robust to the sparse nature of the point clouds, generating consistent results despite varying number and sparsity (and therefore increasing ambiguity) of the conditioning point clouds.

Reviewer comment (Novelty):
My biggest concern is the marginal improvement. It seems that the proposed model only shows a very limited improvement compared to each baseline method.

Author response:
## **W1. Regarding marginal improvement over the baseline method.**

Thank you for your comment. First, we respectfully would like to ask for clarification: could you explain to us in further detail in which aspect you have deemed our model to show ‘marginal improvement’? We understand you mean fidelity improvement which pertains to the general detail quality and realism of generated 3D scenes, and we provide our three-part response to this weakness under such an assumption. If we have misunderstood, please tell us and we will provide additional responses accordingly. 

&nbsp;

### 1. Our method is a plug-and-play module specifically designed for resolving geometric inconsistency problems of baseline text-to-3D methods.

We first emphasize that the main focus of our methodology was designing a **plug-and-play module** that is capable of **resolving the geometric inconsistency problem** of whichever score distillation sampling (SDS)-based text-to-3D baseline it is attached to. Therefore, while our model greatly enhances the geometric consistency of each baseline, as clearly demonstrated by Figures 4 and 5, its characteristics (such as fidelity of results) tend to follow those of the baseline models and their architecture. 

&nbsp;

### 2. Our model achieves drastic quality improvement, and we additionally provide extensive results to demonstrate it.

Nevertheless, we find that such capturing of geometric consistency indeed **facilitates convergence, naturally leading to increased fidelity and quality** of our results in comparison to the baseline. However, in hindsight, we understand and agree with your comment that the effect was not clearly visible in our paper because the majority of our experimental results utilized Score Jacobian Chaining (SJC) [a], an older methodology showing lower fidelity, as its baseline. 

Therefore, to more clearly demonstrate how our model achieves significant quality and fidelity improvement over its baseline, we provide **additional results** in Figure 4 of our **revised paper,** and also in our **new supplementary video.** We show how our model exhibits outstanding performance using the ProlificDreamer [b] baseline, which is far more high-fidelity and recently introduced than SJC. Our model is shown to handle highly **complex prompts** with geometric robustness, while the same prompts induce failures at the baseline model. Through these results, we demonstrate that our model achieves **diverse, highly detailed results with robust geometry** when attached to more powerful baselines. 

This enables us to conjecture that if more powerful SDS-based text-to-3D models are introduced in the future, our model will likewise be very effective in enhancing its performance both in terms of geometry and fidelity. We will provide more extensive results in the Appendix to showcase our model’s capability.  

&nbsp;

### 3. Our model adds powerful controllability to existing baselines.

Lastly, we also call to attention another important advantage of our model, which is how our model **adds significant controllability** to existing text-to-3D baselines. As we have described in Section 4.3 of our paper, “our method enables the overall shape of the scene to be decided before the lengthy optimization process”, allowing the user to more **meticulously control the generation process** and generate specific 3D scenes tailored to their needs. 

To demonstrate this controllability aspect of our method, we include additional experimental results in Part 2 of our **new supplementary video**, demonstrating that **incredibly diverse results under the same text prompt** can be generated under the **user’s control** due to our method’s unique point cloud conditioning. We will also include these results in our revised paper.

This clearly shows that even in cases of text prompts where the baseline model and our method both display exceptional results (thus ours showing **marginal improvement** compared to baseline in such cases), our method still holds a **strong advantage over the baseline** regarding the **controllability** of generated scenes, allowing the user to generate incredibly diverse results as one wishes. We would be grateful if you could take this fact into consideration as well.

Reviewer comment (Evaluation):
My understanding is that the video from supp is generated by render well-optimized NeRF, which should be inherently. But why does the video itself look aliasing?

Author response:
## **W2. Regarding the aliasing-like effect in the supplementary video.**

The reason for the aliasing-like effect that you have mentioned, shown in some of the renderings in our supplementary video, is due to the architectural choice of the SJC backbone that we have used. SJC’s backbone NeRF does not directly model pixel-level RGB but instead models latent 3D features, so **it initially renders a low-resolution feature image and then decodes it into an RGB image through a CNN** (as described in Section 5.2. of the SJC paper), resulting in aliasing-like effects that you have noticed. As this is an issue that stems from the original SJC model that its authors have uploaded, notice that the results from the baseline model also evidently display such effects.  

Due to this reason, our experimental results from the ProlificDreamer baseline, which models and **renders pixel-level RGB directly with Instant-NGP [c], do not display such an aliasing-like effect.** We provide additional results from the ProlificDreamer baseline in both our revised paper and the new supplementary video **along with their normal maps** for a clearer demonstration of how the aliasing effect is removed, so please take a look into it. This clearly shows that **this issue does not derive from any component of our proposed methodology, but rather comes from the architecture of the NeRF backbone of the baseline model that we use.**

Reviewer comment (Evaluation):
How stable is the 3D point cloud generative model? Any failure cases of predicted 3D point cloud? How to handle it?

Author response:
## **Q1. Regarding the stability and failure of the 3D point cloud generative model.**

Thank you for pointing this out. We first would like to mention that we observe that the 3D point cloud generative model we currently use, Point-E [f], shows high stability in image-to-point cloud generation when generating coarse geometry.  As our model leverages these models solely for such coarse geometry, its results seem sufficient in our framework. However, as you have mentioned, there are failure cases where artifacts and errors appear during the point cloud generation process. 

In this light, we emphasize that one of the main contributions of our paper, Sparse Depth Injector (described in Section 4.3), specifically focuses on addressing this problem. As we have stated in the "Training the sparse depth injector" section of our paper, our depth injector **"successfully infers dense and robust structural information without needing any auxiliary network for depth completion"** due to the generative capability of the diffusion model.

We find that this ability to infer robust geometric information from sparse point clouds naturally extends to its **robust handling of erroneous point clouds** and artifacts given to it as a condition. Therefore, though there may be errors and artifacts in the predicted point cloud, our diffusion model implicitly ignores them in the process of generating realistic images, effectively having an effect of filtering out such point cloud errors when applied for 3D generation. 

This effect is clearly demonstrated in multiple locations throughout our paper. In **Figure 3** of our paper, our model implicitly infers the overall geometry from the sparse and erroneous point cloud depth map given and robustly generates a realistic, artifact-free image in accordance with the general shape of the given sparse depth map. In **Figure 11** of our paper, the conditioning point cloud of the cup evidently has erroneously missing regions near the top and the bottom, but our model generates a whole, robust cup nonetheless.

Reviewer comment (Experiments):
Why should it predict a point cloud and then convert it to a depth map? What if the proposed module directly predicts a monocular depth map?

Author response:
## **Q2. Regarding directly predicting monocular depth map.**

Thank you for your suggestion. The main difficulty in directly using a depth map in our model derives from the fact that depth prediction models **only reconstruct a single side of the given scene,** and are incapable of modeling the 3D geometry of unseen, occluded areas. As it cannot predict the entire shape of the scene but gives us only **incomplete partial geometry**, it does not fit with our main purpose of removing geometric inconsistency issues from **all viewpoints** of the 3D scene, as these issues (such as multi-face Janus problem) mostly occur at side and back regions occluded from the image. Therefore, in order to model the entire scene in 3D, including the side and back regions, we need to leverage generative models that can infer geometries in unseen occluded regions with its generative prior - which is why we mainly resort to sparse point cloud generative model, Point-E.