Paper: DMV3D: Denoising Multi-view Diffusion Using 3D Large Reconstruction Model (ICLR-2024)

Reviewer comment (Experiments):
It seems the model learns from the objaverse and mvimagnet, which contain mostly single objects or separated objects. Even the examples in out of domain results, in figure 6, the objects are not complicated as people use in SD-based models.

Author response:
1. **Complex prompts**: our text-to-3D model has limited capability to handle complex prompts due to the limited scale of multi-view data. But as the reviewer points out, we can use Stable Diffusion to generate a single image from a complex prompt, and then use our image-conditioned model to lift it into 3D. We show the results here (https://dmv3d.github.io/rebuttal_sdcond.html).

Reviewer comment (Experiments):
As mentioned in weakness 1, I would like to see some results of "bunny seating on pancake", this kind of generation. Even if it is hard to do text to 3D, since the training set doesn't have compound objects, is it possible to do 2D conditioned 3D generation with this kind of prompt?

Author response:
1. **Complex prompts**: our text-to-3D model has limited capability to handle complex prompts due to the limited scale of multi-view data. But as the reviewer points out, we can use Stable Diffusion to generate a single image from a complex prompt, and then use our image-conditioned model to lift it into 3D. We show the results here (https://dmv3d.github.io/rebuttal_sdcond.html).

Reviewer comment (Experiments):
In contrast to Image-based diffusion models, the runtime efficiency might be compromised since the reconstruction model operates during every iteration of sampling. Investigating whether the reconstruction can be repurposed or distributed over the intermediate denoising phases could be insightful, especially since the current intermediate reconstructed model is discarded (would be great if they could be visualized), leading to potential wastage.

Author response:
1. **Runtime efficiency**: we agree with the reviewer that it might be a good idea to interleave geometry-based multi-view denoising and geometry-free multi-view denoising for best runtime efficiency, as rendering 4 views from a triplane at each denoising step does have a negative influence on the inference speed. We leave this point for future exploration. We also visualize the intermediate reconstructed results during denoising process at this link (https://dmv3d.github.io/rebuttal_x0vis.html).

Reviewer comment (Experiments):
How the camera viewpoints are sampled during the training process? Would the reconstruction model easily fall into a local minima where the 3D results become trivial by generating planes that are parallel to the image planes?

Author response:
3. **Viewpoint sampling**: we randomly sample 4 views during training to use as our input views, and another randomly selected 4 novel views as supervision. During inference, we sample 4 structured viewpoints around an object to denoise.  

4. **Degenerate solution**: we use novel view supervision to prevent the degenerate solution of generating 4 flat planes to explain the input views (see table 3). The results without novel view supervision can be found at this link (https://dmv3d.github.io/rebuttal_wonovelview.html).

Reviewer comment (Evaluation):
A balanced perspective is lacking due to the absence of discussion on the paper's limitations, which could provide valuable insights for potential areas of improvement.

Author response:
3. **Limitations**: one limitation with our text-to-3D model is that we cannot handle complex text prompts as well as 2D diffusion models like stable diffusion, as a result of the limited scale of multi-view data compared with 2D data. Another limitation is that the generated textures at unseen views are still a bit blurry compared with input view. It will be interesting to further boost the quality of texture sharpness in future work.

Reviewer comment (Evaluation):
Given the inherent training characteristics of diffusion models, how does DMV3D achieve a training timeframe analogous to LRM? It would be helpful if the authors could provide an explanation.

Author response:
4. **Training time**: despite our method being a diffusion model, we only sample a single time-step at a time to train our multi-view denoiser. This single-step setting is similar to what LRM does except our input images are noisy; hence our model trains as fast as LRM during training.

Reviewer comment (Novelty):
It is important to clarify the extent of overlap between this work and the LRM submission in order to understand the distinctiveness of the contributions in this manuscript.

Author response:
1. **DMV3D vs LRM**: We would like to clarify the different contributions of LRM and DMV3D. Please note that, in DMV3D, we acknowledge LRM as prior work and have explicitly cited LRM in section 3.2. Our work makes orthogonal contributions,  focusing on different challenges that are not covered by LRM. In particular, LRM focuses on the task of single-image reconstruction only and introduces the first scalable transformer model trained on massive multi-view data. However, in essence, LRM is a deterministic model that cannot model the inherent uncertainty (multiple plausible solutions exist for unseen parts of the 3D objects) in single-image 3D reconstruction. In contrast, DMV3D aims to build a general probabilistic diffusion model for 3D generation. Our framework is not only applicable to single-image reconstruction but also text-to-3D generation – a task completely uncovered by LRM.   

    To do so, we propose a novel single-stage multi-view diffusion framework and adopt a transformer model to achieve multi-view denoising. Again, please note that we have cited LRM (by anonymously putting it in the supplemental material) and clearly stated in Sec.3.2 that our transformer architecture is inspired by LRM to avoid any misconception about dual submission. We emphasize that the whole story of our paper, including our entire method section (Sec. 3), does not focus on the original transformer contribution made by LRM. Instead, our focus is mainly on: 1) how to build a multi-view diffusion framework for 3D generation (Sec. 3.1), 2) how to improve/extend the transformer model to support multi-view denoising (Sec. 3.2), and 3) how to enable both text and image conditioning (Sec. 3.3). These aspects are distinct from the scope of LRM. Furthermore, our transformer architecture is also different from the one in LRM in taking noisy multi-view images as input, being conditioned by the diffusion time step, utilizing a new ray-conditioning technique to better handle the diverse camera intrinsics, and supporting text conditioning.

    As a result, even for the same single-image reconstruction problem, DMV3D and LRM address it in very different ways. DMV3D leverages multi-view diffusion denoising, whereas LRM  performs single-view image-to-NeRF translation. The transformer inputs (multi-view noisy images vs a single-view clean image) and the inference processes (DDIM-based multi-step diffusion denoising vs single-step transformer inference) are highly different. Moreover, because of being a probabilistic generative model, DMV3D can generate multiple high-quality 3D assets given the same single image input (as shown in Fig. 1 and our project page), while LRM can only generate one single result. The probabilistic modeling introduced by DMV3D also enhances the realism and reduces the blurriness of generated textures for unseen parts; in contrast, LRM tends to generate blurred results due to mode averaging. 

    In sum, different from the deterministic model in LRM, our DMV3D introduces a novel probabilistic model for 3D generation and contributes 1) a novel multi-view diffusion framework 2) the general support for both text and image conditioning and  3) new architecture designs to support multi-view denoising and text/image conditioning. The contributions are made orthogonal to those of LRM and we believe these are valuable contributions to the field towards a general, practical, and high-quality 3D generative model.

Reviewer comment (Experiments):
For multi-view diffusion, what is the number of views for training and inference?

Author response:
We thank the reviewer for your appreciation of our method and result. We address your questions below.

1. Train from scratch: all of our models are trained from scratch without using pretrained diffusion models. Generalization of our model is backed up by large-scale training on the massive multi-view data including Objaverse and MvImgNet (See Fig. 7). 

2. Number of views: we use 4 views during training and inference (see section 4.3).