Paper: MVDream: Multi-view Diffusion for 3D Generation (ICLR-2024)

Reviewer comment (Evaluation):
One potential major weakness is that the robustness of the method does not seem to be tested. In terms of the generated results, it would be good to see multiple versions generated from the same prompt (i.e., corresponding to something like variance in quantitative metrics from various trained models). This is very important because text-to-3D is notoriously brittle, and it's important to understand if the proposed contribution removes the Janus problem 5% of the time, 50% of the time, or 100% of the time. Without this sort of comparison and extensive evaluation, it's impossible to judge the magnitude of the contribution.

Author response:
In the anonymous project page https://mvdream-anonymous.github.io/test_0.html, we have shown the qualitative comparison over 40 prompts without cherry picking nor prompt-wise parameter tuning. It could be seen that our model almost eliminates the multi-face Janus problem by a 100% chance (with a fixed configuration). We have also conducted an experiment for the same prompt but different random seeds. However, we found the difference to be rather subtle because of the large CFG guidance used during SDS: https://github.com/mvdream-anonymous/mvdream-anonymous.github.io/blob/master/static/seeds.mp4

Reviewer comment (Experiments):
A minor weakness is that there seem to be a large number of hyperparameters where it's not entirely clear why their values were chosen. Why use a multi-view diffusion model outputting only 4 views instead of more? The justification for how camera parameters are embedded also seems a bit ad-hoc. I don't see this as a major weakness as they lead to good performance, but I wonder about how brittle the method is as a result.

Author response:
- **number of views** Since we are proposing a new type of neural networks, it will inevitably comes with a few new hyper-parameters. For the number of views, as shown in Figure 11 in Appendix A.2, we found 4 views to a minimum number of views to ensure multi-view consistency for SDS. Therefore, we choose to stop at 4 views to balance the tradeoff between performance and resource cost. 
- **camera parameters** In our experiments, we have tried different types of camera inputs (16-dim extrinsics and 25-dim intrinsics+extrinsics) and different ways to embed them (cross attention vs. time embedding). All of these choices work to some extent. So we choose our current design due to its simpleness and empirical robustness.
- Overall, our pipeline already has much less hyper-parameters compared with prior studies such as Dreamfusion and Magic3D. In particular, our model can in fact use SDS loss alone to generate 3D content while prior studies have many regularization losses to ensure 3D quality. Similar to the response to Question 1, we believe that the stableness of our method can be evaluated from the visual examples in the project page.

Reviewer comment (Presentation):
It is unclear what the point of each experiment is, as there is no overview section to clarify this. Although it can be inferred after multiple reads, the reading experience is challenging.

Author response:
we have added an overview paragraph in the revised paper to explain the experiments. In particular, we evaluated our method on three tasks: (1) multi-view image generation for evaluating image quality and consistency (Sec. 4.1), (2) 3D (NeRF) generation with multi-view score distillation as a main downstream task (Sec. 4.2), and (3) DreamBooth for personalized 3D generation (Sec. 4.3).

Reviewer comment (Experiments):
While it is interesting to use a text-to-multi-view model to train a text-to-3D model, it raises the question of how well the multi-view diffusion model would work to generate 3D directly.

Author response:
In order to generate 3D content directly without score distillation (optimization), a few-shot reconstruction model will be needed to generate 3D content from 4 views. However, we do not have access to such a few-shot reconstruction model for the moment. We believe this is achievable by training such a model or extending multi-view diffusion to more views. But this is out of the scope of this paper and will be part of the future work.

Reviewer comment (Experiments):
Many experimental details are under-explained. For example, the attention module choice experiment is difficult to understand because Section 4.1 contains both the overview and results in the same paragraph. The experiment is not fully specified, missing details such as the caption for Figure 4 or specifics of the 3D attention. Additionally, the experiment consists of only one qualitative example, making it difficult to draw confident conclusions.

Author response:
We will try to add more explanations to the experiments, including the attention part. The main motivation of the attention experiment is to compare the multi-view consistency, for which we are not aware of any good quantitative metrics for the moment. So we choose to present qualitative results instead.

Reviewer comment (Presentation):
In Figure 5, the qualitative results are presented within the same paragraph as the experiment in Table 4. It is again unclear why the training set is shown and whether the results are from the text-to-multi-view or text-to-3D model.

Author response:
These are the images rather than the 3D models generated by our model. Here, we show both in-domain data (training) and out-of-domain examples (conterfactual text descriptions). We will explain this Figure more clearly.

Reviewer comment (Experiments):
The significance of adding negative prompts or CFG rescaling is not clarified anywhere in the paper.

Author response:
The motivation of negative prompts and CFG rescaling were mentioned Sec 3.3. We will add a reference in the experiment back to this section.

Reviewer comment (Evaluation):
The Dreambooth results are only briefly mentioned and seem disconnected from the rest of the paper. It is unclear why they are included or why they work.

Author response:
Dreambooth is an extended experiment of our multi-view diffusion model to evaluate its generalizability. It shows the capability how a multi-view diffusion model connects the knowledge from 2D and 3D domain. The explanation of why DreamBooth would work is a tricky problem by itself. Some of the explanations could be found in the original DreamBooth paper. In terms of why it would work for multi-view generation, we think this is because our model is essentially sharing the same modules and weights for 2D and multi-view generation. Thus, the knowledge learned in 2D domain could be potentially transferred to multi-view generation.

Reviewer comment (Experiments):
The main comparison in Fig. 6 is not fair. All other methods are optimization-based by reusing 2d image diffusion. However, the proposed method used additional datasets. A fair comparison would be zero123+SDS?

Author response:
Yes, we do have compared with Zero123XL+SDS, the results are shown in the appendix A.1 and Figure 10 in the original submission. The basic conclusion is that Zero123+SDS tends to blur and flatten the back of objects if the input image has complex poses or details. In contrast, MVDream can synthesize self-consistent 3D content from the text directly.

Reviewer comment (Writing):
Following the above comment, the trained model should be able to combine with SJC, VDS, and maybe other variants of SDS. This should be discussed somewhere in the main paper or the appendix.

Author response:
Theoretically our model is combinable with other SDS variants. For the time limit, we do not have a chance to explore this direction since we think the current SDS results are good enough. We will added such a discussion to the revised paper (Sec 3.2)

Reviewer comment (Writing):
Some description of the training algorithm is still confusing. For example, what is the so-called "3d self-attention"? I assume this is done by reshaping FxHxWxC to (FxHxW)xC and then applying self-attention.

Author response:
- 3D-attention: yes, that's what it is. We will try to explain it more clearly.

Reviewer comment (Reproducibility):
Some description of the training algorithm is still confusing. For example, 32 views are rendered during data preprocessing. How do the authors sample 4 views during training? Uniformly?

Author response:
- Yes. We have explained the sampling of frames in implementation details.

Reviewer comment (Reproducibility):
Some description of the training algorithm is still confusing. For example, I hope there will be an algorithm to describe the training process.

Author response:
- We will add an algorithm block

Reviewer comment (Evaluation):
Following the above comment, the method is trained with a large 3d dataset, however, the generating process still relies on an optimization process which is slow. This limits the potential usage of the method.

Author response:
We agree with the reviewer that the current optimization process is still taking too long for many use cases. However, as discussed in Section 3.2,  we identify two ways of utilizing a multi-view diffusion model for 3D generation: (1) score distillation and (2) use generated multi-view images for few-shot reconstruction. The latter could be a good strategy for accelerating the overall generation process. But since we do not have such a robust 4-view reconstruction model for the moment, we are not able to demonstrate the results of such a pipeline.

Reviewer comment (Theory):
How to ensure consistency across multi-view images? There is no particular 3D prior learned in the proposed method.

Author response:
It is true that MVDream does not include any architecture design to inject induct bias to ensure multi-view consistency. The key insight here is that 3D-specific architecture design prohibits the model from share the architecture and weights with 2D image generation models, and consequently limited in terms of generalizability. \
Therefore, in MVDream, we seek a pure data-driven approach, where we simply re-use image diffusion models for multi-view generation. We believe that this is achievable because dense multi-view generation can be regarded as a special case of video generation, which has already been shown successful. Besides, Zero123 does not have any inductive bias to ensure view consistency either but it is able to generate promising novel views.\
Once our data-drive approach is proved feasible (which we believe is proved by our generated images), it will be more flexible to be combine with different 2D diffusion models to extend 2D vision prior to 3D.

Reviewer comment (Reproducibility):
What does re-using 2D self-attention mean? Does it mean inheriting the weights from a pre-trained model or just adopting the same architecture?

Author response:
This means (1) we inherit the weights of 2D self-attention layers from Image Diffusion UNet, and (2) during reshaping BxFxHxWxC to Bx(FxHxW)xC for self-attention. (for original 2D models, it was Bx(HxW)xC for self-attention).

Reviewer comment (Presentation):
The description of the training set should be included in the main paper rather than the appendix.

Author response:
We will add the training set description into the main paper.

Reviewer comment (Experiments):
Regarding multi-view consistency, it is also important to train the model only on the 3D dataset to see the effect of the utilization of two datasets.

Author response:
We have compared with such a model trained on 3D data only, which is discussed in Table 1 in Sec 4.1 and Figure 12 in Appendix A.3.

Reviewer comment (Experiments):
In Table 1, are the chosen 1000 subjects from the held-out testing set or a subset of the training set? Moreover, in terms of image quality, it would be better if compared with the current popular method such as zero-123.

Author response:
The evaluation was conducted on a subset from the training set because it was a common practice to calculate FID on training set[1][2]. This being said, we have updated the revised paper by recalculating the metrics on a held-out validation set. We then also evaluated Zero123-XL on the same validation set. But it is worth noting that Zero-123 is an image conditioned model, so the comparison will not be fair. The results are as follows:
| Method | Condition | FID | IS | CLIP |
| --- | --- | --- | --- | --- |
| Zero123-XL | Image |  13.91 | 11.77±0.66 | 26.89±3.25 |
| Ours | Text |  39.04 | 12.77±0.60 | 30.38±3.50 |
To evaluate the Zero123 model, we feed one of the ground-truth view (image) of each sample to generate 4 novel views. Because of this, Zero123's results will have a smaller FID, which measures semantic difference between the generated and ground-truth distributions. In contrast, our models shows higher Inception Score and CLIP, which are metrics to evaluate image quality and text-image consistency.


[1] Karras, Tero, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. "Analyzing and improving the image quality of stylegan." In CVPR 2020.
[2] Ho, Jonathan, Ajay Jain, and Pieter Abbeel. "Denoising diffusion probabilistic models." In NeurIPS 2020.