Paper: SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (ICLR-2024)

Reviewer comment (Writing):
There are too many equations. It's better to use fewer equations and more intuitive sentences for method description for application papers.

Author response:
Thanks for your advice. Following your suggestion, in the supplementary material, we further provide a diagram in Fig. 9 to intuitively show the derivation of the multiview diffusion process.

Reviewer comment (Evaluation):
Why the generated image is not always plausible? What's the success rate for syncdreamer?

Author response:
Generating multiview images from a single-view input is an ill-posed problem and accurately learning the joint distribution of multiview images is still a challenging task so the generated images are not plausible. Though SyncDreamer shows some promising results, its generation ability is still limited by training on the Objaverse dataset with thousands or millions of objects, which is still smaller than 2D datasets with billions of images. 

Exactly defining whether a generation result is successful or not is ambiguous and difficult because the generation should give a plausible distribution not a deterministic answer. Instead, we report the min/max/average metrics on the GSO dataset of our method in Table 1 of the supplementary material, which may indicate that SyncDreamer successfully generates good examples of these objects. Exactly evaluating the generation ability on the single view reconstruction task is still an unexplored and interesting problem.

Reviewer comment (Presentation):
No failure case visualization.

Author response:
Some failure generations are shown in Figure 2 of the supplementary material.

Reviewer comment (Evaluation):
Why using stable diffusion initialization is worse than zero-1-to-3 initialization?

Author response:
Finetuning from the Stable Diffusion model still produces multiview-consistent images but with worse quality than finetuning from Zero123. Based on our observations, we find that the batch size plays an important role in enhancing the stability and efficacy of learning 3D priors from a diverse dataset like Objaverse. However, due to limited GPU memories, our batch size is set to 192 which is much smaller than the 1536 used by Zero123. Therefore, we choose to rely on Zero123 to leverage its strong 3D priors trained with large batch sizes. If we use advanced training strategies like accumulating gradients or using more GPUs, finetuning from Stable Diffusion may produce improved results. We included an additional explanation in Sec. 4 of the main paper.

Reviewer comment (Experiments):
Do you try Neus2 instead of Neus? Will Neus2 improve the reconstruction results in terms of time and accuracy?

Author response:
Thanks for this valuable suggestion. NeuS2 mainly relies on the hash-grid representation to speed up the training speed. We have tried hash-grid-based NeuS in Sec. 1.9 and Fig. 8 of the supplementary material. Using a hash-grid-based NeuS indeed improves the training speed from 10 minutes to about 3 minutes with visually similar quality.

Reviewer comment (Experiments):
One notable weakness of SyncDreamer is its computational cost, particularly the significant memory requirements associated with the volume-based approach. This could pose scalability challenges when aiming for high-resolution image generation, which is an important consideration for practical applications.

Author response:
We agree that the volume construction consumes relatively large GPU memory and computations and limits the scalability to higher resolutions. Currently, we can generate 4 instances (64 images) on a 40G A100 GPU using about 40s, which is slower than Zero123 using 23s to generate 64 images on the same GPU. Exploring more effective and efficient ways to correlate multiview images is indeed an interesting and challenging problem and we will continue to delve deeper.

Reviewer comment (Experiments):
A noticeable gap in the level of detail between images generated by Zero123 and SyncDreamer raises questions about the specifics of this discrepancy. Given that SyncDreamer leverages Zero123 as a pretrained model and maintains it frozen during training, it would be valuable to investigate whether this difference is influenced by the volume-based attention mechanism or other factors, providing a deeper understanding of the model's limitations.

Author response:
The generated textures are sometimes less detailed than the Zero123. The reason is that the multiview generation is more challenging, which not only needs to be consistent with the input image but also needs to be consistent with all the other generated views. Thus, the model may tend to generate large texture blocks with less detail, since it could more easily maintain multiview consistency. We have added this discussion to the limitations of Sec. 1.3 in the supplementary material.

Reviewer comment (Experiments):
The fact that SyncDreamer is fine-tuned on Zero123 raises concerns about its generalizability and performance when trained from scratch, especially considering that both models use the same datasets. If SyncDreamer heavily relies on a strong backbone like Zero123 to achieve its results, its broader applicability might be limited, and it would be beneficial to explore how well it performs without the pretraining.

Author response:
Finetuning from the Stable Diffusion model still produces multiview-consistent images but with worse quality than finetuning from Zero123. Based on our observations, we find that the batch size plays an important role in enhancing the stability and efficacy of learning 3D priors from a diverse dataset like Objaverse. However, due to limited GPU memories, our batch size is set to 192 which is much smaller than the 1536 used by Zero123. Therefore, we choose to rely on Zero123 to leverage its strong 3D priors trained with large batch sizes. If we use advanced training strategies like accumulating gradients or using more GPUs, finetuning from Stable Diffusion may produce improved results. We included an additional explanation in Sec. 4 of the main paper.

Reviewer comment (Theory):
The concatenation of features across all views in the volume, without a mechanism to handle occluded views, might lead to suboptimal performance, particularly in cases involving heavily occluded objects. A more refined approach to selectively attending to relevant views during convolutional operations could enhance the model's robustness.

Author response:
Thank you for this suggestion. We agree that using a more advanced attention mechanism to make the feature volume occlusion-aware could be a promising technique to further improve the quality (though with some additional computation and memory consumption). We will explore this interesting idea in the future.

Reviewer comment (Experiments):
The paper chooses to use 16 views for its volume-based approach. It would be helpful to understand the rationale behind this specific choice. Is 16 views the minimal number required for effective multiview image reconstruction, or was this number selected for a different reason? Given the extra computational burden associated with more views, why not start with a smaller number, like 8 views, to assess the trade-off between computational efficiency and performance?

Author response:
NeuS usually requires more than 50 images for accurate reconstruction, while simultaneously generating this amount of images using stable diffusion is too expensive. As a trade-off, we adopted a 16-view setting to balance accuracy and cost. Following your suggestions, we conducted further experiments and found that it is possible to generate fewer views for reconstruction. The experiments are given in Sec. 1.8 of the supplementary material where we use 4 or 8 generated views for NeuS reconstruction. We observed that using 8 views still leads to reasonable reconstruction while 4 views are not enough for a good reconstruction. Generating 8 views may further improve the generation efficiency of SyncDreamer.

Reviewer comment (Evaluation):
I tried the model and found its sensitivity to the size of objects when using the code and released models. Is this sensitivity a result of the volume-based approach or if there are specific reasons behind this behavior. Understanding the limitations related to object size could provide insights into potential improvements or workarounds. Please see below for the attached images: if I scale the 'monkey' a little bit more, the generation could be very different.

Author response:
The reason is that changing the foreground object size corresponds to adjusting the perspective patterns of the input camera and finally affects how the model perceives the geometry of the object. The training images of SyncDreamer have a predefined intrinsic matrix and all are captured at a predefined distance to the constructed volume, which makes the model adapt to a fixed perspective pattern. Thus, our generation results are sensitive to the crop size of the images. We have added this explanation to the Sec. 1.2 of the supplementary material.

Reviewer comment (Experiments):
Epipolar-based attention mechanisms, as demonstrated in works like Generalizable Patch-Based Neural Rendering (ECCV 2022), are technically equivalent to volume-based attention. Did the authors consider or investigate this direction, and do they have any comparative results or insights regarding the choice of attention mechanism in SyncDreamer?

Author response:
Epipolar line attention constructs epipolar lines on every image and applies attention along epipolar lines. Our depth-wise attention is very similar to such an epipolar line attention. If we project a 3D point in the view frustum onto a neighboring view, we get a 2D sample point on the epipolar line. In epipolar line attention, we need to maintain a new tensor of size $H\times W \times D$ containing the epipolar features. This would also cost as large GPU memory as our volume-based attention. We have added this discussion to the Sec.1.11 of the supplementary material.

Reviewer comment (Writing):
The biggest weakness of this paper is its writing. I feel the notations used in section 3.2 are very confusing. Following the standard notations in score-based generative model literature, for quite a while I thought the paper somehow extended the UNet so that it is able to take multiple noisy images as input, but in fact if I’m not mistaken, it still just takes a single noisy image as input, and the conditioning from the other noisy target views are just the injected features.

Author response:
We extend the diffusion models to process multiview images. According to the derivation in Sec. 3.1 and Sec. 3.2, we indeed require a model to take multiview images to get denoised multiview images. A direct implementation of the multiview diffusion formulation is to use $N$ different image denoisers and each one is in charge of denoising a specific one of the multiview images. This solution would be too expensive. In our task, we find that the multiview diffusion can also be implemented by a single-view denoiser to repeatedly denoise $N$ images with all other views as conditions, which is cheaper and easier to implement. Because the single-view denoiser can be pretrained from large-scale 2D datasets such as the Stable Diffusion, which improves the generalization ability of the model.

Reviewer comment (Writing):
A small introduction to zero123 could be helpful, as many of the design choices are very specific to zero123 and require some prior knowledge of how zero123 works.

Author response:
Thanks for your suggestion. We have added a description of zero123 in Section 3.3.

Reviewer comment (Experiments):
The model seems to be limited to a fixed set of viewpoints based on the concatenation design for constructing the 3D feature volumes. In contrast, the initial diffusion model, Zero123, can generate novel-view images at any specified viewpoint. The experiments in the supplementary material also show that a new model needs to be trained to generate 16 images with a different elevation (i.e., 0$^\circ$). Although the model successfully generates diverse 3D objects in the current setting (16 images at an elevation of 30$^\circ$), this limitation still narrows the scope of application of the proposed method. Further analyses/discussions will be helpful: i) what will happen if the same model generates *another 16 images* with different elevations using one of the generated images as the condition? Will the model fail terribly, or some reasonable results can still be produced? ii) The authors might argue that a trained NeRF or NeuS could synthesize other viewpoints, but the quality of the rendered images may be lower than that of the ones generated by SyncDreamer, and some related results would be appreciated.

Author response:
**A1**: We agree that it is a limitation that we can only generate images on fixed viewpoints. The choice of using fixed viewpoints is based on the following considerations: 1) Learning the multiview joint distribution on fixed viewpoints is easier than on arbitrary viewpoints. 2) A set of fixed viewpoints around the object already enables complete 3D reconstruction for most cases. We think it would be a promising future direction to generalize to arbitrary viewpoints. We have added this to the limitation in the supplementary.

i) We included the re-generated results in Sec. 1.6 of the supplementary materials (Fig. 5), which are plausible but different from the original input. Since SyncDreamer is designed to generate images of the absolute elevation of 30 degrees on the object, the re-generated images are still fixed on the elevation of 30 degrees.

ii) We show the novel-view renderings of the NeuS trained on the generations of SyncDreamer in Sec. 1.7 (Fig. 6) of the supplementary material. The image quality of the NeuS is reasonable but indeed blurry than the generated images of SyncDreamer.

Reviewer comment (Evaluation):
For quantitative evaluations, the settings are unclear. The generation process has randomness, so how are the generation results used in quantitative evaluations determined? Is it running multiple rounds and then taking the average metrics, or taking the best numbers out of multiple rounds? The concrete settings must be clarified, and it would be better to report the mean and standard deviation if multiple rounds of generation are executed for evaluation.

Author response:
**A2**: For the metrics computation in the main paper, we generate one instance and compute the metrics using this generated instance for all baseline methods and our method. To show the randomness, we generate 4 instances for an input image using SyncDreamer and report the min/max/average metrics in Table 1 of the supplementary materials. We agree that exactly evaluating the generation quality for single-view reconstruction is difficult because the generated results can be entirely plausible but not identical to the ground truth.

Reviewer comment (Novelty):
The ablation studies in Fig.7 seem to suggest that the proposed model components (3D feature volume + depth-wise attention) cannot learn proper 3D shape priors, and the method has to rely on a frozen Zero123 -- when the UNet weights are fine-tuned together with the new modules, the geometry of the generated images becomes broken. If this speculation is true, the proposed modules can only learn local consistency and must be plugged into a base model with 3D priors (e.g., Zero123) for generating plausible multiview images. This is not necessarily a flaw, but it indeed narrows down the scope of the paper, as the proposed method is more like an extension of Zero123 rather than a general solution for multiview image generation.

Author response:
**A3**: Finetuning from the Stable Diffusion model still produces multiview-consistent images but with worse quality than finetuning from Zero123. Based on our observations, we find that the batch size plays an important role in enhancing the stability and efficacy of learning 3D priors from a diverse dataset like Objaverse. However, due to limited GPU memories, our batch size is set to 192 which is much smaller than the 1536 used by Zero123. Therefore, we choose to rely on Zero123 to leverage its strong 3D priors trained with large batch sizes. If we use advanced training strategies like accumulating gradients or using more GPUs, finetuning from Stable Diffusion may produce improved results. We included an additional explanation in Sec. 4 of the main paper.

Reviewer comment (Reproducibility):
The details of the mesh reconstruction process are missing. The authors mentioned in the caption of Fig.1 that NeuS is used to reconstruct the mesh from the generated images, but there are no relevant implementation details. How long does the NeuS training take, and does it always produce reasonable reconstruction results (as the generation results are random, sometimes wrong or inconsistent images can be generated)?

Author response:
**A4**: We train NeuS for 2k steps with 4096 rays on every step, which costs about 10 minutes. In all our examples, NeuS succeeds in reconstructing the shapes even though some minor inconsistencies exist. The reason is that the MLP-based representation of NeuS inherently has a strong regularization ability to smooth out these minor inconsistencies and produce a plausible reconstruction. We have included additional implementation details about NeuS in Sec. 1.1 of the supplementary materials.

Reviewer comment (Novelty):
Some recent works, like [MVDiffusion](https://arxiv.org/abs/2307.01097) and [MVDream](https://arxiv.org/abs/2308.16512), can also generate consistent multiview images while their methods directly exchange multiview feature inside UNet blocks. It would be beneficial if the authors could discuss the pros and cons of the proposed method and those relevant ones.

Author response:
**A5**: MVDiffusion utilizes a correspondence-aware attention layer based on the known geometry. In SyncDreamer, the geometry is unknown so we cannot build such one-to-one correspondence for attention. MVDream applies attention layers on all feature maps from multiview images, which also achieves promising results. Applying such an attention layer on all the feature maps of 16 images in our setting costs unaffordable GPU memory in training. Finding a suitable network design for multiview-consistent image generation would still be an interesting and challenging problem for future work. We have included an additional discussion in Section 1.11 of the supplementary material.

Reviewer comment (Experiments):
Multi-view image condition: is it possible to use the denoised multi-view images rather than the noisy images from the last step as a condition to the diffusion model?

Author response:
SyncDreamer generates multiview images simultaneously in one reverse process, which means all images are denoised at the same time. When denoising one of these images, the other images are also in the generation process and are still noisy so we can only condition on their noisy states. If we adopt an alternative way to generate images sequentially, we can condition on the previously denoised image. However, the sequential generation process can easily suffer from error accumulation and lack of global consistency.

Reviewer comment (Reproducibility):
3D CNN: how is the 3D CNN trained? Is it pre-trained or trained with the attention module together in the finetuning?

Author response:
The 3D CNN is not pretrained but is trained together in finetuning the diffusion models to generate multiview images.

Reviewer comment (Experiments):
View frustum volume: will it be too heavy to pass a view frustum volume to the UNet as the multi-view condition? Also, how is W in the view frustum volume determined?

Author response:
Thanks for your suggestion and we added the description of the volume size in the implementation details (Sec 1.1) of the supplementary material. The spatial volume has the size of $32^3$ and the view-frustum volume has the size of $32\times 32 \times 48$. We sample 48 depth planes for the view-frustum volume because the view may look into the volume from the diagonal direction. The choice of these sizes is based on the consideration that the latent feature map size of a $256\times 256$ image in the Stable Diffusion is exactly $32\times 32$. These volumes indeed bring additional memory and computation consumption, but the overall cost is still acceptable based on our experiments.