Paper: Consistent123: Improve Consistency for One Image to 3D Object Synthesis (ICLR-2024)

Reviewer comment (Evaluation):
Smoothness in Results. I noticed that the Consistent123 approach generally produces smoother results and seems to miss out on some finer details compared to Zero123. This observation is particularly evident in the first row of Fig. 4, and in the hat geometry depicted in Fig. 7: both in sections (a) bottom and (b) up. Have the authors identified potential strategies or modifications to address this shortcoming?

Author response:
Consistent123 may produce smooth texture in synthesized views for some cases. Have the authors identified potential strategies or modifications to address this shortcoming?

It is an important but challenging task for NVS methods to predict novel views with finer details. To further solve the smooth texture problem, the following techniques may help:

- Lower the PCFG guidance scale and increase inference steps. For all cases in the paper, we use the fixed schedule for progressive classifier-free guidance without per-case tuning to perform fair comparisons.  For some specific cases as mentioned above, the starting and ending points of progressive classifier-free guidance can be tuned (lower for better texture) to alleviate the smoothness problem. Moreover, using larger DDIM sampling steps (e.g., 200) also helps to improve the synthesized texture.
- Improve the quality of training data. As mentioned in other concurrent works (e.g., Instant3D), there is a lot of low-quality training data in Objaverse for novel view synthesis, which may result in the smoothness of the synthesized texture. A careful dataset pruning strategy may further alleviate this problem.

Reviewer comment (Experiments):
Concerns regarding arbitrary-length sampling. The methodology adopted uses a fixed number of views (8 views) during training. I'm concerned that this fixed view might adversely affect performance when dealing with arbitrary-length sampling at inference. This concern arises from a potential mismatch between training and test distributions. Any reason why using fixed number views during training? Is it only for simple implementation and faster training? An ablation study showcasing the performance with a random number of views as well during training would provide valuable insights and address this concern.

Author response:
What is the performance with a random number of views as well during training?

It is interesting to use a random number of views during training. We implement it by selecting a random number of views dynamically for each batch, which ranges from 8 views to 16 views (since we only render 18 views for each object). We evaluate the consistency score on GSO as shown in the following table. It shows that training at a fixed number of views is already effective, and training at a random view may degrade the performance. This is mainly because as the number of training views increases, the difference among views is smaller. It becomes easier for cross-view attention to predict the novel views, thus harming the robustness of the learned model and may lead to the overfitting issue.

|           Method           | PSNR  | SSIM | LPIPS |
| :------------------------: | :---: | :--: | :---: |
|          Zero123           | 22.88 | 0.92 | 0.25  |
|        Zero123 + SC        | 22.30 | 0.93 | 0.21  |
|  Consistent123 (8 views)   | 27.98 | 0.98 | 0.11  |
| Consistent123 (8~16 views) | 26.25 | 0.96 | 0.14  |

Reviewer comment (Experiments):
Ablation on Zero123 pretraining. Could you present results when Consistent123 is trained from scratch without Zero123 pretraining?

Author response:
Can Consistent123 be trained from scratch?

For a fair comparison, we train Consistent123 from scratch on the same renderings as Zero123. We show the consistency score evaluated on GSO in the following table:

|              Method               | PSNR  | SSIM | LPIPS |
| :-------------------------------: | :---: | :--: | :---: |
|   Zero123 (from scratch, 105k)    | 22.88 | 0.92 | 0.25  |
|   Consistent123 (finetune, 10k)   | 25.82 | 0.96 | 0.15  |
| Consistent123 (from scratch, 20k) | 23.59 | 0.92 | 0.17  |

In our experiments, we found that it is challenging to simultaneously maintain the synthesized view quality and consistency. Due to the limited scale of the existing 3D dataset, the model is easy to be trapped in overfitting when directly trained from scratch. To reduce the overfitting, it is reasonable to enable the model with novel view synthesis ability by training like Zero123, then improve the view consistency as Consistent123. Moreover, due to the slow training process (over 7 days for zero123), there is no further time to explore a more effective optimization strategy. A better training strategy (e.g., Efficient 3DiM) may make it possible to train Consistent123 from scratch.

Reviewer comment (Experiments):
It makes the paper stronger if you can show better 3D reconstruction results. For example, you can use your Consistent123 inside RealFusion [1] and Magic123 [2] to show state-of-the-art image-to-3D results.

Author response:
It makes the paper stronger if you can show better 3D reconstruction results (e.g. use Magic123).

Better 3D reconstruction results using Magic123 are shown in the video supplementary (magic123.mp4) and Figure 17.

Reviewer comment (Presentation):
Suggest to add more views in Fig 1 since there are empty space. You can also point out in the figure where Zero123 fails and you succeed to catch the audiences’ attention quickly.

Author response:
Minor suggestions mentioned in the review. 

We thank you for all the kind suggestions and will consider them in the revised paper.

Reviewer comment (Presentation):
Show back views in Fig 7 (a) bottom and (b) up.

Author response:
Minor suggestions mentioned in the review. 

We thank you for all the kind suggestions and will consider them in the revised paper.

Reviewer comment (Presentation):
Better to add cross attention between views in Fig.2 (a) as well, like a few red lines across views at the denoised views.

Author response:
Minor suggestions mentioned in the review. 

We thank you for all the kind suggestions and will consider them in the revised paper.

Reviewer comment (Novelty):
The attention mechanisms are totally borrowed from previous work, such as shared self-attention from Cao et al. and cross-attention from video diffusion models.

Author response:
Although the high-level idea of attention mechanisms is inspired by video generation, they still remain unexplored in 3D generation and can not be directly applied for novel view synthesis. We make several key modifications to adapt these components for better view consistency.

- Shared self-attention is an inference-only strategy in Cao et al., but we adapt it for both training and inference in Consistent123 to further enhance the alignment between novel views and input views.

- To attend to the input view latent, we fix the first pose of the view sequence to be the same as the input image, thus avoiding additional inverse operations and making the training process more efficient.

- We apply shared self-attention in all the diffusion timesteps, which only part of timesteps are used in Cao et al. We also conduct an ablation study in Table 3(b) to decide the position to apply for shared self-attention.

- Finally, other important components of Consistent123 also significantly contribute to the view consistency, including the flexible number and pose at inference (while the frame length in video generation is typically fixed) and progressive classifier-free guidance.

Reviewer comment (Evaluation):
For the shared self-attention layers, when the views are totally orthogonal, how will this shared self-attention act? Can this self-attention find correct correspondence? For example, in Figure 5 (right), when there is no shared self-attention, the resulting first view seems much more interesting. It would be better to have self-attention visualization in this case.

Author response:
Intuitively, with a larger pose difference between the input view and synthesized view, the shared self-attention would become less activated. For these poses, the non-shared self-attention (i.e., U-Net Encoder part) and cross-view attention layers can serve as complementary and the performance can be maintained. 

For a better understanding of how shared self-attention works, we conducted an ablation study. For the poses whose azimuth angle with input view is larger than 90°, we set the shared self-attention to vanilla self-attention by force. We evaluate the consistency score on GSO in the following table. The table shows that the constrained version of shared self-attention achieves similar performance as the vanilla self-attention. This indicates that the performance will degrade when disabling shared self-attention when a large pose difference exists, which makes the model harder to learn when attending keys and values from different views (half of views attend to input view, half attend to themselves).

|              Method               | PSNR  | SSIM | LPIPS |
| :-------------------------------: | :---: | :--: | :---: |
|      Vanilla Self-attention       | 24.64 | 0.94 | 0.17  |
|       Shared Self-attention       | 27.98 | 0.98 | 0.11  |
| Constrained Shared Self-attention | 24.67 | 0.94 | 0.17  |

Reviewer comment (Experiments):
Considering Objaverse has 800K+ 3D models, the authors only picked up 100 objects for Table 1, which seems far from enough.

Author response:
We also agree that more evaluation objects help to reduce the randomness of quantitative results. However, the evaluation of the consistency score is quite time-consuming (constructing a Nerf per object), preventing us from using a large number of evaluation objects. Moreover, our evaluation settings is mainly borrow from Zero123, which only use 20 objects in GSO (as well as most concurrent works). 

To demonstrate more convincing results, we extend the number of evaluation objects to 1k in the following table, which shows that the proposed model consistently outperforms baselines by a large margin.

|    Method     | PSNR (100) | SSIM (100) | LPIPS (100) | PSNR (1000) | SSIM (1000) | LPIPS (1000) |
| :-----------: | :--------: | :--------: | :---------: | :---------: | :---------: | :----------: |
|    Zero123    |   21.72    |    0.92    |    0.23     |    20.90    |    0.89     |     0.25     |
| Zero123 + SC  |   22.09    |    0.92    |    0.21     |    20.78    |    0.89     |     0.24     |
| Consistent123 |   24.98    |    0.96    |    0.14     |    23.89    |    0.94     |     0.16     |

Reviewer comment (Experiments):
The proposed Consistent123 loads pretrained weight from Zero123 and fixes these weights. It would be fair to also have a version training from scratch.

Author response:
For a fair comparison, we train Consistent123 from scratch on the same renderings as Zero123. We show the consistency score evaluated on GSO in the following table:

|            Method            | PSNR  | SSIM | LPIPS |
| :--------------------------: | :---: | :--: | :---: |
|           Zero123            | 22.88 | 0.92 | 0.25  |
|  Consistent123 (finetuned)   | 25.82 | 0.96 | 0.15  |
| Consistent123 (from scratch) | 23.59 | 0.92 | 0.17  |

In our experiments, we found that it is challenging to simultaneously maintain the synthesized view quality and consistency. Due to the limited scale of the existing 3D dataset, the model is easy to be trapped in overfitting when directly trained from scratch. To reduce the overfitting, it is reasonable to enable the model with novel view synthesis ability by training like Zero123, then improve the view consistency as Consistent123. Moreover, due to the slow training process (over 7 days for zero123), there is no further time to explore a more effective optimization strategy. A better training strategy (e.g., Efficient 3DiM) may make it possible to train Consistent123 from scratch.

Reviewer comment (Theory):
Will the compromise solution introduce view inconsistency? It looks like there is no connection between the sampled views and the next round views.

Author response:
- First, Consistent123 can synthesize highly consistent multi-views when the number of synthesized views is large. Although there is no direct connection between sampled views and next-round views, the pose difference between the next-round view would be very small, thus the inconsistency can be ignorable.
- Second, our models can generate more than 100 views simultaneously, which to our best knowledge is one of the largest among all concurrent methods and totally enough for common cases.

Reviewer comment (Evaluation):
The results on 3D reconstruction seem poor, where the results from Neus are blurry and low-quality.

Author response:
- 3D reconstruction with direct pixel supervision (e.g., Neus) is quite challenging since it requires exact pixel-level consistency among the generated multi-views. For our baseline Zero123, the reconstruction results of Neus are much worse. With the boosted consistency of the proposed method, the 3D reconstruction quality would be greatly improved. For a qualitative comparison with the reconstruction result of Zero123, please refer to the updated video supplementary. 
- The quality of these tasks downstream tasks is highly dependent on their implementation and hyper-parameter settings. To show more convincing reconstruction results, we apply Consistent123 to the more advanced reconstruction method Magic123, whose results are shown in the video supplementary and Figure 17.

Reviewer comment (Experiments):
For the shared self-attention layers, is there any positional embedding? If not, does introducing a camera pose-aware positional embedding help? Do you have any insight on this?

Author response:
No positional embedding is introduced in Consistent123 and it seems not quite necessary. Actually, pose-aware positional embeddings (R and T) have been introduced at cross-attention layers, and they would be used implicitly in shared self-attention and cross-view attention layers. It may be redundant to add pose embeddings again at shared self-attention and cross-view attention layers.

Reviewer comment (Theory):
Since the current conditions on R and T are still geometry-free, I am worried that the proposed method's upper bound is limited.

Author response:
Although the view consistency may be possibly limited by the geometry-free pose conditioning, our method achieves the trade-off between view quality and consistency. Geometry-aware methods may suffer from the following issues:

- The view quality may be relatively low compared with geometry-free methods (see the qualitative comparison with the recent geometry-aware method syncdreamer in Figure 12).
- It is quite time-consuming (10x more than geometry-free methods) in both the training and inference process.
- The pose and number of views are often restricted in geometry-aware methods. 

Therefore, we adopt the same geometry-free conditioning as zero123 in a purely data-driven manner.

Reviewer comment (Experiments):
Novelty is clear, while there are several publications available with open-sourced papers, such as magic123, zero123 xl, sync-dreamer, for synthesizing new views and doing the 3D reconstruction using SDS or direct pixel loss based on NeuS. Wonder if the author may compare the results with the open-sourced recon-models.

Author response:
> Can Consistent123 be compared with recent work magic123, zero123 xl, sync-dreamer?

- Magic123 is an SDS-based method for image-to-3D based on Zero123, which can be regarded as a downstream application of Consistent123. Therefore, we show the experiment results of Magic123 based on Consistent123 rather than directly comparing with it. Please refer to the video supplementary (magic123.mp4) and Figure 17 for more details.

- Both Zero123 XL and Syncdreamer are used for novel view synthesis, so we directly compare them in synthesized multi-views. Note that Syncdreamer is constrained at fixed poses, thus we only compare with it qualitatively. Please refer to supplementary Figure 12 for a qualitative comparison with these baselines.

Besides the qualitative comparison in Figure 17 and Figure 12, we also evaluate the consistency score on GSO dataset for the quantitative comparison.

|    Method     | PSNR  | SSIM | LPIPS |
| :-----------: | :---: | :--: | :---: |
|    Zero123    | 22.88 | 0.92 | 0.25  |
|  Zero123 XL   | 24.13 | 0.94 | 0.21  |
| Consistent123 | 27.98 | 0.98 | 0.11  |

Reviewer comment (Experiments):
From the experimental results after 3D reconstruction, it looks like a black biased back side is generated. In my perspective, these results are no better than the published methods such as those implemented in threestudio [url: https://github.com/threestudio-project/threestudio], which was available before the submission. The author may explain why the reconstructed results show this issue.

Author response:
> Why the reconstructed results are no better than the published methods in threestudio?

The reconstruction results of dreamfusion are highly dependent on hyper-parameter tuning. For a fair comparison, all the dreamfusion experiments shown in the paper follow the same hyper-parameter settings without further per-case tuning (and note that all these experiments are also conducted under the implementation of threestudio). As for some failure textures, we could use stronger optimization-based reconstruction methods (e.g., Magic123) to improve the quality. To further demonstrate the effectiveness of Consistent123, we show better 3D reconstruction results using Magic123 in the video supplementary (magic123.mp4) and Figure 17.

Reviewer comment (Evaluation):
The generated views are still not fully consistent before the 3D model is reconstructed, while the rendered image after reconstruction looks much worse. Is there any thought on further improving the consistency so the quality gap between diffused output and render-view can be minimized?

Author response:
> Are there any thoughts on further improving the consistency so the quality gap between diffused output and render-view can be minimized?

- First, introducing geometry-aware conditioning may help. For instance, spatial volume or epipolar attention can be used to further enhance the consistency. However, such architecture may reduce the quality and flexibility of synthesized views (e.g., the number and pose of view must be fixed at inference).
- Second, a better data filtering strategy may help. There are lot of low-quality data in our training dataset Objaverse as indicated by other concurrent methods (e.g., Instant3D). Training on the full dataset of Objaverse may result in oversmoothed textures and increase the difficulty of being consistent.

Reviewer comment (Experiments):
How does it generalize towards more sophisticated images? Please also provide some failure cases for a better understanding of the limitations.

Author response:
> How it generalizes towards more sophisticated images?

More sophisticated cases are shown in supplementary Figure 13, indicating that Consistent123 can be generalized to sophisticated images while preserving good consistency.

> Please also provide some failure cases for a better understanding of the limitations.

Failure cases are provided in supplementary Figure 14. The main problem falls on the latent diffusion decoder, which struggles to reconstruct some complicated pattern of input images (e.g., text, thin objects). A direct solution is to use a stronger latent decoder like the Dalle3 decoder (OpenAI). Using a cascaded diffusion model may be another solution.

Reviewer comment (Writing):
The name "shared self-attention" is confusing to me. It in fact is a cross-attention from the novel views to the input view. Why is it called self-attention?

Author response:
> Why is the name "shared self-attention"? It seems a cross-attention from the novel views to the input view.

The shared self-attention is similar to the cross-attention between the novel view and input view, but they have several key differences:

- It reuses the pre-trained weight of vanilla self-attention layers in Zero123. 
- It is non-trainable but only changes the calculation of self-attention. 

Therefore, we name it “shared self-attention” to distinguish these differences.

Reviewer comment (Evaluation):
Only qualitative results were presented for image-to-3D tasks.

Author response:
> Only qualitative results were presented for image-to-3D tasks.

Following the settings of Zero123, we calculate the Chamfer Distance and Volume IoU of image-to-3D reconstruction based on SDS loss. Note that these metrics only evaluate the geometry quality, ignoring the texture improvement of Consistent123 (which is more significant compared with geometry improvement). Please refer to the updated video supplementary (dreamfusion.mp4) for more qualitative comparisons. 

|    Method     | Chamfer Distance ↓ | Volume IoU ↑ |
| :-----------: | :----------------: | :----------: |
|    Zero123    |       0.0880       |    0.4719    |
| Consistent123 |     **0.0843**     |  **0.4818**  |

Reviewer comment (Experiments):
How many views were used in the NeuS experiment?

Author response:
> How many views were used in the NeuS experiment?

3D reconstruction methods like Nerf and Neus typically require a large number of views. Therefore, we use 64 views for all Neus experiments.

Reviewer comment (Presentation):
No texture on the spray bottle in Figure 7?

Author response:
> No texture on the spray bottle in Figure 7?

In the spray bottle case, we deliberately remove the texture from the object to show the synthesized geometry of Neus, which more concentrates on geometry reconstruction. We also add the textured version of this case into the supplementary Figure 15.

Reviewer comment (Evaluation):
Is the Super Mario a failure case since the object has flattened? Could it be related to the progressive CFG? Figure 3 seems to suggest that a large CFG could lead to flat objects.

Author response:
> Is the Super Mario a failure case since the object has flattened?

Super Mario is not a failure case. As a Lego toy, it is “flat” by design (feel free to google for more Lego Mario images). This case also shows that the object geometry can be constructed more reasonably with improved consistency.