Paper: TOSS: High-quality Text-guided Novel View Synthesis from a Single Image (ICLR-2024)

Reviewer comment (Evaluation):
I think the fundamental weakness for this paper is over-claim. In Sec.3.2.1, authors list two advantages of introducing text-prompt: plausibility and controllability, and I agree these two points. However, in the other parts of the paper, for example the Fig.1's caption, authors list *multiview-consistency* as another improvement, which I totally disagree. Fundamentally, TOSS is like the Zero123 but with more input condition, it doesn't adopt some explicit design to guarantee the multiview-consistency like in SyncDreamer[1], so naturally it shouldn't be multi-view consistent. And although authors claim this in a lot of places in paper, I didn't find explanation towards this point. Also, in Fig.13, while author titled it as "Generated Multiview-consistent images", it's actually not. For example in the provided microphone samples, I can clearly find it's not consistent across different views. So I think authors should revise the manuscript and avoid over-claim on being multivew-consistent.

Author response:
1. **Over-claim multiview-consistency**: We agree that we do not introduce explicit modules to encourage multiview-consistency like in SyncDreamer. **However, we claim that text prompts improve the multiview-consistency with semantic constraints.** We support the conclusion with both qualitative and quantitative comparisons. **For qualitative comparison**, we refer to the minion in Figure 16 as an example. As we provide a front view to be the input condition image, Zero123 generates severe multi-view inconsistent results (minion with/without backpack), even under similar camera poses. However, our method significantly improves multi-view consistency with semantic constraint from text prompt "minion without backpack". **For quantitative comparison**, we evaluate 3D consistency scores in Table 2, which shows that our method indeed improves the multiview-consistency. In comparison to SyncDreamer, our improvement on multi-view consistency is not limited by azimuth or elevation angles. We also agree that it's extremely challenging to achieve 100% consistency with single-view novel view synthesis as eventually, this problem is severely under-constrained. We will revise the manuscript to avoid over-calim on being multivew-consistent but rather with improvements on multivew-consistency. We'll also cite more works on explicit multiview-consistency improvement for readers' reference.

Reviewer comment (Presentation):
Another minor weakness is I hope authors can emphasize more on the TOSS's controllability. I would like to see the authors provide more visualization results where from the same input image, different text can generate different novel views, currently the results are mainly in supplementary and I think it's better to emphasize this ability and move the part to the main paper.

Author response:
2. **Emphasize the controllability**: Thanks for pointing this out. We now provide more results on controllability in Figure 15.

Reviewer comment (Presentation):
There is no section 3.2.2, so there is also no need for section 3.2.1. It is better to use just section 3.2.

Author response:
4. **Section 3.2.1**: Thanks a lot for pointing this out. We have removed the title of section 3.2.1 in the revision.

Reviewer comment (Experiments):
The authors claim that the text prompts increase the plausibility of novel view synthesis results. However, only the corresponding quantitative results (especially in the ablation study) can be found in the paper. Apart from the illustration in Figure 1, more qualitative results should be presented to support this claim.

Author response:
Thank you very much for the insightful questions and suggestions. We’ve provided more results according to your comments in the new revision.
1. **Plausibility results**: In Figure 14 we've provided more qualitative results emphasizing plausibility improvements via text prompts.

Reviewer comment (Experiments):
For the 3D reconstruction, the authors should also compare with relevant methods such as Magic123 and Consistent123.

Author response:
Thank you very much for the insightful questions and suggestions. We’ve provided more results according to your comments in the new revision.
3. **More 3D reconstruction baselines**: We've further included the comparison to Magic123 in Table 3. Unfortunately, the code of consistent123 is not open-sourced yet, we plan to add its result and comparisons to more methods upon availability. As Magic123 utilizes Stable Diffusion and Zero123 separately without joint training on image and prompts, the quality of its results is highly reliant on the quality of text prompts. Therefore we use the same Blip2-generated captions across all methods for fair comparison.

Reviewer comment (Experiments):
During the cross-attention process, have you tried other orders or combinations of text, image, and camera pose?

Author response:
Thank you very much for the insightful questions and suggestions. We’ve provided more results according to your comments in the new revision.
6. **Ablation on cross attention process**: We supplement results of three ablation experimets on cross attention process below: (1) reverse the order of token-level attention and text prompt attention; (2) inject the camera pose information to text prompt attention instead of to token-level attention; (3) replace the token-level attention with gated attention and train it as an adapter with other parameters freezed. We can conclude from results of (1)(2) that different order of attention modules or different camera pose injecting choices lead to decrease in performance. Besides, experiment (3) reveals that **finetuning significantly enhances model performance compared to adapter training**.


    | Method | PSNR | SSIM |
    | -------- | -------- | -------- |
    | (1) Reverse the order of  attention     | 16.63     | 0.8328     |
    | (2) Inject the camera pose with text prompts     | 16.84     | 0.8275     |
    | (3) Gated attention | 13.51 | 0.7105 |
    | TOSS | 16.95 | 0.8369 |

Reviewer comment (Evaluation):
My main concern is Whether the texts bring enough improvement. As shown in Table 4, I find "Token-level attention" improves the model the most, while the "Text prompt" is not the best. Meanwhile, the author should show these cases visually: 1. Zero123 + Token-level attention 2. Zero123 + Text prompt.

Author response:
Thank you very much for the insightful questions. We’ve provided more results according to your comments in the new revision.
1. **Contribution of text prompt**: As requested, we've provided additional visual results showing contributions of the introduced modules in Figure 13. We can see that even though "Token-level attention" indeed improves the model the most in terms of metrics, the "Text prompt" **still significantly enhances the quality of the generated novel views** visually, including more plausible and intricate details (for example the lights on the top of the police car). We would like to point out that **the contribution of text prompts exceeds what the metrics demonstrate**. The reviewer's intriguing question precisely aligns with the observation in 3DiM[1]: **The standard metrics (PSNR/SSIM) are not sufficient to effectively evaluate geometry-free models for view synthesis**. Since NVS from a single image is an ill-posed problem (e.g. novel view generation of the backview given the front view), sometimes a plausible generation  which is inconsistent with the ground truth may result in a decrease in PSNR/SSIM. \
[1]Watson, Daniel, et al. Novel View Synthesis with Diffusion Models. Oct. 2022.

Reviewer comment (Experiments):
I'm curious about the effect of this model on the human portrait since the Zero123 performance is bad in human portraits (Dataset limitations). The author can show two cases visually: 1. Taylor Swift (you can select an image of Taylor Swift as input) 2. Young male (you can randomly select a young male image).

Author response:
Thank you very much for the intriguing question. We provide additional 3D reconstruction results on Taylor Swift, young male and more in Figure 20. We acknowledge that there remains a quality gap between our method and the human-centric approaches, but **our results are of higher quality than zero123**. We agree with the reviewer that the quality concern in human portraits comes from the pretrain datasets (Objaverse), which lack sufficient human data. Therefore we believe the issue can be alleviated by finetuning on human-centric datasets which remains for future work.