Paper: Deceptive-NeRF: Enhancing NeRF Reconstruction using Pseudo-Observations from Diffusion Models (ICLR-2024)

Reviewer comment (Evaluation):
It would be valuable to include a comparative analysis between the generated pseudo observations and the ground-truth images. This could provide insights into the fidelity of the pseudo observations and their accuracy in replicating the real data.

Author response:
**Quality of Pseudo-Observations. (Question 2)** Thank you for your thoughtful suggestion to include a comparative analysis between our generated pseudo-observations and the ground-truth images. We have been working on it and will provide these visual comparisons soon to further strengthen our arguments.

**Quality of Pseudo-Observations.**  We have included a comparative analysis between our generated pseudo-observations and the ground-truth images in Section D of our revised manuscript. Additionally, as suggested by Reviewer QhhZ, we provide a comparison with image restoration models, along with quantitative results. We hope they will aid in understanding the behavior of the deceptive diffusion model.

Reviewer comment (Experiments):
In Table 1, a more robust baseline for scene reconstruction might be considered, such as the monoSDF method that utilizes monocular depth and normal maps as additional sources of supervision. Comparing the proposed method's performance to such a strong baseline would provide a clearer picture of its relative merits and limitations.

Author response:
**Stronger Baselines (Question 4).**  Thanks for pointing out our missing baselines and we'll report their results soon.

**MonoSDF.** Thank you for suggesting MonoSDF [1] as an additional baseline. We experimented with MonoSDF integrated into SDFStudio [2], but in our few-shot setting (5-20 input views), MonoSDF failed to provide reasonable reconstruction. This is likely due to the fact that MonoSDF, as a neural implicit surface reconstruction approach, prioritizes accurate reconstruction of the object surface and therefore requires dense observations to achieve reasonable results.
The comparison in Table 1 covers most of the state-of-the-art methods in the field of few-shot novel view synthesis at the time of submission. While approaches for the task of neural implicit surface reconstruction [1,3,4] are highly relevant to our task, they have a different focus. Specifically, these methods prioritize the reconstruction of object surfaces, while our focus is on photo-realistic novel view rendering directly without recovering the surface. Additionally, neural implicit surface reconstruction often requires additional inputs such as normals, depths, or masks, in addition to RGB inputs, to ensure the accuracy of the surface reconstruction, while our method does not.
We have revised the paper to include a discussion on MonoSDF in Section 4.1.


**References**    
[1] Yu et al., “MonoSDF Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction”. (NeurIPS 2022)     
[2] Yu et al., “SDFStudio: A Unified Framework for Surface Reconstruction”. (https://github.com/autonomousvision/sdfstudio)    
[3] Wang et al., “NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction”. (NeurIPS 2021)    
[4] Oechsle et al., “UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction”. (ICCV 2021)

Reviewer comment (Experiments):
One biggest concern of this idea is that some diffusion-synthesized samples are view-inconsistent. This method proposed a 50% filtering strategy to alleviate this issue, but I don't know whether this issue can be fully bypassed. Introducing confidence score as in NeRF-W may help.

Author response:
**Filtering Strategy (Weakness 1).** Based on our experience, our filtering strategy, together with our other designs, has to some extent alleviated the occurrence of view inconsistency. We appreciate the reviewer's suggestion to adopt the confidence score as in NeRF-W and will implement it as soon as possible and report the results.

Reviewer comment (Experiments):
Computational cost. Training a diffusion model for 10 days to further finetune a NeRF sounds inefficient to me. Also, can this finetuned CLDM be applied to any in-the-wild NeRF reconstruction dataset?

Author response:
**Computational Cost (Weakness 2).** Please refer to our discussion in the general response. Please kindly note that the 10-day fine-tuning time was achieved using personal computer-level computing resources (a single NVIDIA GeForce RTX 3090 Ti GPU) and can be greatly improved.

Reviewer comment (Experiments):
Beside the concerns stated above, I wonder whether you can apply your method to other popular nerf benchmars, such as DTU and LLFF?

Author response:
**Other benchmarks (Weakness 2 and Question).** Please refer to our discussion in the general response. We have included the experimental results on LLFF in the appendix, and our method outperforms most of the baselines. We will report our method's results on other indoor scene datasets to demonstrate its generalization ability.

Reviewer comment (Writing):
Some of the writing requires improvements, for example in related work the first paragraph ends with the statement that NeRFs require numerous images which is followed by an entire paragraph falsifying that very statement presenting recent advances on few-sample learning with NeRFs. There are also some language issues.

Author response:
**Writing issue (Weakness 1).** Thanks for pointing that out. We will improve writing to ensure logical flow and minimize grammatical and vocabulary errors.

Reviewer comment (Evaluation):
The evaluation seems insufficient. Given that the diffusion model has been trained on a denoising task to reduce computational burden, it would seem meaningful to evaluate this concept on some existing image restoration networks in comparison to the proposed fine-tuning of a diffusion model.

Author response:
**Image Restoration Networks.** Based on your suggestions, we compared the pseudo-observations synthesized by our deceptive diffusion model with those obtained using image restoration networks [1], as detailed in Section E. Our findings indicate that image restoration models typically struggle to convert coarse NeRF images into reasonable pseudo-observations. This issue arises primarily because they are designed and trained for specific image restoration tasks. In contrast, our deceptive diffusion model, which is specifically tailored for generating pseudo-observations, effectively leverages depth and texture cues to produce more photo-realistic results.

Reviewer comment (Reproducibility):
How is the depth map obtained from the nerf?

Author response:
**Depth from NeRFs (Question 2).** We compute the depth estimation of a pixel as
$
   \hat{z}(\mathbf{r}) = \sum_{k=1}^{K} w_k t_k,
$
   where $t_k$ are the sampling locations, and the rendering weights
$
   w_k = T_k (1 - \exp(-\sigma_k \delta_k)).
$
 $\sigma_k$ is the volume density value inferred from the NeRF MLP. $\delta_k = t_{k+1} - t_k$ is the distance between adjacent sampled points, and 
$
   T_k = \exp\left(-\sum_{i=1}^{k-1} \sigma_i \delta_i\right).
$

Reviewer comment (Theory):
It is surprising to me that the optimization scheme does not result in inconsistency issues. Could the authors provide some intuition for why there is no issue?

Author response:
**View consistency (Question 3).** In our pipeline, the diffusion model does not generate a new view image from scratch, which can easily lead to inconsistent views (as seen in many 3D diffusion generation efforts). Instead, it takes as input an image rendered from a coarse NeRF and refines it to remove artifacts. The coarse image processed by the diffusion model is consistent with other views in nature and the view consistency can still be maintained after diffusion’s refinement. Moreover, our progressive strategy involves gradually moving away from the input view to synthesize a new view. As a result, the artifacts encountered by the diffusion model at any given time are limited, reducing the difficulty of the refinement task it faces.

Reviewer comment (Writing):
What is meant by "We optimize a shared latent text embedding s [...]"?

Author response:
**Shared text embedding (Question 4).** As described in [2], we optimize for a new embedding vector within the embedding space of a frozen text-to-image model. This vector represents elements that appear in each input image but cannot be directly described using natural language, such as the stylistic information of a room. This embedding can be treated like any other word and is utilized to compose novel textual queries for our generative models.