Paper: ED-NeRF: Efficient Text-Guided Editing of 3D Scene With Latent Space NeRF (ICLR-2024)

Reviewer comment (Writing):
The paper seems to be casual on writing, and there might be mistakes. Like Eq. 12 and Eq.13, why are L_{tot} and L_{Mrec} part of each other? Further, there should be a formal table in the main paper on computing time comparison, rather than coarse text in the supp. The Efficiency improvements are one of the major claim and contributions of the paper, it is even added into the title of the paper ": EFFICIENT TEXT-GUIDED...", thus putting the time comparison in supp. is not good.

Author response:
> **Q1: The paper seems to be casual on writing, and there might be mistakes. Like Eq. 12 and Eq.13, why are L\_{tot} and L\_{Mrec} part of each other?**

- Sorry for the typo and thank you for your comment. It’s a mistake in Eq. 12 and Eq 13 that L{tot} and L{Mrec} are part of each other. we changed these in the revised paper.

> **Q2: Further, there should be a formal table in the main paper on computing time comparison, rather than coarse text in the supp. The Efficiency improvements are one of the major claims and contributions of the paper, it is even added into the title of the paper ": EFFICIENT TEXT-GUIDED...", thus putting the time comparison in supp. is not good**.

- Thank you for your feedback. We included the efficiency comparison part in our main quantitative results section.

Reviewer comment (Experiments):
Lacking of object removal and insertion experiments. Maybe I'm wrong, but most examples shown in the paper seem to only about modifying the centric object material, identity, or the background, it doesn't show how the methods works about removing an object, or inserting another object.

Author response:
> **Q3: Lacking of object removal and insertion experiments. Maybe I'm wrong, but most examples shown in the paper seem to only about modifying the centric object material, identity, or the background, it doesn't show how the methods works about removing an object, or inserting another object.**  **.**

- Thank you for your comment. We kindly remind the reviewer that object removal or inpainting on 3D scenes is not within our task scope, and baseline methods also did not show the results on those tasks. Instead, we show the object replacement task in our project page (e.g. castle-\> croissant). That said, the suggested direction is important, which we will investigate as a follow-up work.

Reviewer comment (Evaluation):
It seems the view consistency is not as good as several previous methods, while the small gap is acceptable, I am wondering what would be an example of view inconsistency, this can help us understand the space to improve.

Author response:
> **Q4: It seems the view consistency is not as good as several previous methods, while the small gap is acceptable, I am wondering what would be an example of view inconsistency, this can help us understand the space to improve;**

- To address the reviewer's concern, we show the Freeview rendering video on our project page [https://ed-nerf.github.io](https://ed-nerf.github.io). Our method shows view-consistent edited outputs. Although there are slight artifacts on the rendered outputs, we are improving the problems with more parameter tuning.

Reviewer comment (Theory):
In Eq. 9, I guess the order of these two items are wrong? The target SDS item should sit first, or it doesn't matter? I am also wondering what if we combine the original SDS loss in addition to the DDS loss, would it further improve the performance?

Author response:
> **Q5: In Eq. 9, I guess the order of these two items are wrong? The target SDS item should sit first, or it doesn''t matter? I am also wondering what if we combine the original SDS loss in addition to the DDS loss, would it further improve the performance?**

- Sorry for the typo. We corrected them in our revised paper. Also, per the reviewer's request, we trained the model with a combination of DDS and SDS. Please see the results in our appendix. The results show that a combination of two losses does not bring an advantage, even making blurry outputs.

Reviewer comment (Evaluation):
Reflection is an issue in NERF if not handling well, and the tensor-RF backbone doesn't model that. However, it seem that Figure 4 the car example demonstrated the vivid reflection, is this true? If this is the case, should we give the credit to the refinement layer?

Author response:
> **Q6: Reflection is an issue in NERF if not handling well, and the tensor-RF backbone doesn't model that. However, it seem that Figure 4 the car example demonstrated the vivid reflection, is this true? If this is the case, should we give the credit to the refinement layer?**

- Thank you for your interesting comment. We conjecture that expression of reflection comes from both of latent space NeRF with DDS and refinement layer. NeRF represents the color of a point through volume rendering, making it unable to capture surface reflections caused by material properties.[1] However, in the case of NeRF trained in latent space, the final output image is obtained through the decoder of Stable Diffusion from the rendered latent. This allows for the utilization of a rich representation that volume rendering alone cannot express. Additionally, when updating NeRF using DDS, considering the material properties of the target text prompt allows for the expression of rich reflections.(e.g. "gold") Moreover, by employing a refinement layer, our work effectively reduces blurry artifacts, enabling the representation of high-frequency details.
[1]: NeRFReN: Neural Radiance Fields with Reflections}

Reviewer comment (Experiments):
Although the paper showcases impressive qualitative editing results, it does not provide free-viewpoint rendering results in video format. Consequently, it is hard to determine whether the proposed method can generate view-consistent editing. This is my primary concern regarding the acceptance of this submission, and I am open to increasing my rating if the author can provide video results.

Author response:
- Thank you for your constructive comment. To address the reviewer's concern, we made an anonymous project page that includes free-view rendered videos.See [https://ed-nerf.github.io].

Reviewer comment (Evaluation):
The author presents a time analysis in the appendix; however, it is not comprehensive and does not sufficiently support the claim of "improved efficiency." The analysis only reports the time for inference, excluding training time. Given that the "Generate source image and latent feature map" paragraph in Appendix A states their approach involves generating novel view images and extracting their latent feature maps, the time required for this procedure should also be reported and factored in when comparing with baseline approaches.

Author response:
- Thank you for your constructive comments. To address the reviewer's concern, we included a comparison of full processes including pretraining NeRF backbones in the appendix and quantitative results in the main script. Also, we included a comparison of GPU memory usage to further show the efficiency of our method.
- Please See the details in the general comment and revised paper.

Reviewer comment (Theory):
The paper presents an innovative approach for editing pre-trained 3D implicit networks while preserving the surrounding environment. It would be helpful if the author could provide additional insights into why their method can edit the targeted object while preserving other parts without introducing unwanted artifacts. This would enhance the understanding of the proposed method and its underlying mechanisms.

Author response:
-  Taking the analysis from the original paper of DDS, we can consider the SDS gradient as ‘direction’ for guidance. In the paper, SDS gradient for the output z and target text y can be decomposed into two components $\nabla_\theta \mathcal{L}\_ {SDS} (z,y,\epsilon,t):=\delta\_{\text{text}}+\delta\_{\text{bias}}$. Here $\delta\_{\text{text}}$ is component which can properly guide the output to the target text direction, and $\delta\_{\text{bias}}$ is an interference term which makes the blurry and degraded outputs on unrelated regions. For the source image, we can also calculate SDS gradient $\nabla_\theta \mathcal{L}\_{SDS}(\hat{z},\hat{y},\epsilon,t)$. In this case, as our source latent and source text is aligned, $\delta\_{\text{text}}$ in this case should be zero in the optimal case. Therefore, we can assume $\nabla\_\theta \mathcal{L}\_ {SDS}(\hat{z},\hat{y},\epsilon,t) \approx \hat{\delta}\_{\text{bias}}$. As we add the same noise to both of target and source cases, we can also assume $\delta\_{\text{text}} \approx \hat{\delta}\_{\text{text}}$ which makes DDS gradient $\nabla_\theta \mathcal{L}\_ {SDS}(z,y,\epsilon,t)-\nabla\_\theta \mathcal{L}\_{SDS}(\hat{z},\hat{y},\epsilon,t) \approx \delta\_{\text{text}}$ 
which is an accurate editing direction without interference from unrelated regions. Therefore DDS gradients can edit the local regions that are related to text condition by updating pixel parameters directly.[1] However, applying the DDS to parameter space, not pixel space, can make artifacts in the unwanted area because the gradient of DDS can affect the whole parameter update process. To address this issue, we preserve DDS updates outside of the target region and regulate those areas with a reconstruction loss. By reconstructing the areas outside the target region using the original latent representation during the editing process, our method proves effective as the rendered latent itself closely follows the text condition.

      [1] Delta Denoising Score, A Hertz, K Aberman, et al., ICCV 2023  
- Upon the edit-friendly nature of DDS gradient, we can edit the 3D objects by fine-tuning NeRF models. However, we empirically found that vanilla usage of DDS still shows artifacts in unwanted regions, therefore we included proper regularization strategies for further improving the local editing performance.

Reviewer comment (Reproducibility):
The paper mentions adopting a 4-channel latent representation for the proposed method. It would be beneficial if the author could provide more details about the adopted latent feature vectors. Specifically, it would be helpful to know if this representation is strong enough for the task of editing, and how the feature maps are visualized since they appear visually similar to original RGB images.

Author response:
- Thank you for your comment. We kindly remind the reviewer that there are several advantages when using latent representations **.**
- Our latent representation has decreased dimension of 64x64x4, therefore we can reduce the GPU memory during training and also obtain faster rendering. As previous methods using models directly trained on RGB space require full-size image rendering for using them as input for Stable Diffusion, it requires large memory consumption for training. To further show the efficiency of our method, we show the comparison on time and memory aspect in the main script.
- Please See the details in the general comment and revised paper.
- To show whether the latent space actually contains the context of corresponding RGB images, we show the latent visualization using the linear decoding used in Latent-NeRF in our appendix. We can see the latent features and images have similar contexts and common spatial information.

Reviewer comment (Evaluation):
The rendered depth map presented in the paper appears somewhat coarse. It would be useful if the author could explain this observation.

Author response:
- We kindly remind the reviewer that our Latent space NeRF is trained on dimension-reduced latent space Stable Diffusion. Therefore, our depth map extracted from the NeRF model also has a smaller depth map size of 64x64. Although the depth map resolution is reduced, we can see that the map contains geometry and distance information of original images, and the rendered output quality is still high. This indicates that our latent space NeRF has enough capacity to contain real-scene representations.

Reviewer comment (Novelty):
The key contribution of the proposed method lies in the integration of several small techniques, e.g., latent NeRF representation and DDS distillation loss, without too much novel insight. I am not pretty sure whether it is enough for the ICLR.

Author response:
>  **Q1 : The key contribution lies in the integration of several small techniques, e.g., latent NeRF representation and DDS distillation loss, without too much novel insight. I am not pretty sure whether it is enough for the ICLR.**
- We kindly remind the reviewer that although previous work of Latent-NeRF [1] first proposed to train NeRF in the latent space of Latent Diffusion Models, the task of Latent-NeRF is only targeting to generate **'virtual'** 3D object from given text conditions, not to **'real'** 3D data. In contrast, our approach pioneers training Latent space NeRF for real 3D scenes by embedding real-scene RGB images as latent representations, improving reconstruction performance with a novel refinement layer. Training NeRF in latent space offers advantages such as reduced training burden and enhanced editability through direct use of rendered output in LDM models.
- Therefore, we believe our work's contribution is significant as it achieves technical breakthroughs by showcasing the potential for expanding existing NeRF research, which predominantly concentrates on training with natural RGB.
- In addition, our method is the pioneer in extending the DDS framework to the 3D editing space of NeRF, which was previously exclusive to 2D images. Unlike previous text-guided editing NeRF methods primarily relying on default SDS or CLIP models, our approach recognizes the unique editing capability of DDS in 3D NeRF editing. This novelty is evident in its significantly superior performance compared to using SDS. Our successful application of DDS to 3D NeRF editing marks a groundbreaking contribution.

     [1] Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures, Metzer, Gal, et al., CVPR 2023

Reviewer comment (Experiments):
One of the motivations of the method is the efficiency of NeRF editing. To this end, the latent NeRF representation is adopted, while the efficiency comparison with the existing methods is not shown in the main paper. Moreover, there is a lot of work for fast NeRF training/finetuning, which makes the improvement less interesting.

Author response:
> **Q2 : One of the motivations of the method is the efficiency of NeRF editing. To this end, the latent NeRF representation is adopted, while the efficiency comparison with the existing methods is not shown in the main paper. Moreover, there is a lot of work for fast NeRF training/finetuning, which makes the improvement less interesting.**
- As reviewer commented, there are several off-the-shelf NeRF backbone models which enable faster training. However, editing a NeRF model with Diffusion model demands rendering high-resolution images (e.g., 512x512) and computing gradients with respect to it, leading to significant GPU memory consumption. In contrast, our model is trained on smaller-dimension data (e.g., 64x64), reducing memory requirements for fine-tuning the NeRF. By addressing the fundamental issue of training efficiency through data dimension reduction, we achieve more resource-efficient training compared to using NeRF models in image space. Efficiency comparisons are detailed in the general reply and revised paper.

Reviewer comment (Novelty):
The design of Refinement layers is straightforward to me, and it is more like adding some additional layers after the volume rendering, which is a normal way to improve the rendering quality. Moreover, from Fig. 2, it looks like some layers are adopted directly from the SD VAE. What's the benefit of such a design?

Author response:
> **Q3 : The design of Refinement layers is straightforward to me, which is a normal way to improve the rendering quality. Moreover, from Fig. 2, it looks like some layers are adopted directly from the SD VAE. What's the benefit of such a design?**

- We kindly remind the reviewer that we used a refinement layer as a bridge between the latent space of stable diffusion (SD) and neural radiance fields (NeRF) for the first time, making it different from conventional rendering improvement methods. From this perspective, we adopted the structure of an SD VAE with attention, which allows encoding that considers pixel correlation in the latent space. Additional experiments on various refinement layer structures, presented in the revised manuscript's appendix, validate that our proposed network structure exhibits the best reconstruction performance.

Reviewer comment (Experiments):
In the experiment, most of the examples are about color changes or local texture editing. I am wondering whether the proposed method works for other 2D editing, such as the whole image style transfer or object replacement as shown in Instruct-NeRF2NeRF. Moreover, from Fig.4, we can see that compared to Instruct-NeRF2NeRF, the improvement lies in the accuracy of the editing for certain areas. I am wondering whether Instruct-NeRF2NeRF will perform better just with the mask constraint.

Author response:
> **Q4 : In the experiment, most of the examples are about color changes or local texture editing. I am wondering whether the proposed method works for other 2D editing, such as the whole image style transfer or object replacement as shown in Instruct-NeRF2NeRF. Moreover, from Fig.4, we can see that compared to Instruct-NeRF2NeRF, the improvement lies in the accuracy of the editing for certain areas. I am wondering whether Instruct-NeRF2NeRF will perform better just with the mask constraint.**

- We conducted additional experiments on style transfer and object replacement. We also show the object replacement results on our project page (e.g. castle -\> croissant)(https://ed-nerf.github.io). 
We show the style transfer results in our revised Appendix.
- To address the reviewer's concern, we added a comparison with masked Instruct-N2N in the revised Appendix. Applying regional masks to In2n enhances local editability without altering undesired regions, but the output quality often falls short of meeting text conditions. We conjecture that Instruct N2N performance is significantly influenced by Instruct Pix2Pix's editing capabilities. Since In2n utilizes the edited 2D outputs for training, the translation performance of Instruct Pix2Pix directly impacts output quality. Improper changes to image sets in InP2P can degrade the output quality.