Paper: IPR-NeRF: Ownership Verification Meets Neural Radiance Field (ICLR-2024)

Reviewer comment (Novelty):
W1. Novelty and effectiveness. The Black-box watermark extraction method is a straightforward way to apply diffusion models. However, it relies on a catastrophic performance on unseen samples, only empirically validated on samplings (Sec 4.3). The below question Q2 raises an issue of false positive detection.

Author response:
We are delighted to have addressed your concerns and would like to express our gratitude for your recognition of our revised version. Your thoughtful and constructive feedback has been integral to making these improvements possible.

We have **updated** the Section 3.1 (black-box) and Section 4.3 (False positive detection). Specifically, we’ve mentioned in the updated Section 3.1: 
- This work was inspired by (Ho et al., 2020) to propose a diffusion-based method to learn the detector *d*. This is because the nature of diffusion models to gradually add noise to the image provides natural robustness against various forms of image degradation attacks as the main objective of diffusion models is denoising. Thanks to the powerful denoising ability of diffusion models, our idea is to first convert the views *x* into Gaussian noises, then we denoise from these Gaussian noises into the watermark *w*. As a result, we can seamlessly translate between *x* and *w* through *d*. 

Then in Section 3.2 (white-box), we added 
- To achieve this, normalization layers were appended to the MLP implementation. 

Also, we have updated the **Section 4.3**, as well as **Appendix A.5** with experiment details: 
- Both the protected and unprotected NeRF models in this experiment employed scenes from the NeRF-Synthetic 360 dataset, including Lego,  Chair, and Drum, as well as scenes from the LLFF-Forward dataset, namely Fern, Fortress, Room, 
and Flower.  To further understand why the false positives did not happen, we compute the histogram of 100 different views from each scene of protected/unprotected and 100 random images. 
- A new confusion matrix (Figure 14) and Figure 15 (Qualitative result of the various recovered watermarks from the rendered image of the protected model, rendered image of the unprotected model and random images) are added to further clarify the False Positive Detection Prevention.

In the appendix, new experiment using **Unbound-360 Dataset** is also included in **Appendix A.10** to clarify the impact of dataset selection on ownership verification. 
- We conducted an experiment using the 'Garden' and 'Kitchen' scenes from the Unbound-360 dataset. We found that the quality of the recovered watermark still maintained a high standard with an SSIM of 0.95 overall (Kitchen: watermark SSIM 0.9625; Garden: watermark SSIM 0.9526). 

Finally, the font in all figures has been adjusted to ensure better clarity and comprehension.

Reviewer comment (Reproducibility):
W2. Clarity on the white-box method. The author did not clearly state where are the normalization layers in the NeRF model. As raised in question Q1, the readers cannot pinpoint the normalization layers in the original NeRF model, so reproducibility is unclear.

Author response:
(a)Clarification on Normalization Layers:
- We sincerely apologize for the oversight. Yes, it is correct that the original NeRF model [1] did not have normalization layers. In our work, we modified the NeRF model by appending normalization layers to all Multi-Layer Perceptron (MLP) layers. This modification aimed to embed a digital signature, optimized through sign loss objectives, thereby enhancing Intellectual Property Rights (IPR) protection in a white-box scenario. 
- Our experiments (Table 2) demonstrated that the introduction of these normalization layers did not adversely impact the rendering performance. We will include this description in the revised version of our manuscript to ensure reproducibility.

[1] https://arxiv.org/pdf/2003.08934.pdf

Reviewer comment (Evaluation):
Q2. In Fig 5, a substantial portion of the protected model (red) and random images (blue) are overlapping. How do you say that they are significantly different?

Author response:
(b)Response to Figure 5 Distribution:
- Although a substantial portion of the distribution of the protected model and the random images overlaps, there is still a difference of 0.2979 and 2.77 in the computed average maximum mean discrepancy and average Wasserstein distance across all scenes. The details of each scene are as follows:

|  NeRF-Synthetic| Lego   | Chair  | Drum   | Average    |
|-----------------------|--------|--------|--------|------------|
| Maximum mean discrepancy| 0.2876         | 0.3427 | 0.3343 | 0.3215 |
| Wasserstein distance   | 2.3769         | 2.9267 | 2.8434 | 2.7157 |

| LLFF-Forward   | Fern   | Fortress | Room   | Flower | Average|
|-----------------------|--------|----------|--------|--------|--------|
| Maximum mean discrepancy| 0.2714         | 0.3372 | 0.3136   | 0.1984 | 0.2802 |0.2802|
| Wasserstein distance   | 2.2140         | 2.8717 | 2.6364   | 3.5205 | 2.8107 |2.8107|


- The trained diffusion-based watermark detector can only extract a watermark from the overfitted distribution of the protected model. As a result, it is unable to extract the watermark from the distribution of random images and the unprotected model in which the extracted image is essentially a noise image similar to watermark image extracted by 2D steganography method (see Figure 1 of the main paper). 
- This is because the trained diffusion-based watermark detector was exclusively and strongly overfitted to the distribution of the protected model, effectively preventing false positive detections. We apologize for any confusion caused by our writing and will address this issue by refining our manuscript in the revised version.

Reviewer comment (Writing):
In Sec 3, Para 2, by initiate -> by initiating

Author response:
(c)Apology for Grammatical Mistakes:
- Turning to the matter of grammatical errors, we extend our sincere apologies for any oversights. We have revised our manuscript, focusing specifically on rectifying grammatical mistakes, including those highlighted by your insightful feedback.

Reviewer comment (Writing):
Appendix A.5, to qualitatively -> to quantitatively

Author response:
(c)Apology for Grammatical Mistakes:
- Turning to the matter of grammatical errors, we extend our sincere apologies for any oversights. We have revised our manuscript, focusing specifically on rectifying grammatical mistakes, including those highlighted by your insightful feedback.

Reviewer comment (Novelty):
My major concern is that It lacks the original technical contribution. It is interesting to see that the diffusion models (DDIM) are quite effective in this task. However, I think it may not be a sufficient contribution to be a full conference paper.

Author response:
Dear Reviewer,

Thank you for your thoughtful comments and concerns regarding the technical contribution of our work. We appreciate the opportunity to clarify the key points you raised.

(a) Technical Contribution to Black-box Protection:     
- We propose diffusion models as the foundation for a watermark detector, enabling the extraction of embedded watermarks within the rendered images of NeRF models. Our methodology has demonstrated noteworthy resilience against noise, a point substantiated in our research. More importantly, our work stands as a pioneering effort in uncovering IPR protection strategies for NeRF models, a pursuit of paramount significance for the 3D vision community. The escalating commercial use of NeRF, particularly in applications like the metaverse, underscores the urgency for a robust NeRF IPR protection framework.

- While our black-box protection methodology hinges on diffusion models, it's crucial to highlight the non-trivial nature of jointly optimizing watermark injection into the rendered scenes of NeRF (Eq 8). This optimization task becomes even more intricate when striving for a high watermark recovery rate. Consequently, the primary contribution of our work lies in pioneering the application of diffusion models for black-box NeRF protection. We demonstrate the feasibility of the joint optimization process (Eq 8) for embedding watermarks into rendered scenes and extracting embedded watermarks using diffusion models (Eq 4 and Eq 5), a valuable insight for the 3D community. In essence, our work contributes to the evolving landscape of NeRF IPR protection. Also, it serves as a testament to the viability and applicability of diffusion models in the joint optimization process. We believe these advancements are crucial steps toward a more secure and resilient NeRF IPR protection framework.

Reviewer comment (Experiments):
I wonder if the DDIM method can generate high-fidelity images. The comparison to the previous method, e.g., staganerf would be great.

Author response:
(c) Fidelity Analysis:     
- Thanks! We appreciate the suggestion. And Yes! Our proposed method does not affect the fidelity of the rendered images. We have included a comprehensive fidelity comparison between our proposed method and the existing methods, namely StegaNeRF[1] and CopyRNeRF [2] in our revised manuscript, as well as below.

|                     | NeRF   | IPR-NeRF(Ours) | StegaNeRF | CopyRNeRF |
|------------------|--------|----------------|-----------|-----------|
|**NeRF Synthetic**|
| **PSNR** | 31.21  |31.05  |31.14| 30.93|
| **SSIM** | 0.9593 |0.9571  |0.9585    | 0.9531    |
| **LPIPS**| 0.0528 |0.0573 | 0.0537    | 0.0584    |
|**LLFF-Forward**|
| **PSNR**    | 27.59  |27.39 | 27.42     | 27.21     |
| **SSIM**    | 0.8293 |0.8237|0.8277    | 0.8224    |
| **LPIPS**   | 0.1685 |0.1705|0.1693    | 0.1718    |

We hope these clarifications demonstrate the significance of our contributions and address your concerns. We appreciate your valuable feedback.

Reviewer comment (Experiments):
Scalability to explicit NeRF representations: To accelerate NeRF inference and rendering, multiple works [1,2,3] have proposed to use explicit representations (e.g., grid, mesh, and point cloud) instead of MLP as the NeRF representations. Modifying the weights of explicit NeRF representations seems to have a larger effect on the rendering quality as compared to implicit representations because it can be regarded as changing the location/color of the grid/mesh/point cloud. Thus, it is not sure whether the proposed protection framework can still maintain the rendering quality of those explicit NeRF representations.

Author response:
(a)Scalability to explicit NeRF representations: 
- Although our proposed method was designed for implicit NeRF representations, it can be *directly* applied to explicit NeRF representations. This is because our black-box diffusion-based detector does not depend on the architecture of NeRF models but rather on the rendered images. Additionally, our white-box protection scheme only adds/modifies the normalization layer into the MLP layers. 
We will conduct a further investigation on directly applying our proposed methods to explicit NeRF representations in our future work.

Reviewer comment (Experiments):
Insufficient details in "False Positive Detection Prevention": In Sec. A.5, the dataset used in this experiment is not mentioned. Does Fig. 5 show the averaged histogram for different scenes in a specific dataset or the histogram for a specific scene? The unprotected standard NeRF is claimed to be trained in the same dataset as a protected one. Is it trained on the same scene or a different scene but in the same dataset?

Author response:
Also, we have updated the **Section 4.3**, as well as **Appendix A.5** with experiment details: 
- Both the protected and unprotected NeRF models in this experiment employed scenes from the NeRF-Synthetic 360 dataset, including Lego,  Chair, and Drum, as well as scenes from the LLFF-Forward dataset, namely Fern, Fortress, Room, 
and Flower.  To further understand why the false positives did not happen, we compute the histogram of 100 different views from each scene of protected/unprotected and 100 random images. 
- A new confusion matrix (Figure 14) and Figure 15 (Qualitative result of the various recovered watermarks from the rendered image of the protected model, rendered image of the unprotected model and random images) are added to further clarify the False Positive Detection Prevention.

Reviewer comment (Experiments):
Will the dataset affect the selection of ownership verification? A threshold of 0.75 is set for the black-box ownership verification, and the author claimed that 0.75 is selected "as the visibility is still evident." However, it is uncertain whether different datasets have different optimal thresholds. For example, scenes in Unbound-360 [4] or ARKitScene [5] is more close to the real-world applications [6,7]. Will the optimal threshold still be 0.75 on those datasets?

Author response:
(c)Clarification on the Impact of Dataset Selection on Ownership Verification:
- For clarification, different datasets  would not require to have different optimal thresholds. We are sorry for the misunderstanding. This section primarily centers around the crucial quality of extracted watermarks for asserting ownership, for which we chosen the threshold = 0.75 (SSIM). This choice is made empirically, aligning with Figure 13, where the watermark is still distinctly discernible for ownership claim. Additionally, our experimental findings (refer to Table 2) consistently demonstrate that the quality of the extracted watermark consistently surpasses 0.95 (SSIM) across diverse scenes within the NeRF-Synthetic and LLFF-Forward datasets.
- Additionally, we conducted an experiment using the 'Garden' and 'Kitchen' scenes from the Unbound-360 dataset [1], as suggested. We found that the quality of the recovered watermark still maintained a high standard with an SSIM of 0.95 overall (Kitchen: watermark SSIM 0.9625; Garden: watermark SSIM 0.9526). We have included these additional experimental results in our revised manuscript for better clarification.

[1] https://jonbarron.info/mipnerf360/

Reviewer comment (Theory):
The motivation for using the diffusion model to learn black-box protection is unclear. It would be great if the authors could provide more elaboration.

Author response:
(a) Elaboration on Applying the Diffusion Model:                     
- Our initial findings using simple solutions such as DeepStega [1] and HiDDen [2] as well as more advanced methods such as StegaNeRF [3] demonstrated a few issues: (i) Ineffective watermark extraction on the rendered scene [1,2] due to the embedded watermark being smooth out during rendering (please refer to Figure 1 in the main paper); and (ii) The extraction of watermarks from the rendered scene is not robust to the introduction of noise [3], as it was trained under noise-free conditions (please refer to Figure 6 and Figure 7 in the main paper)

- As a result, we propose to use diffusion model as the foundation for our black-box approach. First, the nature of diffusion models to gradually add noise to the image provides natural robustness against various forms of image degradation attacks as the main objective of diffusion models is denoising (please refer to Figure 6 and Figure 7 in main paper). This resilience is a result of the inherent characteristics of the diffusion model. Second, the process of watermark embedding has minimal impact on the rendering quality of the NeRF models (please refer to Table 2 in main paper). Third, it can retrieve embedded watermarks within the rendered images of NeRF models (please refer to Table 2 in main paper). 

- We sincerely apologize for the lack of an explanation of our motivation for applying the diffusion model as the black-box protection, we will add the above description to our revised manuscript.

Reviewer comment (Experiments):
Is there any simple alternative solution for black-box protection? If so, could the author provide some comparisons?

Author response:
(b) Simple Alternative Solution for Black-Box Protection of NeRF:                   
- Yes, some examples of simple alternative solutions are DeepStega [1] and HiDDen [2], which involve directly watermarking the training images before model training. However, as illustrated in Figure 1 of the main paper, the embedded watermark could not be extracted as it was smoothed out during training and rendering. Hence, these simple alternative methods are deemed not suitable for black-box protection of NeRF models.

[1] https://proceedings.neurips.cc/paper_files/paper/2017/file/838e8afb1ca34354ac209f53d90c3a43-Paper.pdf                      
[2] https://dl.acm.org/doi/10.1007/978-3-030-01267-0_40      
[3] https://arxiv.org/abs/2212.01602

Reviewer comment (Evaluation):
Just out of curiosity, is there any reason that StageNeRF is significantly vulnerable to Gaussian noise?

Author response:
(c) Vulnerability of StegaNeRF to Gaussian Noise:                
- The vulnerability of StegaNeRF [3] to Gaussian noise is due to the ideal conditions of the watermark detector's training procedure, which does not consider ``noise'' as a form of attack. Therefore, the watermark detector in StegaNeRF is implemented as a simple autoencoder-based detector. Consequently, when noise is added to the rendered scene as a form of removal attack, the features of the rendered image are compromised, causing the embedded watermark unable to be extracted effectively.

Reviewer comment (Presentation):
Enlarging the font in the figures could be helpful.

Author response:
(d) Addressing Font Clarity in Figures:                 
- We sincerely apologize for the oversight regarding the unclear font in the figures, including legends and axis labels. To enhance the reader's viewing experience, we will enlarge the font in these elements to ensure better clarity and comprehension in the revised manuscript.