Paper: Self-Evolving Neural Radiance Fields (ICLR-2024)

Reviewer comment (Evaluation):
The authors mention various methods for distilling the knowledge of nearby reliable rays, but the specific difference in performance between these methods is not sufficiently discussed.

Author response:
Thank you for your suggestion of the need for additional discussion of differences between various methods for distilling the knowledge of nearby reliable rays. Following works in self-supervised training[1][2][3][4][5], we have **done multiple experiments to decide how the threshold should be determined.** Specifically, we use fixed thresholding, adaptive thresholding, and a unified equation for the distillation. The difference in performance between these methods is shown in **Table 4. Thresholding ablation of our paper.** The results of this experiment show that the adaptive thresholding method results in the most performance gain, which is used as the default method in our framework. We also made a more specific explanation and thorough analysis for each of the methods in Section B.3 of the supplementary materials. We hope this addresses your concern.

[1] Gokhan Tur, Dilek Hakkani-Tür, and Robert E Schapire. Combining active and semi-supervised learning for spoken language understanding. Speech Communication, 45(2):171–186, 2005.

[2] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41–48, 2009.

[3] Paola Cascante-Bonilla, Fuwen Tan, Yanjun Qi, and Vicente Ordonez. Curriculum labeling: Revisiting pseudolabeling for semi-supervised learning. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 6912–6920, 2021.

[4] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, 34:18408–18419, 2021.

[5] Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj, and Marios Savvides. Softmatch: Addressing the quantity-quality trade-off in semi-supervised learning. 2023.

Reviewer comment (Experiments):
The experimental section could have benefited from additional comparisons against other few-shot learning methods or self-supervised training methods. For example, SparseNeRF [1], SimpleNeRF [2], and Vip-NeRF [3].

Author response:
Thank you for your suggestions! We would like to first point out that our work tackles the task of training NeRF with a limited number of views **where sparse SfM [1] depth points are not available**. This is due to the fact that SfM algorithms generally require sufficient images of large overlapping images to successfully reconstruct the 3D scene and estimate the camera poses. Therefore, we consider the setting where external sensors or a pre-calibrated fixed camera array are employed to obtain accurate camera poses of the limited number of views, which is used from prior works in multiple settings such as virtual or augmented reality, autonomous driving, and few-shot NeRF. In contrast, the works you have mentioned, such as SparseNeRF [2], SimpleNeRF [3], and Vip-NeRF [4] are methods relying on additional depth supervision achieved by the sparse point clouds generated from the SfM process or incorporating external prior models such as DPT [5]. However, training external models requires extra long training time with the need for sufficient data and also suffers from performance degradation when the scenes are not included in the domain of the training data. We would like to emphasize that our work is aligned with the prior works(InfoNeRF [6], DietNeRF [7], RegNeRF [8]) that solve the task of **training NeRF with sparse views without any depth information,** and the works you have mentioned are aligned with the prior works(DS-NeRF[9] that solve the task of **training NeRF with sparse views with additional depth information which will guide the network with accurate geometry information.**

However, we agree that extra comparison with such results would help us in demonstrating our models’ performance, and as ViP-NeRF provides the results of their model trained solely using the visibility prior mask and not the sparse point clouds on the LLFF dataset using three views, we have made a comparison with their results. We show that the sparse depth supervision of Vip-NeRF is crucial for their performance. 

| $\text{Method}$ | $\text{Average}$ | $\text{fern}$ | $\text {orchids}$ | $\text{horns}$ | $\text{leaves}$ | $\text{trex}$ | $\text{room}$ | $\text{fortress}$ | $\text{flower}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $\text{ViP-NeRF w/o sparse depth}$ | $17.71$ | $16.31$  | $14.37$ | $16.34$ | $15.12$ | $16.60$ | $19.21$ | $23.66$ | $18.15$ |
| $\text{ViP-NeRF with sparse depth}$ | $18.92$ | $17.49$  | $14.24$ | $18.27$ | $12.61$ | $18.16$ | $21.97$ | $24.12$ | $20.82$ |
| $\text{SE-NeRF (NeRF) w/o sparse depth}$ | $18.10$ | $19.70$  | $14.10$ | $17.92$ | $15.46$ | $15.53$ | $19.03$ | $22.86$ | $20.16$ |

[1] Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4104–4113, 2016.

[2] Guangcong Wang, Zhaoxi Chen, Chen Change Loy, and Ziwei Liu. Sparsenerf: Distilling depth ranking for few-shot novel view synthesis. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pp. 9065–9076, October 2023a.

[3] Nagabhushan Somraj, Adithyan Karanayil, and Rajiv Soundararajan. SimpleNeRF: Regularizing sparse input neural radiance fields with simpler solutions. In *SIGGRAPH Asia*, December 2023. doi: 10.1145/3610548.3618188.

[4] Nagabhushan Somraj and Rajiv Soundararajan. Vip-nerf: Visibility prior for sparse input neural radiance fields. In *ACM SIGGRAPH 2023 Conference Proceedings*, SIGGRAPH ’23, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701597. doi: 10.1145/3588432.3591539. URL https://doi.org/10.1145/3588432.3591539.

[5] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In , pp. 12179–12188, October 2021.

[6] Mijeong Kim, Seonguk Seo, and Bohyung Han. Infonerf: Ray entropy minimization for few-shot neural volume rendering. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 12912–12921, 2022.

[7] Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5885–5894, 2021.

[8] Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 5480–5490, 2022.

[9] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 12882–12891, 2022.

Reviewer comment (Evaluation):
The authors mention that the performance of the model on the NeRF Synthetic dataset is highly affected by the randomly selected views, which indicates an aspect of fragility in the model. This limitation could have been discussed in more detail.

Author response:
Thank you for your review. We appreciate your feedback and would like to address your concern regarding the fragility of the NeRF model on the NeRF Synthetic dataset. As you have pointed out, the performance of the model is highly dependent on the randomly selected views. However, we would like to clarify that the **problem is generally shared among all NeRF-like models not only ours**, especially when dealing with sparse views in complex scene settings. Thus, we have taken great care to minimize the effect of this problem in our experiments. Specifically, we randomly select views in each scene and keep them fixed while evaluating all comparison models. This ensures that our **“score improvement” is independent of the fluctuating effect caused by random view selection**. Additionally, **we introduce a novel evaluation protocol, "NeRF Synthetic Extreme,**" specifically tailored for evaluating the robustness of few-shot NeRF scenarios. In this protocol, we deliberately select the most challenging three views for each scene, strategically placing them in locations that have access to only limited information about the entire scene. This intentional selection aims to provide accurate reconstruction information for only the limited viewpoints while providing only partial information for the overall scene, creating a situation where models are prone to overfitting. Consequently, the protocol offers a fair environment for testing how well each model overcomes the overfitting challenge.

We apologize for any ambiguity in the paper, and we will upload a revised paper that clarifies the ambiguity. We hope that this explanation addresses your concern and clarifies our approach to dealing with the fragility of the NeRF model.

Reviewer comment (Experiments):
Could the authors elaborate on their choice of VGGNet for estimating the reliability of pseudo labels? How would using another architecture affect the estimation?

Author response:
We appreciate the reviewer’s interest in our work and their insightful question. In our study, **we utilized pixel-level feature maps** to evaluate the reliability of pseudo labels. As a result, **any architectural framework** that can provide **pixel-level feature maps** can be seamlessly integrated into our proposed methodology. Traditionally, 2D CNN is well-known for the expressiveness of its features, which is the reason it has been adopted as a feature extractor in various works [1][2][3][4]. Following prior works, we selected VGG-19 [5] as our default feature extractor **after conducting experiments** employing different feature extractors such as VGG-19, ResNet50 [6], and U-Net [7] in our framework. 

For VGG-19 and U-Net, we selected the extracted feature map prior to the first 4 pooling layers, whose dimensions are HxW, H/2xW/2, H/4xW/4, and H/8xW/8. For ResNet50, we selected the feature maps prior to the first 3 pooling layers, whose dimensions are H/2xW/2, H/4xW/4, and H/8xW/8. They are upsampled to HxW using bilinear interpolation and then concatenated to form latent vectors aligned to each pixel. All cases resulted in noticeable performance enhancements across the networks under consideration. For our framework, we selected the model that results in the highest performance improvement among those. The result of experiments we have done using different feature extractors is attached. 

| $\text{Methods}$ | $\text{Average}$ | $\text{chair}$ | $\text {drums}$ | $\text{ficus}$ | $\text{hotdog}$ | $\text{lego}$ | $\text{mater.}$ | $\text{ship}$ | $\text{mic}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $\text{K-Planes}$ | $15.45$ | $15.61$ | $13.23$ | $18.29$ | $12.45$ | $14.67$ | $16.30$ | $13.35$ | $19.74$ |
| $\text{SE-NeRF (K-Planes) with U-Net}$ | $17.25$$\scriptsize\color{green}(+1.80)$ | $19.81$$\scriptsize\color{green}(+4.20)$ | $13.37$$\scriptsize\color{green}(+0.14)$ | $18.33$$\scriptsize\color{green}(+0.04)$ | $20.19$$\scriptsize\color{green}(+7.74)$ | $16.29$$\scriptsize\color{green}(+1.62)$ | $16.74$$\scriptsize\color{green}(+0.44)$ | $13.47$$\scriptsize\color{green}(+0.12)$ | $19.78$$\scriptsize\color{green}(+0.04)$ |
| $\text{SE-NeRF (K-Planes) with ResNet50}$ | $17.39$$\scriptsize\color{green}(+1.94)$ | $19.96$$\scriptsize\color{green}(+4.35)$ | $13.50$$\scriptsize\color{green}(+0.27)$ | $18.42$$\scriptsize\color{green}(+0.13)$ | $20.36$$\scriptsize\color{green}(+7.91)$ | $16.28$$\scriptsize\color{green}(+1.61)$ | $16.89$$\scriptsize\color{green}(+0.59)$ | $13.96$$\scriptsize\color{green}(+0.61)$ | $19.77$$\scriptsize\color{green}(+0.03)$ |
| $\text{SE-NeRF (K-Planes) with VGG-19}$ | $17.49$$\scriptsize\color{green}(+2.04)$ | $20.54$ $\scriptsize\color{green}(+4.93)$ | $13.38$ $\scriptsize\color{green}(+0.15)$ | $18.33$ $\scriptsize\color{green}(+0.04)$ | $20.14$ $\scriptsize\color{green}(+7.69)$ | $16.65$ $\scriptsize\color{green}(+1.98)$ | $17.01$ $\scriptsize\color{green}(+0.71)$ | $13.72$ $\scriptsize\color{green}(+0.37)$ | $20.13$ $\scriptsize\color{green}(+0.39)$ |

[1] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4690–4699, 2021.

[2] Jingyang Zhang, Yao Yao, and Long Quan. Learning signed distance field for multi-view surface reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6525–6534, 2021b.

[3] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4578–4587, 2021.

[4] Minseop Kwak, Jiuhn Song, and Seungryong Kim. Geconerf: Few-shot neural radiance fields via geometric consistency. arXiv preprint arXiv:2301.10941, 2023.

[5] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. *arXiv preprint arXiv:1409.1556*, 2014.

[6] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.

[7] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image
segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi (eds.),
Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, pp. 234–241, Cham, 2015.
Springer International Publishing. ISBN 978-3-319-24574-4.

Reviewer comment (Experiments):
How does the method perform with larger or more complex datasets?

Author response:
Thank you for your interest in our work! As the current baseline models lack the capacity of modeling large scenes such as [1] and [2], we did not conduct an experiment on these datasets in our paper. However, as existing large-scale NeRFs [3,4] also utilize ray-based volume rendering, **our framework can also be easily applied to these settings.** Also, our proposed rendering view selection of first selecting rendering views near the known viewpoints will also result in the rendering views containing reliable regions and therefore **our distillation scheme can be similarly applied.**

[1] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3354–3361. IEEE, 2012.

[2] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213–3223, 2016.

[3] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8248–8258, 2022.

[4] Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua
Lin. Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering. In European
conference on computer vision, pp. 106–122. Springer, 2022.

Reviewer comment (Evaluation):
Could the authors clarify how they determine that some viewpoints are reliable and others are not? How generalizable is this approach to diverse real-world cases with more pronounced viewpoint variety?

Author response:
Thank you for sharing your concerns. We apologize for any confusion caused by our incomplete explanation of how we select rendering viewpoints within our framework. We select rendering viewpoints by perturbing the views given for the sparse inputs. Although any random perturbation to the views is possible, we find that initially selecting rendering viewpoints from regions near known viewpoints and then selecting rendering viewpoints further from the known viewpoints in a progressive manner acts most robustly, as the probability of containing reliable rays is higher in adjacent regions. By repeating this process iteratively, we increase the range of the training view to contain larger regions of the scene. **As the assumption of close regions containing more reliable regions is very natural in real-world cases**, we believe that this type of approach can be seamlessly applied to real-world images. An additional explanation can be found in **Figure 12. of Section B.4 in the supplementary materials.** We hope that this revised explanation is more clear and helpful.

Reviewer comment (Experiments):
How does the magnitude of performance improvement compare between the first iteration and subsequent iterations? Does the model reach a plateau after a certain number of iterations?

Author response:
In general, we observe the most significant performance improvement during the first to second and second to third iterations, after which the model tends to converge. However, since the magnitude of performance improvement per iteration under our framework is i**nfluenced by the underlying architecture and its own training strategy**, there is a slight difference according to the adopted baseline models. For instance, the $K$-Planes [5] baseline, which learns rapidly in relatively fewer steps than vanilla NeRF [6], accounts for 80% of the overall performance improvement within just the first two iterations.

[5] Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In CVPR, 2023.

[6] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1): 99–106, 2021.

Reviewer comment (Theory):
The method of estimating ray reliability raises concerns. The authors use a 2D-CNN to obtain super-pixel features, meaning that the features for similarity computation are derived from a pixel region, not individual pixels. This approach seems misaligned with the NeRF setting, which relies only on per-pixel information instead of per-region data. Such a discrepancy makes the ray reliability estimation seem less accurate and contradicts the proposed method.

Author response:
Thank you for your valuable feedback. We appreciate your concerns regarding the method of estimating ray reliability. We agree that the feature map our framework relies on should contain pixel-level information, not pixel-region information. We would like to clarify that we **use a pixel-level feature map**. Specifically, feature maps are extracted prior to the first 4 pooling layers, upsampled using bilinear interpolation, and concatenated to form **latent vectors of size of the original image.** With the process of upsampling feature maps extracted from different levels of the 2D CNN network, **we can achieve a pixel-level feature map.** More details can be found in Section B.2 of the supplementary materials. Please note that we qualitatively demonstrate that our feature map operates on pixel-level in Figure 7 and Figure 8. 

In addition, the process of making pixel-level feature maps by upsampling different sizes of feature maps has been leveraged in pixelNeRF[1], successfully learning the prior of 2D features and rendering novel viewpoints in the task of Generalized NeRF settings. We also show a comparison of performance using U-Net[2] as the feature extractor backbone, which is designed to extract a pixel-level feature map, and VGG-19[3], which is used as our default feature extractor.

| $\text{Methods}$ | $\text{Average}$ | $\text{chair}$ | $\text {drums}$ | $\text{ficus}$ | $\text{hotdog}$ | $\text{lego}$ | $\text{mater.}$ | $\text{ship}$ | $\text{mic}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $\text{K-Planes}$ | $15.45$ | $15.61$ | $13.23$ | $18.29$ | $12.45$ | $14.67$ | $16.30$ | $13.35$ | $19.74$ |
| $\text{SE-NeRF (K-Planes) with U-Net}$ | $17.25$$\scriptsize\color{green}(+1.80)$ | $19.81$$\scriptsize\color{green}(+4.20)$ | $13.37$$\scriptsize\color{green}(+0.14)$ | $18.33$$\scriptsize\color{green}(+0.04)$ | $20.19$$\scriptsize\color{green}(+7.74)$ | $16.29$$\scriptsize\color{green}(+1.62)$ | $16.74$$\scriptsize\color{green}(+0.44)$ | $13.47$$\scriptsize\color{green}(+0.12)$ | $19.78$$\scriptsize\color{green}(+0.04)$ |
| $\text{SE-NeRF (K-Planes) with VGG-19}$ | $17.49$$\scriptsize\color{green}(+2.04)$ | $20.54$ $\scriptsize\color{green}(+4.93)$ | $13.38$ $\scriptsize\color{green}(+0.15)$ | $18.33$ $\scriptsize\color{green}(+0.04)$ | $20.14$ $\scriptsize\color{green}(+7.69)$ | $16.65$ $\scriptsize\color{green}(+1.98)$ | $17.01$ $\scriptsize\color{green}(+0.71)$ | $13.72$ $\scriptsize\color{green}(+0.37)$ | $20.13$ $\scriptsize\color{green}(+0.39)$ |

[1] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4578–4587, 2021.

[2] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image
segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi (eds.),
Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, pp. 234–241, Cham, 2015.
Springer International Publishing. ISBN 978-3-319-24574-4.

[3] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. *arXiv preprint arXiv:1409.1556*, 2014.

Reviewer comment (Experiments):
There's a noticeable absence of an ablation study for the unreliable ray distillation. The proposed method for this distillation isn't entirely convincing. An ablation study demonstrating the effects of removing this module would have been beneficial in validating its effectiveness.

Author response:
We appreciate your insightful question and apologize for any confusion that may have arisen. To address your concern, please refer to **Table 3** in our paper, which provides the necessary details.

In this table, the method labeled as **$K$-Planes + Reliable** represents the results obtained when the **unreliable ray distillation is removed from our proposed method**. As can be observed from these results, the reliable ray distillation significantly contributes to the performance improvement of our method. Furthermore, we find that additional performance enhancements can be achieved by incorporating the unreliable ray distillation.

We acknowledge that the notation used in the table may have led to some confusion, and we will make revisions to ensure clarity. We hope this addresses your concern and demonstrates the effectiveness of the unreliable ray distillation in our proposed method. Thank you for bringing this to our attention.

Reviewer comment (Evaluation):
The performance improvement appears to be marginal. According to Table 1, the PSNR value for the NeRF Synthetic dataset and NeRF base model only saw a slight enhancement from 19.38 to 20.53. Given this quality level, the baseline model's performance is somewhat volatile, potentially fluctuating between 2-4dB with changes like altering the selected view. The paper itself even acknowledges this. A more significant achievement would have been, for example, if the authors could elevate the performance from 30dB to 32dB.

Author response:
Thank you for sharing your concerns. We would like to emphasize that training NeRF with sparse views in a complex scene setting is inherently challenging, and the scene knowledge itself is largely dependent on view selection. This is the fundamental reason why NeRF performance fluctuates in terms of view selection. We are aware of this problem and have carefully designed our evaluation protocol to minimize its effect. Specifically, for fair comparison, we randomly select views in each scene and **keep them fixed while evaluating all comparison models**.  Therefore, **our score improvement is independent of the fluctuating effect** and can be seen as a **general improvement**. Furthermore, we propose a new evaluation protocol for fair comparison called “NeRF Synthetic Extreme,” where all views are selected from one side of the scene. While it may seem that our performance improvement is somewhat modest, it's important to note that the scale of the performance value is tenfold, as a 1db increase in PSNR indicates 10 times less noise in the image.

Reviewer comment (Experiments):
The research lacks comparisons to other few-shot NeRF models. Several methods, including PixelNeRF, MSVNeRF, and IBRNet, address the few-shot neural rendering challenge. Integrating the proposed method with these established works would have provided valuable insights into its effectiveness. Unfortunately, such experiments are absent.

Author response:
Thank you for your suggestions! First, we point out that our method is a **prior-free few-shot NeRF model**, following prior works such as DietNeRF [1], InfoNeRF [2], and RegNeRF [3], which **does not utilize** any encoder networks trained across different scenes as priors for few-shot NeRF, but relies on regularization methods to successfully model a single-scene through per-scene optimization. In contrast, the works you have mentioned, such as pixelNeRF [4], IBRNet [5], MVSNet [6] are **prior-based methods** relying on the **learned generalized priors** acquired by training a feature predictor across numerous scenes for few-shot reconstruction, which introduces numerous constraints such as the requirement of large 3D training data, lengthy training time, and shows low performance when the scenes for inference are not included in the domain in of the training data. Although the comparison with prior-based methods as a pre-trained baseline in conjunction with prior-free optimization methods is possible, we would like to emphasize that these **two types of methods take an orthogonal approach** to tackle the few-shot NeRF problem.

Reviewer comment (Evaluation):
There's a discrepancy in the results presented in two different tables. For the LLFF dataset, 3-view setting, and the K-plane baseline model, outcomes are documented in both Table 1 and Table 7. While the results for SE-NeRF(K-Planes) are consistent across the two tables (e.g., 16.30dB for the PSNR metric), the baseline model's results differ: Table 1 shows 14.18dB, while Table 7 indicates 15.77dB. This inconsistency raises questions about the data's accuracy.

Author response:
We would like to express our gratitude for your thorough review of our experimental results. As we have mentioned in the paper, the performance of the NeRF trained with sparse views shows volatile results and for a fair comparison of multiple methods, we conducted multiple experiments in a random view setting and reported the average values. Although the values in both tables are correct, the 15.77dB in Table 7 is the results after being averaged, and the 14.18dB in Table 1 is one of the results of our experiment that was used to calculate the averaged results. We apologize for any confusion caused by the results. We will correct the values in Table 1 and upload the revised paper. The correct values in Table 1 for the LLFF 3-view setting are as follows:

| $\text{Methods (LLFF)}$  | $\text{PSNR}\uparrow$ | $\text {SSIM}\uparrow$ | $\text{LPIPS}\downarrow$ | $\text{Avg.}\downarrow$ |
| --- | --- | --- | --- | --- |
| $\text{K-Planes}$ | $15.77$ | $0.44$ | $0.46$ | $0.41$ |

Reviewer comment (Evaluation):
The major issues of NeRF with sparse views are two aspects: 1) few constraints for optimization; 2) wrong camera estimations. But, I don't see any designs for camera correction. Evenly the reliable regions are detected, but the cameras are still wrong. More discussions on this are needed.

Author response:
Thank you for sharing your concerns. We would like to first clarify that in the field of few-shot NeRF, there are two lines of research of training 1. NeRF with sparse views (DietNeRF[1], InfoNeRF[2], RegNeRF[3]) together **with accurate poses** and 2. NeRF with sparse views together with **noisy poses** (Sparf[4]). The current task we are tackling in this paper is the setting **where sparse images with accurate poses are given**. This can be achieved using external sensors or pre-calibrated fixed camera arrays to obtain accurate poses. We **follow prior works** (DietNeRF, InfoNeRF, RegNeRF) that have been developed under this setting and compare our works to these works in our paper. However, as you pointed out, NeRF’s application will be more practical when it becomes possible to optimize the cameras together and we believe that **our work can also be extended to these settings** by estimating the reliability with the inaccurately estimated pose information taken into account. We currently leave extending our work to pose-free or noisy poses as future work.

[1] Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5885–5894, 2021.

[2] Mijeong Kim, Seonguk Seo, and Bohyung Han. Infonerf: Ray entropy minimization for few-shot neural volume rendering. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 12912–12921, 2022.

[3] Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 5480–5490, 2022.

[4] Prune Truong, Marie-Julie Rakotosaona, Fabian Manhardt, and Federico Tombari. Sparf: Neural radiance fields from sparse and noisy poses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4190–4200, June 2023.

Reviewer comment (Evaluation):
Another major concern is about the quality: although artifacts are reduced, the visual quality are still far from satisfactory, seeing from Fig 1, 4, and 5. I am wondering why we need such a self-training strategy. To me, sparse-view NeRF is a very challenging task, resorting to prior model might be a better way to involving self-training strategy.

Author response:
Thank you for your suggestions! We agree that training NeRF in sparse views is a challenging task. As you suggested, prior works using generative priors have been proposed, such as 2D generative prior (NeRDi[1]), 2D CLIP[2] prior (DietNeRF[3]), 2D normalizing flow model (RegNeRF[4]) and show the improvement in visual quality. However, we would like to emphasize that 1. using generative priors as additional guidance and 2. solely modeling the 3D scenes with only the given limited information are **orthogonal approaches**. Using an entirely different approach, compared to RegNeRF, we **achieve much higher results** of 4db and 7db in the NeRF Synthetic Extreme and the NeRF Synthetic dataset and achieve **competitive results** on the LLFF dataset. Our work aligns with the prior-free works adopting carefully designed regularization methods such as the entropy minimization used in InfoNeRF[5].

In addition, we would like to point out that when resorting to prior models in a sparse view setting, the overall architecture needs to be carefully designed. In extreme settings such as sparse three views, these priors often act stronger than the rendering loss, guiding the network to **generate completely new objects that do not maintain the original geometry.** This can be seen in the ‘Lego’ figure generated by DietNeRF in Figure 4 of our paper, making the Lego object seem toy-like due to the prior knowledge of CLIP. Furthermore, we found that even with the incorporated priors, **existing works still suffer from overfitting to known views**, as shown in Figure 1. Our work is the first to tackle this fundamental overfitting problem by leveraging additional data rendered from the teacher NeRF and carefully designed reliability estimation methods. We show that we successfully take the **first step of resolving the overfitting problem from existing works** by showing a much gentle graph of PSNR as the views get further from known viewpoints and a **general improvement in performance** **entirely in a self-supervised manner** even when applied to different methods.

Also, we agree that the adaptation of prior models can support our framework to synthesize images of higher quality, and as the improvements shown in the paper are solely from our ‘general’ plug-and-play manner framework, we believe that the incorporation of prior models has the potential of higher quality results.

[1] Congyue Deng, Chiyu Jiang, Charles R Qi, Xinchen Yan, Yin Zhou, Leonidas Guibas, Dragomir Anguelov, et al. Nerdi: Single-view nerf synthesis with language-guided diffusion as general image priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20637–20647, 2023.

[2] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In *International conference on machine learning*, pp. 8748–8763. PMLR, 2021.

[3] Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5885–5894, 2021

[4] Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 5480–5490, 2022.

[5] Mijeong Kim, Seonguk Seo, and Bohyung Han. Infonerf: Ray entropy minimization for few-shot neural volume rendering. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 12912–12921, 2022.

Reviewer comment (Evaluation):
What is the reason behind SE-NeRF generally achieving greater improvements on vanilla NeRF compared to K-Planes (as indicated in Tab. 2)?

Author response:
Though our framework is a general one that can be applied to any NeRF model, it can not be ignored that **the distillation process is inevitably influenced by the baseline model and its own training strategy.** Our analysis of vanilla NeRF[1] achieving greater improvements is that in the case of the $K$-Planes[2] baseline, the model converges with much fewer steps than NeRF, leading to fewer steps of the distillation process from the teacher to the student which results in different degrees of improvement. However, we show that our framework can be **generally applied to different existing models and enhance the performance across different scenes.**

Reviewer comment (Experiments):
Would it be possible for the authors to provide results based on other NeRF representations, such as the more prevalent Instant-NGP [1] or Gaussian Splatting [2]?

Author response:
Thank you for your suggestions. Our framework can fundamentally be applied to **all ray-based NeRFs** that learn the scene using the volume rendering equation. To demonstrate this, **we have applied our framework to the Instant-NGP[3] you suggested.** We train SE-NeRF on the Instant-NGP for 5 minutes for each iteration on the NeRF-Synthetic Extreme setting using a single RTX 3090 GPU. Below is the quantitative comparison per scene using our framework.

| $\text{Methods}$ | $\text{chair}$ | $\text {drums}$ | $\text{ficus}$ | $\text{hotdog}$ | $\text{lego}$ | $\text{mater.}$ | $\text{ship}$ | $\text{mic}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $\text{NeRF}$ | $15.08$ | $11.98$ | $17.16$ | $13.83$ | $16.31$ | $17.31$ | $10.84$ | $16.29$ |
| $\text{K-Planes}$ | $15.61$ | $13.23$ | $18.29$ | $12.45$ | $14.67$ | $16.30$ | $13.35$ | $19.74$ |
| $\textbf{Instant-NGP}$ | $17.66$ | $12.75$ | $18.44$ | $13.67$ | $13.17$ | $16.83$ | $13.82$ | $19.05$ |
| $\text{DietNeRF}$ | $16.60$ | $8.09$ | $18.32$ | $19.00$ | $11.45$ | $16.97$ | $15.26$ | $10.01$ |
| $\text{InfoNeRF}$ | $15.38$ | $12.48$ | $18.59$ | $19.04$ | $12.27$ | $15.25$ | $7.23$ | $16.76$ |
| $\text{RegNeRF}$ | $15.92$ | $12.09$ | $14.83$ | $14.06$ | $14.86$ | $10.53$ | $11.44$ | $16.12$ |
| $\text{SE-NeRF (NeRF)}$ | $19.96$ $\scriptsize\color{green}(+4.88)$ | $14.72$ $\scriptsize\color{green}(+2.74)$ | $19.29$ $\scriptsize\color{green}(+2.13)$ | $16.06$ $\scriptsize\color{green}(+2.23)$ | $16.45$ $\scriptsize\color{green}(+0.14)$ | $17.51$ $\scriptsize\color{green}(+0.20)$ | $14.20$ $\scriptsize\color{green}(+3.36)$ | $21.09$ $\scriptsize\color{green}(+4.80)$ |
| $\text{SE-NeRF (K-Planes)}$ | $20.54$ $\scriptsize\color{green}(+4.93)$ | $13.38$ $\scriptsize\color{green}(+0.15)$ | $18.33$ $\scriptsize\color{green}(+0.04)$ | $20.14$ $\scriptsize\color{green}(+7.69)$ | $16.65$ $\scriptsize\color{green}(+1.98)$ | $17.01$ $\scriptsize\color{green}(+0.71)$ | $13.72$ $\scriptsize\color{green}(+0.37)$ | $20.13$ $\scriptsize\color{green}(+0.39)$ |
| $\textbf{SE-NeRF (Instant-NGP)}$ | $20.40$ $\scriptsize\color{green}(+2.74)$ | $13.34$ $\scriptsize\color{green}(+0.59)$ | $19.07$ $\scriptsize\color{green}(+0.63)$ | $18.15$ $\scriptsize\color{green}(+4.48)$ | $15.99$ $\scriptsize\color{green}(+2.82)$ | $17.94$ $\scriptsize\color{green}(+1.11)$ | $14.61$ $\scriptsize\color{green}(+0.79)$ | $20.23$ $\scriptsize\color{green}(+1.18)$ |

Additionally, as the Gaussian Splatting[4] you have mentioned is not a ray-based volumetric rendering approach, and thus the reliability-based ray distillation of our framework, which we tend to be crucial for the performance boost, cannot be applied. However, theoretically, if we can estimate the reliability of the Gaussian splats predicted by the model, we can distill the learned information of Gaussian splats, such as the mean and variance to the student network as additional pseudo labels. We are considering applying our framework to newly proposed methods of 3D reconstruction as future work.

[1] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1): 99–106, 2021. 

[2] Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In CVPR, 2023.

[3] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(4):1–15, 2022.

[4] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (ToG), 42(4):1–14, 2023.

Reviewer comment (Reproducibility):
While the concept of estimating unreliable rays via adjacent reliable rays is intuitive, it becomes apparent that an unreliable ray might not find a reliable ray within a local region, as illustrated in Fig. 15. How does the approach handle supervision for an unreliable ray when the neighboring reliable rays form an empty set?

Author response:
Thank you for pointing this out. For cases when the neighboring reliable rays form an empty set, we **do not include the specific unreliable ray in the distillation process**, as regularization from reliable rays is not possible. This is important from the perspective of semi-supervised training, as the incorporation of unreliable rays can **lead to the confirmation bias problem [1].** This is also shown in our experiments of using the unified equation to distill the knowledge of the teacher to the student, which is explained in Section B.3 of the supplementary materials. The unified equation can be understood as applying a soft-thresholding method to the similarity values we have estimated. In this setting, we distill the teacher's knowledge to the student scaled by the similarity value that has been filtered using a sigmoid-like function. Also, we mimic the regularization of unreliable rays by regularizing rays with adjacent rays that have higher values of similarity. However, we found that the more the performance is enhanced as the sigmoid-like function becomes more similar to a step-function (which will make the whole process identical to hard-thresholding). These findings support our design choice of ignoring unreliable rays with no adjacent reliable rays in the distillation process. Please refer to Section B.3 in the supplementary materials for a more detailed explanation.