Paper: Geometry-Aware Projective Mapping for Unbounded Neural Radiance Fields (ICLR-2024)

Reviewer comment (Experiments):
I suspect most of the performance comes from adaptive sampling rather than the novel mapping function. The sampling should be compared against other sampling methods suggested in conjunction with NeRF (e.g., depth adaptive sampling, coarse-to-fine approach, etc.). Here are some suggestions:

@article{fang2021neusample,  
 title={NeuSample: Neural Sample Field for Efficient View Synthesis},  
 author={Jiemin Fang and Lingxi Xie and Xinggang Wang and Xiaopeng Zhang and Wenyu Liu and Qi Tian},  
 journal={arXiv:2111.15552},  
 year={2021}  
}  

@InProceedings{Hu_2022_CVPR,  
 author    = {Hu, Tao and Liu, Shu and Chen, Yilun and Shen, Tiancheng and Jia, Jiaya},  
 title     = {EfficientNeRF  Efficient Neural Radiance Fields},  
 booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},  
 month     = {June},  
 year      = {2022},  
 pages     = {12902-12911}  
}

Author response:
**[W2]** As you know, general neural rendering frameworks commonly consist of a sampling and composite function. For unbounded scenes, we additionally impose a mapping function and its ray parameterization. That’s, we would like to highlight that the sampling is out-of-scope from our work. The sampling is a strategy that selects a value within an interval of the ray parameter. In this work, we are now exploiting the coarse-to-fine sampling in our baseline models, including iNGP and NeRF, after our ray parameterization.
Nevertheless, we carry out an additional experiment with the two models that you introduce on the mip-NeRF 360 bicycle scene to check if the suggested sampling methods can handle the unbounded scene. The results show that they have difficulty in handling the unbounded scene without the mapping function, whose result is below:


Table 1. [PSNR of adaptive sampling methods] 
|**Model**   |  **PSNR** |
|-------------|:---------:|
| NeuSample     |   19.21   |
| EfficientNeRF |   18.45   |
| **ours**      | **23.36** |


If you think that this experiment result is helpful for readers to understand our work, we will add it in the Appendix at the next round.

Reviewer comment (Presentation):
The paper dedicates too much to explaining the mapping. While the analysis can be interesting, the two main contributions start on page 6. I think it would be better to position it around page 3. I would also recommend putting more qualitative results in the main paper and more comprehensive results in the appendix.

Author response:
**[W3-Contents relocation]** Thank you for your suggestion. Unfortunately, to move the contents in page 6 into page 3, we have to explain the concept of Stereographic project first. That’s, we need to position Sec. 4.1 and 4.2 in Sec. 3, which is identical to the current version. If you have any solution to this issue, please let us know at the next round. We are always welcome to your helpful suggestions.

**[W2 & W3]** Thank you for understanding the practical implementation issue and respecting the order of our contents.

Reviewer comment (Theory):
What is the computational complexity in evaluating p?

Author response:
**[Q1-Computational complexity on p]** Estimation of p value via RANSAC takes about 240 seconds. For this, we have added it in Sec.E of Appendix.

Reviewer comment (Writing):
Although the paper dedicates a significant amount of space to discussing the mapping, it is not very clearly explained. Especially in Section 1, it is very obscure to understand the explanation associated with Figure 1. For example, in the figures, it is not obvious what the x and y axes are and how they relate to actual camera parameters. It is never clearly defined. I suggest condoning the explanation (Figures 1, 2, and 3 seem somewhat repeated) and clearly defining the depicted space. Also, please connect consistently with the actual projection function and the figures. I suppose $\zeta$ and $\mathbf{x}_m$ are the same; and $\xi$ is $\mathbf{x}_b$; and $\psi$ is $\mathbf{x}$. Please introduce them together with the corresponding equations. What do you mean by 'hyper-axis,' and how does it relate to homogeneous coordinates? Also, what are the x and y axes in Figure 9?

Author response:
**[Q2]** The x-axis means a sliced coordinate axis of 3D space and the y-axis indicates a hyper-axis which is an added dimension to the 3D space. And, we appreciate your suggestion and have unified the notations as well as fixed the figures. 

The homogeneous coordinate is a coordinate system that uses points projected onto a plane. It adds an extra dimension to describe the projection. In the same way, we introduce the hyper-axis to describe a modified stereographic projection. Based on your comment, we bridge up the gap between the homogeneous coordinate and the hyper-axis in the caption of Fig. 1.

Please check the rebuttal revision.

Reviewer comment (Reproducibility):
How do you decide the scene origin and choose 'r' in contract mapping?

Author response:
**[Q3]** We follow the original implementation from mip-NeRF 360. The scene origin of contract mapping is the mean of camera position, and the ray function ‘r’ is parameterized by normalized distance from camera position. For this, we have explained it in Sec.F of Appendix.

Reviewer comment (Presentation):
What are x1 ... x8 in Table 2?

Author response:
**[Q4]** x1…x8 denotes the average distance difference of cameras from the scene origin. x2, x4 and x8 mean that the cameras are totally located in unbounded regions. For this, we have added it in Sec. 5.2 of the updated version.

Reviewer comment (Novelty):
This paper might be relevant:  
@inproceedings{Choi_2023_CVPR,  
 author  = {Choi, Changwoon and Kim, Sang Min and Kim, Young Min},  
 title   = {Balanced Spherical Grid for Egocentric View Synthesis},  
 booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},  
 month   = {June},  
 year    = {2023},  
 pages   = {16590-16599}  
}

Author response:
**[Q5]** Thanks for letting me know about the related paper. We refer to the paper in the Sec. 2 of the revised version.

Reviewer comment (Experiments):
Confusion in understanding. To my understanding, since increasing the value of p can increase the capacity of the near space, further points can be mapped into the bounded region to achieve a better representation of the unbounded scene. Why is it from your ablation experiments that higher p-values are better for proximal expression?

Author response:
**[W1-Increasing p value]** Our methodology is to formulate the one-to-one correspondence between the unbounded scene and the bounded space with a p-norm based function. The role of the p value here is to determine where the points in the unbounded space map to the bounded space. If the p value is large, the points will map more broadly to the scene origin in the bounded space. This means that more capacity is allocated to nearby objects. In this case, the nearby object means that they are close to the scene origin, not the camera. The results of ablation study show different results when the camera distance from the scene origin varies. The closer the camera is to the scene origin, the larger p value represents the near object better because it is more aligned with the camera.

Thanks for your comment. We have added the summary of this answer in Sec. 5.2. Please check the uploaded paper.

Reviewer comment (Evaluation):
Insufficient analysis of results. As can be seen from the results in Table 1, for the representation of NeRF, the effect of the p-norm mapping method proposed in this paper is not more significant than that of the voxel-based methods, and even the effect of the contract mapping function is better on NeRF. Is this method more effective for voxel grid representation?

Author response:
**[W2/Q2-Better representation for Voxel-based method]** Voxel-based methods like DVGO use explicit correspondences between coordinates and features. The explicit representation can be evenly allocated in the embedding space through our mapping function. In practice, we have a limited capacity to express neural radiance fields. That’s, in terms of making maximum use of the limited capacity, our mapping function is beneficial for the explicit representation. In contrast, when we utilize the implicit representation based on a stochastic sampling like NeRF, we cannot sometimes specify the exact correspondences, which may cause inaccurate allocation of the samples in the embedding space.

In the revised version, we have further discussed it in Sec. 5.1.

Reviewer comment (Evaluation):
From the results in Figure 6, it can be seen that the mapping function proposed in this paper can better express distant objects, but the effect of near objects is not very good. Could you please give an analysis to facilitate a better understanding of the advantages of this method?

Author response:
**[W3-Better understanding of advantage of our method]** As you mentioned, some near objects may not look so good, but this problem can be balanced according to the choice of p-value. In Fig. 6, we display the examples that are obtained by maximizing PSNR and SSIM metrics of the whole images. We have added an experiment in Fig. 16 in the Appendix, to show spatially-variant rendering quality with respect to p values. Unlike existing methods which use fixed mapping functions, we have the advantage of being able to focus on near objects by changing the p value, and vice versa.

Reviewer comment (Theory):
Since the value of $p- norm$ at $p$ less than 1 is a non-convex function, and the value of $p$ is greater than 1, does the value of $p$ have a certain upper limit?

Author response:
**[Q1-Upper limit of p value]** There is no upper limit, but only one constraint that p value should be positive. When p value is less than 1, the foreground shrinks significantly, which is not common. If it is too large, the background tends to be degraded. We add the result of these two extreme cases: in Appendix Fig. 18 in our revised paper.

Reviewer comment (Evaluation):
Why is the mapping function proposed in this paper better for the lifting of voxel-based methods? Could you please give an appropriate analysis, that can help understand your contribution better?

Author response:
**[W2/Q2-Better representation for Voxel-based method]** Voxel-based methods like DVGO use explicit correspondences between coordinates and features. The explicit representation can be evenly allocated in the embedding space through our mapping function. In practice, we have a limited capacity to express neural radiance fields. That’s, in terms of making maximum use of the limited capacity, our mapping function is beneficial for the explicit representation. In contrast, when we utilize the implicit representation based on a stochastic sampling like NeRF, we cannot sometimes specify the exact correspondences, which may cause inaccurate allocation of the samples in the embedding space.

In the revised version, we have further discussed it in Sec. 5.1.

Reviewer comment (Evaluation):
The experimental results show that the performance in the foreground areas is sometimes inferior compared to competing approaches. Even though this has a theoretical explanation (competing approaches use the identity mapping in nearby regions), it is a limiting factor of the proposed approach.

Author response:
**[W1-Foreground areas]** We design the NeRF models to minimize the photo-consistency error between all pixels of rendered images and ground-truth. Our model is thus designed to render the whole image better. Due to the nature of this, the foreground parts from our method look inferior, but the overall assessment metric is better. In this rebuttal, we want to highlight that our method can selectively focus on the quality of either foreground or background regions. As shown in  Fig. 16 of Appendix, we display the selective rendering images from our method. Different from the previous methods, our method produces region-selective novel-view images.

Reviewer comment (Experiments):
The mapping function is dependent on the COLMAP performance for computing an accurate sparse 3D point cloud. There are situations (e.g. low and/or repetitive textured scenes) where COLMAP may fail, and, hence, the proposed mapping function would be inferior to the inverted sphere or the contract approaches.

Author response:
**[W2/Q4-p value with SfM error]** To address your concern, we conduct an additional experiment about p-value estimation when COLMAP fails. In general, NeRF cannot be trained because accurate camera poses are unavailable. However, we assume that point clouds suffer from some errors such as sparse 3D prediction due to textureless regions and 3D noise due to the repeated patterns. These results are included in Fig. 14(a)(b) of Appendix E. Although the p-values became inaccurate for the large error in both cases, our RANSAC-based approach predicts relatively good p values which causes little performance drop with respect to rendering novel view images in Fig. 14 of Appendix E.

Reviewer comment (Experiments):
From what I can understand, one p-value is selected for each NeRF model. Since the distribution of the objects and structures in the scene might vary significantly, and the 3D point density also varies, the p-value is only a very coarse approximation of the 3D space distribution. The paper doesn’t focus on selecting the optimal p value for a determined area, which would make the paper even more relevant for the community.

Author response:
**[W3-Selecting optimal p value]** The p value is determined based on only scene geometry, so the same p value is applied to all NeRF models if the scene is identical. To obtain an optimal p that fully reflects the distribution and structure of the scene, we need to obtain the exact scene geometry and determine the exact boundaries to be optimized in the 3D scene. These two conditions are infeasible in practice. Therefore, we approximate the p value from the available 3D points. Additionally, we have found that this approximation is valid as long as the point cloud structure does not change significantly, as we mention Appendix E and Fig. 14(a)(b).

We have described your comment as our future work that region-aware p-value will be helpful for the advanced model. We have added it to Sec 6 in the revised paper.

Reviewer comment (Evaluation):
I am primarily concerned about the quality achieved. From both qualitative and quantitative evaluations, this approach does not appear to be better than prior approaches -- the improvement looks marginal, especially qualitatively. I suspect this is due to the suboptimal p value, which leads us to my next point.

Author response:
**[Q1-Marginal improvement]** Although the performance differences between ours and the other methods seem to be marginal, we observe significant improvements when all cameras are located extremely far from the boundary, $\times$4 and $\times$8 scenarios. These cases cannot be easily handled with the contract mapping, the second best in our experiment. As shown in Appendix Fig.17, we visualize the results on the extreme cases, which highlights the importance of our work. Of course, in these cases, the performances become better if we have a proper p value. For this, we will describe how to estimate optimal p values, which is totally based on your idea.

**[Q]** "I'm still concerned about the visual quality achieved ..."

Thanks for your proposal to make the analysis of our work better one more time. As you know, in the case of the bicycle scene, the camera viewpoints between the training set and test set are somewhat aligned, which could lead to your concerns. In the main paper, for strict fair comparison, we have evaluated ours and SoTA methods in the aligned conditions.

Additionally, as newly illustrated in Fig. 13 of Appendix, we report the rendering results on completely misaligned novel viewpoints to demonstrate the capability of our method for general unbounded scenes. We achieve significant improvement in visual quality at these diverse camera poses, even at three times further from the scene origin compared to the test set. In contrast, the SoTA mapping function is struggling to render it. We believe that the result supports the strength of our p-norm-based mapping strategy.

Reviewer comment (Experiments):
The selection of the p value is unsatisfactory. COLMAP provides only an initial, sparse geometry, so its SFM points are not error-free. Then these points go through a randomized process -- RANSAC -- that provides no chance to recover from SFM's mistakes. Have the authors considered iteratively extracting geometry from the NeRF in training and adaptively update the p value? That would give the network a chance to recover from where SFM failed.

Author response:
**[Q2-Iterative estimation for optimal p values]** We also recognize that SfM sometimes fails to reconstruct scene geometry, which can have a negative impact on estimating proper p values. As following your suggestion, we design an iterative manner to infer optimal p values. We first define any p with an arbitrary value for initialization. Then, we train a NeRF model with the initial p value to extract point clouds. Here, we render all training views, and measure absolute errors between them and their ground-truth. We next select pixels whose error is below a pre-defined threshold. Based on the selected pixels, we make a point cloud using its depth values which can be acquired from the trained NeRF model. Using the point clouds, we estimate a better p value with our RANSAC method. With the estimated p value, we train NeRF model repeatedly until the p value converges. We would like to thank you for the iterative manner that provides more accurate p values than the RANSAC-based approach. We have added the description and results to the appendix Fig. 11, Fig. 12. Please check the uploaded revision version.

We note that due to the time limit in this rebuttal, we implement this procedure for iNGP under an assumption that camera poses are not severely erroneous. Instead, we have mentioned this iterative approach for the better p value estimation as one of future works in Sec.6.

Reviewer comment (Theory):
I am unsure if the p value leading to the maximum distances among the points is the best p value. Imagine the case where two physically faraway points are used for estimating p, and there's nothing between them. Wouldn't we want to squash/compress the space between them? Why would we want a p that maximizes their distance?

Author response:
**[Q3-Maximum distance among points]** Our hypothesis is that the best p-value leads to maximizing the capacity of NeRF. To obtain the best p-value, we need to prevent allocating capacity to the free space and assign more capacity to the space with the object. It is difficult to determine free space with only point clouds in practice. Instead, we assume that the mapping function makes the points being evenly distributed within the bounded area after mapping. To do this, every point must be sufficiently far apart from each other. Obviously, the objective function of RANSAC is designed to maximize the distance among them.  It implies that there is neither too much nor too little space between two points. That’s, it avoids allocating too much free space or, conversely, too little space to represent an object.

For clearer explanation, we add more description about it. Please check Sec.4.3 of the uploaded paper.

Reviewer comment (Writing):
Some nits for the authors to improve their manuscript: (1) Figure 2 caption: pupple-colored -> purple-colored. (2) main-fold -> mani-fold.

Author response:
**[Q4-Typos]** We have fixed them. Please check the uploaded revision version.