Paper: PORF: POSE RESIDUAL FIELD FOR ACCURATE NEURAL SURFACE RECONSTRUCTION (ICLR-2024)

Reviewer comment (Evaluation):
Why should your method even outperform NeuS in Table 2 with GT poses? Does it mean your optimized pose is better than GT's or GT poses are also erroneous?

Author response:
[Outperform NeuS with GT poses]. The Ground Truth (GT) poses on the DTU dataset (Table 2) are accurate, obtained through multi-camera calibration. Conversely, the GT poses on the MobileBrick dataset (Table 4) are erroneous. Our method exhibits a slightly worse performance than NeuS with GT poses (0.85 vs. 0.77 in Chamfer Distance) in Table 2, and it outperforms NeuS with GT poses (75.67 vs. 73.74 in F1 measure) in Table 4.

Reviewer comment (Experiments):
The proposed method was only evaluated on two ideal object-centric datasets, which cannot demonstrate the generalization ability to different types of real-world data. I suggest the authors consider testing their performance on datasets like KITTI and mipnerf 360's dataset.

Author response:
[Test on more real-world datasets] Thank you for your suggestions. We conducted additional experiments on a diverse real-world dataset after the submission. Specifically, we integrated our PoRF into the **Nerfstudio** library [1] and performed ablation studies with the built-in state-of-the-art baseline method (Nerfacto) on the Nerfstudio dataset. Note that the "Nerfstudio Dataset" comprises 10 in-the-wild captures obtained using either a mobile phone or a mirror-less camera with a fisheye lens. The data was processed using COLMAP or the Polycam app to obtain camera poses and intrinsic parameters. This dataset offers researchers more than 360 real-world captures not limited to forward-facing scenes. It is similar to MipNeRF-360 but does not focus on a central object and includes captures with varying degrees of quality. The results (PSNR, the higher the better) are summarized in the following table.

|               | Egypt     | person    | kitchen   | plane     | dozer     | floating-tree | aspen     | stump     | sculpture | Giannini-Hall |
|---------------|-----------|-----------|-----------|-----------|-----------|---------------|-----------|-----------|-----------|---------------|
| Nerfacto      | 20.06     | 23.37     | 20.11     | 20.96     | 19.98     | 19.24         | 16.31     | 21.89     | 20.14     | 18.68         |
| Nerfacto-PoRF | **22.64** | **25.32** | **20.28** | **21.87** | **20.45** | **19.91**     | **16.72** | **23.13** | **21.44** | **19.18**     |

Note that Nerfacto contains the built-in pose parameter optimization method, similar to BARF and NeRFmm. We simply replaced this module with our proposed PoRF (we did not use the proposed epipolar geometry loss here for a fair comparison with the baseline), and it consistently improved performance in all 10 scenes. This demonstrates that our method is generalizable to various real-world scenarios.

Reviewer comment (Experiments):
In Eq. (7), the square of the inlier rate is used to weigh the loss terms. Why did you just simply use squares rather than other forms, e.g., cubic? Is this just an intuitive design or a careful selection after experiments?

Author response:
[Square weights in Eq 7]  We have tested multiple weight choices, including linear, square, cubic, and exponential ones. The results show that they have very minor differences, and we finally chose square weights for their simplicity and intuitiveness.

Reviewer comment (Presentation):
[Fig.2] Based on Eq.6, $\alpha$ should multiply the residuals. However, Fig.2 shows $\alpha$ multiply the input.

Author response:
[Fig. 2] Apologies for any confusion; we will revise Fig 2 in the final version for clarity.

Reviewer comment (Novelty):
[Design of PoRF] The PoRF formulates camera pose estimation as a pose refinement procedure, which takes initial poses as input and predicts the offsets. The final prediction is the initial poses plus the offsets. This paper claims the PoRF was inspired by Nerf and can be treated as one of its main contributions. However, similar modules like recurrent pose refinement or iterative offset regression are a standard module in pose estimation, including 6D object[1,4] or human pose estimation[2,3]. Compared to existing modules, PoRF only introduces an additional index as input. This makes the novelty incremental. It would be better to highlight the difference or advantage of PoRF compared to those existing modules.

Author response:
[Novelty of PoRF] We will discuss the mentioned papers in the final version, however, we argue that [1-4] addresses the generic pose-prediction problem across scenes, while our method focuses on per-scene pose optimization. The differences are:

-   [1-4] follows a "training-inference" paradigm, which is a standard machine learning approach trainable on a large dataset, predicting results (camera poses or pose offsets) during inference in new scenes. The pose network conditions on generalizable feature embeddings extracted from image appearance or geometry information like correspondences or optical flow.
    
-   Our PoRF is a per-scene optimization method, like NeRF. Our method is randomly initialised in each scene and optimizes camera pose parameters for all frames. Thus, it can take frame indices as the only input—refer to "PoRF-w/o pose” in Fig. S5. Noteworthy related works include BARF, SPARF, and Nope-NeRF, which directly optimize camera pose parameters (or pose offsets) for each frame.

Furthermore, recurrent pose refinement and iterative offset optimization have historical roots in classical multi-view geometry methods such as Visual SLAM and Structure-from-Motion systems, which are also predating [1-4]. Here, our paper asserts novelty as the first to leverage an MLP-based solution for camera pose refinement in the context of NeRF and NeuS.

Reviewer comment (Novelty):
[Epipolar] Similar Epipolar geometry constraints are also used in multi-view tasks like [5,6]. They all extract keypoints and then use Sampson distance. The framework is like a combination, which limits its novelty.

Author response:
[Novelty of Epipolar Loss] The use of epipolar geometry constraints has a historical foundation in 3D computer vision and multi-view geometry. We contend that our application of epipolar geometry to the novel domain of NeRF problems constitutes a unique contribution. Please refer to our response to Reviewer 7s1S for more discussion on the novelty of our epipolar loss.

Reviewer comment (Experiments):
The effects of training the additional MLP and computing the additional epipolar loss is likely not significant when compared to training without them but it would still be good to quantify any difference in terms of overall time to convergence. If this additional time is negligible it could further support the use of PORF as a no-brainer.

Author response:
Training time with and without PoRF. We appreciate your mention of this crucial point. Our method takes 2h35min to train the model with direct pose optimization in 50k iterations on a single A40 GPU, and it takes 2h40min to train the model with PoRF optimization. This demonstrates that PoRF introduces minor additional overhead.

Reviewer comment (Experiments):
Its a minor thing but I am curious as to how much the time/frame id input is used by PORF. Ablating with and without time input would show whether PORF mostly compensates for global shifts or whether it can learn to fix individual poses.

Author response:
Ablation with and without time/frame id input. The results are in the supplementary material—refer to Section B2 and Fig. S5. We believe the frame index is a necessary input to the MLP as it allows each frame to be uniquely identified. For instance, when two frames are captured at very similar positions (the initial pose is very similar), the frame index can be used to distinguish them.

Reviewer comment (Reproducibility):
The axis angle formulation for pose is a common way to parameterize updates to poses. It would be good to also clarify how the algorithm goes back from these to the full 4x4 (or 3x4) pose matrix used for transformations in the rest of the method.

Author response:
Conversion between axis angle and rotation matrix. We utilize the function in the Kornia library, specifically, "kornia.geometry.conversions.rotation_matrix_to_axis_angle(rotation_matrix)" and "kornia.geometry.conversions.axis_angle_to_rotation_matrix(axis_angle)." Further implementation details will be provided in the camera-ready version.

Reviewer comment (Reproducibility):
Is any position encoding used for the time/frame id input to PORF?

Author response:
Position encoding. We experimented with position encoding, but its use did not distinctly enhance overall performance. Therefore, we opted not to include it in the current version for simplicity.

Reviewer comment (Novelty):
I've noticed that the concept of applying Epipolar Constraints in situations involving inaccurate or absent poses has been previously examined in [1]. It would be interesting to see if there's any discussion or comparison of this aspect in the context of PoRF's approach.

Author response:
Thank you for your positive comments on the novelty of our proposed PoRF and the evaluation results. In the following, we primarily address concerns related to the difference between our proposed epipolar geometry loss and the 3D-3D loss proposed in [1].

While the primary contribution of our paper lies in the PoRF, our proposed epipolar loss exhibits clear distinctions from previous methods, such as SPARF and [1]. These differences include::

-   [1] computes the 3D-3D loss (Euclidean distance between two 3D points), where the 3D points originate from NeRF.
    
-   SPARF calculates 3D-2D reprojection errors (projects 3D points to 2D images and computes pixel shifts), with the 3D points also originating from NeRF.
    
-   Our method computes 2D-2D Sampson errors (the squared distance between a point x and the corresponding epipolar line x'F). Therefore, **our method does not involve NeRF and 3D representation** in computing the epipolar loss.

The 3D-3D loss [1] has been tested in SPARF, and the authors observed that the 3D-3D loss is not as robust as the 3D-2D reprojection loss in the joint optimization of camera poses with NeRF. This is because NeRF may render inaccurate depths. Although the 3D-2D reprojection loss [SPARF] surpasses the 3D-3D loss [1], it still relies on NeRF reconstruction. Consequently, we use 2D-2D Sampson distance, which does not depend on 3D representation. As a result, our method (refer to L3 in Fig. 5), utilizing L_EG without PoRF, consistently outperforms SPARF (refer to Table 1) on the DTU dataset, achieving rotation errors of 0.2 compared to 0.39 degrees.  This illustrates the superiority of our loss design over previous methods.

Reviewer comment (Novelty):
Considering that reference [1] is already available, I would appreciate it if you could provide some insight into how the PoRF Epipolar Geometry loss presents a novel approach compared to the 3D loss mentioned in [1].

Author response:
Thank you for your positive comments on the novelty of our proposed PoRF and the evaluation results. In the following, we primarily address concerns related to the difference between our proposed epipolar geometry loss and the 3D-3D loss proposed in [1].

While the primary contribution of our paper lies in the PoRF, our proposed epipolar loss exhibits clear distinctions from previous methods, such as SPARF and [1]. These differences include::

-   [1] computes the 3D-3D loss (Euclidean distance between two 3D points), where the 3D points originate from NeRF.
    
-   SPARF calculates 3D-2D reprojection errors (projects 3D points to 2D images and computes pixel shifts), with the 3D points also originating from NeRF.
    
-   Our method computes 2D-2D Sampson errors (the squared distance between a point x and the corresponding epipolar line x'F). Therefore, **our method does not involve NeRF and 3D representation** in computing the epipolar loss.

The 3D-3D loss [1] has been tested in SPARF, and the authors observed that the 3D-3D loss is not as robust as the 3D-2D reprojection loss in the joint optimization of camera poses with NeRF. This is because NeRF may render inaccurate depths. Although the 3D-2D reprojection loss [SPARF] surpasses the 3D-3D loss [1], it still relies on NeRF reconstruction. Consequently, we use 2D-2D Sampson distance, which does not depend on 3D representation. As a result, our method (refer to L3 in Fig. 5), utilizing L_EG without PoRF, consistently outperforms SPARF (refer to Table 1) on the DTU dataset, achieving rotation errors of 0.2 compared to 0.39 degrees.  This illustrates the superiority of our loss design over previous methods.