Paper: USB-NeRF: Unrolling Shutter Bundle Adjusted Neural Radiance Fields (ICLR-2024)

Reviewer comment (Novelty):
The contribution and the novelty are limited. The effect of the rolling shutter is well-known in 3D vision community, and many works have been trying to solve it in the past years. The proposed methods only contain the basic concept of modeling a rolling shutter.

Author response:
**Q1: The contribution and the novelty are limited. The effect of the rolling shutter is well-known in 3D vision community, and many works have been trying to solve it in the past years. The proposed methods only contain the basic concept of modeling a rolling shutter.**

A1: Although rolling shutter effect correction has been researched for many years, prior methods still face challenges to solve it. In contrary, we are the first to formulate the problem under the framework of NeRF and deliver very promising results for the problem.

Reviewer comment (Evaluation):
I think the ATE (absolute trajectory error) in Table 3 is the same as the absolute translation error (I'm used to the term `translation` instead of `trajectory`). Therefore, only the translation errors are given and no rotation errors are provided. Moreover, the unit of the ATE is unclear (I think it is in meters).

Author response:
A2: Yes, the ATE metric is mainly for absolute translation error and the unit is in meters. We have added the unit explanation in the table caption of the main paper to make it clearer.

We re-computed the rotation error with the RPE metric (relative pose error) for rotation. The unit is in ``degrees per frame'' (i.e. $\Delta =1$). The results for both synthetic and real datasets are presented in the following table. Due to the space limit, we only report the average metrics here and the details for each sequence can refer to both Table 9 and Table 10 of the main paper. It demonstrates USB-NeRF also has a better rotation estimation accuracy than prior methods. 

|  |  COLMAP | BARF | RSBA | NW-RSBA | USB-NeRF-linear | USB-NeRF-cubic |
|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
| Synthetic |  2.367$\pm$ 1.348 | 8.386$\pm$ 5.119 | 5.179$\pm$ 2.627 |  2.090$\pm$ 1.492  | 0.446$\pm$ 0.337 | __0.307__ $\pm$ 0.140 |
| TUM-RS  | 0.393 $\pm$ 0.218 | 0.429 $\pm$ 0.430 | 2.095 $\pm$ 2.607 |  8.379 $\pm$ 9.928 | 0.283 $\pm$ 0.285 | __0.227__ $\pm$ 0.246 |

Reviewer comment (Evaluation):
The cubic B-Splines interpolation is suitable for complex camera trajectories, however, it can be worse than the linear interpolation method when the camera moves at a constant velocity.

Author response:
A3: Cubic B-Splines interpolation is indeed a bit worse than the linear interpolation method when the camera moves at a constant velocity, from the results of Table 1 in the paper. The PSNR metric becomes 0.25 dB worse, which we think is still tolerable.

Reviewer comment (Evaluation):
From Fig. 6, BARF looks much worse than NeRF; and from other tables and figures, BARF performs almost the same as NeRF. Are there any explanations for this?

Author response:
A4: Thank you for pointing out. We are sorry that we put a wrong figure of BARF in Fig. 6. We corrected it in the main paper now. However, we find that BARF still performs a bit worse than NeRF on the **White Room** sequence. The main reason is that we need to optimize the camera poses of BARF for novel view synthesis experiment, to align the novel view image against the trained scene representation for evaluation. Therefore, the camera pose might converge to a wrong solution sometime if both the learned 3D representation and images are distorted. In contrary, NeRF does not require this since they fix the camera poses which are estimated from COLMAP.

Reviewer comment (Experiments):
It is expected that BARF can fail under the rolling-shutter setting since each row of the image is recorded at different timestamps. My question is can we build a stronger baseline method that associates each row with a camera pose, then we can use BARF to optimize these camera poses and obtain better results? It can be time-consuming since an image can often have >400 rows, and then we have to optimize too many parameters. A simplified way is to split the image into R row blocks, where R <<< the width/height of that image. Then we have a simplified stronger baseline than BARF.

Author response:
A5: We conduct this interesting experiment with BARF. In particular, we split every image into both 3 blocks (160 lines per block), and optimize camera pose of each block separately via BARF. We also repeat the experiment with 5 blocks (96 lines per block). The experimental results are presented in following Table on synthetic datasets. It demonstrates that it can have improved performance on some sequences, but could also be worse for others. We think there are two possible reasons: 1) divide the whole image into chunks could indeed help since it suppress the distortions. However, each chunk could still have considerable distortion if the total number of chunks is small, which would affect the performance; 2) divide the whole image into too many chunks would lead to insufficient information of each chunk for individual pose optimization, which would also affect the final performance. Therefore, it would beneficial to model the RS effect into NeRF for better performance. 

| | |Blue Room| | |Living Room| | |White Room| | |Avg.| |
|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
| | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$  | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$  | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| BARF | 19.14 |  0.576 | 0.3446  | 21.48 |  0.690 | 0.3341  |  18.43  | 0.545 |  0.3837 | 19.68 | 0.604 |  0.3541 |
| BARF*(3 blocks) |  19.65  | 0.621 |  0.3552 |  19.74  | 0.682 |  0.3565 |  18.47  | 0.532 |  0.3546 | 19.29 | 0.611 |  0.355  |
| BARF*(5 blocks) | 20.12  | 0.634 |  0.2429 |  17.97  | 0.650 |  0.4310 | 19.04  | 0.557 |  0.3354 | 19.04 | 0.614 |  0.3364 |
| USB-NeRF (ours) | __31.85__ | __0.909__  | __0.0573__ | __34.89__ | __0.939__  | __0.0415__ | __30.57__ | __0.892__  | __0.0576__ | __32.44__ | __0.914__ | __0.0521__ |

Reviewer comment (Experiments):
It would be interesting to see more analysis/discussions on how much is the performance gap when modelling rolling shutter effect vs not modelling with datasets from modern DSLR cameras and smart phone cameras.

Author response:
We captured 5 additional sequences using GoPro HERO6 Black, Canon camera (EOS M3), and iPhone 14 Pro, and made more analysis on real-world datasets. All rolling shutter images are captured at the frequency of 30Hz. We find that the captured images with modern cameras still exhibit obvious RS effects if they undergo fast motions, which can easily happen for certain scenarios, e.g. GoPro carried by a flying drone, which is a common set-up for most hobbyists or taking photos on a moving vehicle.

Since it is hardly to capture corresponding GS images, we only make qualitative comparisons on these datasets. Please refer to Figure 10, Figure 11, and Figure 12 in Appendix A.3 of the main paper for the experimental results on real-world datasets.

The experimental results demonstrate that: 1) both NeRF and BARF fail to correct the RS effect distortion; 2) both NeRF and BARF produce significant artifacts if the camera motion is not in a constant direction (e.g. the camera is moving forward and then backward); 3) our method corrects the RS effect distortion with no artifact; 4) it is still valuable to model RS effect for certain scenarios with modern cameras.