Paper: UC-NERF: Neural Radiance Field for Under-Calibrated Multi-View Cameras in Autonomous Driving (ICLR-2024)

Reviewer comment (Writing):
I think the introduction of this paper is not well written. It took me some time to understand why this work needs virtual warping and what are the differences between single-camera NeRF and multi-camera NeRF (since each camera in a multi-camera system can be deemed as a single camera).

Author response:
**1) The writing of introduction.**

Thanks for your suggestions. We have revised the introduction to better understand the multi-camera system and the motivation for virtual warping.

Reviewer comment (Experiments):
The virtual warping step relies on the MVS method to generate dense depth maps, which may not generalize to street views (I'm not certain about this) and may need further pertaining.

Author response:
**2) The usage of MVS methods.**

We use the public model of CER-MVS without finetuning. Virtual warping can enhance color correction and rendering quality, regardless of the depth sparsity. We also demonstrate its effectiveness with sparse lidar input in Tab. 1 below.

Table 1. The performance of rendering with different sources of depth for virtual warping. Tested on a randomly selected Waymo scene.
| Depth source| PSNR ↑   | SSIM ↑   | LPIPS ↓ |
|-------|-------|-------|-------|
|No virtual warping | 28.74| 0.874 | 0.338|
|Lidar virtual warping | $\underline{29.04}$ | $\underline{0.879}$ | $\underline{0.335}$ |
| MVS virtual warping (Ours) | $\mathbf{29.49}$ | $\mathbf{0.881}$ | $\mathbf{0.334}$|

Reviewer comment (Writing):
In Eq.(5), the author does not explain what is $\mathbf{b}$ and $\mathbf{d}$ denote.

Author response:
**3) The explanation of $\textbf{b}$ and $\textbf{d}$ in Eq.5.**

Thanks for your suggestion. In Eq. 5, $\textbf{b}$ and $\textbf{d}$ should be $\textbf{x}$ and $\textbf{y}$ respectively, which represent the translation term of the color correction matrix. We have revised them in the paper.

Reviewer comment (Reproducibility):
The final training loss is missing, e.g. $\mathcal{L} = \mathcal{L}_{\text{pho}} + \lambda_1 \mathcal{L}_{\text{reg}} +  \lambda_1 \lambda_2 \mathcal{L}_{\text{rpj}} $.

Author response:
**5) The final training loss is missing.**

Thanks for your suggestion. We add the final training loss in Sec. 3.5 of the revised paper.

Reviewer comment (Theory):
The pose refinement step is quite straightforward. Since the relative pose constraints $\Delta \mathbf{T}$ in the same rig can be obtained through calibration, I think it is naive to decompose the camera pose into the ego pose and a relative transformation $\Delta \mathbf{T}$. Moreover, the pose refinement step requires point correspondences, which could introduce outliers since it is well known that SOTA point matching methods are prone to repetitive structures and moving objects.

Author response:
**6) The pose refinement step relies on key points, which could introduce outliers.**

We have included the RANSAC[1] algorithm for filtering outliers before bundle adjustment.
Generally, pose optimization is based on point correspondences or photometric consistency. However, images taken by different cameras have significant color differences, which violates the assumption of photometric consistency. Furthermore, joint optimization of NeRF, color correction, and poses based on photometric loss only might not provide sufficient constraints. 

In contrast, urban scenes are typically endowed with rich textures, which allow for the extraction of a substantial number of key points. We posit that the establishment of image correspondences serves as a more effective and direct strategy for pose optimization. We demonstrate that optimizing poses through explicit pixel correspondences is better than NeRF-pose joint optimization with photometric loss in the multi-camera setting in Tab. 6 and Fig. 12 of our revised paper (in Appendix A.3.2).

Reviewer comment (Theory):
The pose refinement step relies on keypoints, which could be a shortcoming. Did the author consider DBARF (CVPR 2023) and FlowCam (NeurIPS 2023), which jointly optimize consecutive camera poses and NeRF? Actually, due to the vibrations during driving, the relative camera poses in a rig could change. I think the author mentioned it in the introduction, but the case is not handled in the formulation (Eq. (7)).

Author response:
**7) The consideration of DBARF[2] and FlowCam[3].**

Thanks for your suggestion. We finetune FlowCam and DBARF to our Waymo scenes, with results in Tab. 2 below. The performance of our method is better than FlowCam and DBARF in a multi-camera setting. DBARF optimizes poses through cost feature maps between images, which can be affected by significant color differences in images of different cameras. FlowCam requires the calculation of optical flow for pose optimization. However, given the restrained overlap between images from different cameras, the effectiveness of current optical flow techniques and subsequent pose estimation is limited in a multi-camera setting. 

Table 2. Comparisons of our method with FlowCam and DBARF. Tested on a randomly selected Waymo scene.
| Method | PSNR ↑   | SSIM ↑   | LPIPS ↓ |
|-------|-------|-------|-------|
|DBARF | 23.03 | 0.672 | 0.472 |
|FlowCam | 23.50 | 0.721 | 0.401 |
|UC-NeRF (Pose Refinement Only) | $\textbf{27.79}$ | $\textbf{0.867}$ | $\textbf{0.363}$|

**8) The change of the relative camera poses.**

Thanks for your suggestion. As you pointed out, the relative pose between cameras may change during driving, but such considerable changes are typically caused by accumulated shakes over a vehicle's long-term drive like months. NeRF models generally operate on short video sequences lasting from minutes to hours, during which the changes in the relative poses between cameras are subtle. 
On the contrary, directly modeling the transformation between cameras without our proposed spatiotemporal constraint is feasible, but it may lead to less accurate pose estimation. This is primarily because of the fact that images captured by a moving vehicle share limited overlaps, especially among the images captured by different cameras. This could potentially result in insufficient connections between images taken by different cameras, making the pose optimization relatively less constrained and unstable. Considering the two factors, we propose to utilize the spatiotemporal constraint to enhance the pose refinement.

We add an experiment in Tab. 3 below to analyze the spatiotemporal constraint. The experiment is conducted in ten commonly used self-driving sequences without any special selection. Compared to directly modeling the transformation without any constraint, our spatiotemporal constrained pose refinement leads to higher rendering quality of the reconstructed NeRF with more accurate camera poses. Fig. 12 of our revised paper also illustrates the improvement of rendering from this constraint.  

Table 3. Effectiveness of spatiotemporal constraint tested on ten Waymo scenes.
| Method  | PSNR ↑   | SSIM ↑   | LPIPS ↓ |
|-------|-------|-------|-------|
| W/O Spatiotemporal Constraint | 27.89 | 0.835 | 0.368|
| W Spatiotemporal Constraint  | **28.13** | **0.842** |**0.356**|

Reviewer comment (Reproducibility):
It would be great if the authors could share their code; even promising to share upon acceptance will be understandable. NeRF code is readily available.

Author response:
**1) The share of our code.**

We promise to share our code upon acceptance.

Reviewer comment (Reproducibility):
Are there weights on the losses before adding them?

Author response:
**2) The formulation of the loss.**

$L_{rpj}$  is used for pose refinement only. The total loss for NeRF training is the weighted sum of the photometric loss, $L_{sky}$ and $L_{reg}$. We add the formulation of the loss and revise the training strategy in Sec. 3.5. Please check out general response 1 about training strategy and loss formulation.

Reviewer comment (Writing):
What is mathbf{b} and mathbf{d} in (5)?

Author response:
**3) The definition of $\mathbf{b}$, $\mathbf{d}$, $d_v$, $d_o$.**

Many thanks. In Eq. 5, $\mathbf{b}$ and $\mathbf{d}$ should be $\mathbf{x}$ and $\mathbf{y}$ respectively, which represent the translation term of the color correction matrix. In Eq. 6, $d_v$ and $d_o$ refer to the pixel depth in
virtual views and the corresponding pixel depth in original real views. All required definitions have been added to the paper.

Reviewer comment (Writing):
Define d_v and d_o in (6).

Author response:
**3) The definition of $\mathbf{b}$, $\mathbf{d}$, $d_v$, $d_o$.**

Many thanks. In Eq. 5, $\mathbf{b}$ and $\mathbf{d}$ should be $\mathbf{x}$ and $\mathbf{y}$ respectively, which represent the translation term of the color correction matrix. In Eq. 6, $d_v$ and $d_o$ refer to the pixel depth in
virtual views and the corresponding pixel depth in original real views. All required definitions have been added to the paper.

Reviewer comment (Novelty):
In terms of the first key innovation, namely layer-based color correction, why we can not use some classical multiple views color correction solutions in the structure-from-motion field as a pre-processing step instead of a module inside the NeRF? It should be justified. Besides, some existing NeRFs also addresses similar problem such as RAWNeRF and block-NeRF, what are the main differences between the proposed method and these works?

Author response:
**1) Why we can not use some classical multiple views color correction solutions in the structure-from-motion field as a pre-processing step instead of a module inside the NeRF?**

First, limited overlap in images from different cameras hampers classical color correction methods, still causing color inconsistencies in NeRF training.
Secondly, traditional multi-view color correction methods [1,2] propagate the reference colors to the corresponding 2D pixels in the target frames, which can be challenging to ensure global color consistency across a large set of images. However, NeRF can directly learn the color correction in the 3D space, thus naturally ensuring global color consistency in the rendered 2D images. 

**2) What are the main differences between the proposed color correction method and existing works, such as Block-NeRF and Raw-NeRF[3]?**

Existing works, including Block-NeRF, alleviate the inconsistent color supervision by modeling image-dependent appearance with a global latent code for each image, which does not perform well in our setting (as illustrated in Fig. 11 and Tab. 5 of our paper). Compared to them, **(i)** we propose the concept of layer-wise color correction, which approximates color correction in foreground and sky regions as different affine transformations. This not only avoids the disentanglement of attributes unrelated to color inconsistency when the latent code implicitly models the image-dependent appearance, but also ensures the ability to model different color transformations in different regions of a single image. 
**(ii)** we provide a novel module ``Virtual Warping'' to further improve color correction. Learning one color correction for each image leads to overfitting when the training images lack color and viewpoint diversity. Virtual warping provides diverse color observations from novel viewpoints of different cameras for each 3D region, alleviating the overfitting of color correction and improving the 3D recovery.

Raw-NeRF inputs the noisy mosaicked linear raw images from the sensor rather than the common images processed by a camera pipeline. Thus, it cannot be directly applied to our scenario.

Reviewer comment (Novelty):
In terms of the spatiotemporally constrained pose refinement, there are some similar NeRFs that also consider the spatial and temporal connections between cameras for pose optimization. Name a few but not completed lists such as BARF (Lin et al. ICCV 2021) and BAD-Nerf (Wang et al. CVPR 2023). What is the novelty of the proposed method over these works?

Author response:
**3) What is the novelty of the proposed method over existing NeRF-pose joint optimization methods, such as BARF, and BAD-NeRF[4]?**

Existing methods, including BARF and BAD-NeRF, jointly optimize poses with NeRF by photometric loss. However, they do not consider two difficulties of pose optimization in multi-camera systems.

**(i)** Images taken by multiple cameras exhibit noticeable color inconsistencies. Joint optimization of NeRF, color correction, and poses based on photometric loss only might not provide sufficient constraints. Therefore, we optimize the poses more directly by establishing pixel correspondences. We add an experiment to demonstrate our superiority to BARF in Tab. 1 below. BAD-NeRF is tailored for the rendering of blurring images, which is not comparable with our method.

Table 1. Comparison of our pose refinement method with BARF. Tested on a randomly selected Waymo scene.
| Method  | PSNR ↑   | SSIM ↑   | LPIPS ↓ |
|-------|-------|-------|-------|
|BARF|  28.65 | 0.856 |  0.373 |
|UC-NeRF (Ours) | **29.14** | **0.867** | **0.355** | 

**(ii)** Inaccurate relative transformations between different cameras will cause rendering artifacts. However, directly optimizing the camera transformations without any constraint is unstable and inaccurate in our setting, since the images captured by multiple cameras share limited overlaps. We introduce a constraint that explicitly models the relative transformations between cameras as temporally consistent but optimizable variables to achieve more accurate pose refinement.

Reviewer comment (Experiments):
The experiment comparisons are limited since only Mip-NeRF was used. Why not compare to some large-scale NeRDs such as block-NeRF or multi-views NeRFs such as MC-NeRF and NeRF-MS. The authors should justify the reason.

Author response:
**4) The experiment comparisons are limited since only Mip-NeRF was used. Why not compare to some large-scale NeRFs such as block-NeRF or multi-views NeRFs such as MC-NeRF[5] and NeRF-MS[6].**

Thanks for your suggestion. We have compared the strategies used for color correction and pose refinement in Block-NeRF in Tab. 5, Tab. 6, Fig. 11, and Fig. 12 of our revised paper (in Appendix A.3.2). 
Block-NeRF does not release its code. According to its paper, it is built on Mip-NeRF. For a fair comparison, we replace our color correction strategy and pose refinement strategy in ZipNeRF with the strategies used in Block-NeRF. (Block-NeRF uses the color correction strategy from NeRF-in-the-wild and pose refinement strategy same as NeRF$--$.) 

We add an experiment in Tab. 2 below, demonstrating our method's superiority over NeRF-MS. NeRF-MS lacks specific modeling of color correction for multi-camera systems, potentially correcting appearance unrelated to color inconsistency and resulting in inferior performance.  MC-NeRF is a contemporaneous work posted on arXiv in September 2023 without public codes. Therefore, we cannot compare it with ours within the limited discussion time. It focuses on optimizing the intrinsics of different cameras during pose optimization while we propose the spatiotemporal constraint between different cameras to enhance pose optimization. The discussion of these works is also updated in the related work of our revised paper.

Table 2. Comparison of our method with NeRF-MS in a randomly selected Waymo scene.
| Method  | PSNR ↑   | SSIM ↑   | LPIPS ↓ |
|-------|-------|-------|-------|
| NeRF-MS | 26.87 | 0.828 | 0.392 |
| UC-NeRF (Color Correction Only) | **28.15** | **0.851** | **0.374** |

Reviewer comment (Experiments):
The whole pipeline seems verbose since three independent modules are stitched together with few connections. Could the proposed UC-NeRF be trained in an end-to-end manner? Additionally, the efficiency comparisons of different methods are expected to be provided in the experiments.

Author response:
**1） The whole pipeline seems verbose since three independent modules are stitched together with few connections. Can the proposed UC-NeRF be trained in an end-to-end manner?**

Yes, our NeRF is trained end-to-end with the learnable color correlation and virtual warping as data augmentation. The pose refinement is performed as a preprocessing step to provide accurate poses for NeRF training. 
Our method presents a unified framework for solving the problems of neural rendering with under-calibrated multi-view cameras, where the three modules are designed to tackle the key problems in this setting, such as inaccurate poses, and inconsistent color images.
The proposed three modules can complement each other and contribute as a whole to improve the final rendering results, which are correlated with each other and effective.

**2） The efficiency comparisons of different methods.**

Thanks for your suggestion. We add Tab. 1 below to the paper for efficiency comparison. Note that our training time includes both steps (pose refinement and NeRF training) described in Sec. 3.5. Zip-NeRF is more efficient than other methods except Instant-NGP, which is specifically designed for NeRF acceleration. Since our method is built upon Zip-NeRF, our method consumes a bit more time than Zip-NeRF but achieves a significant improvement in rendering quality.

Table 1. Efficiency Analysis. Tested on one NVIDIA Tesla V100 GPU with image resolution $1920 \times 1280$.
| Method  | Training | Inference (per image) | PSNR |
|-------|:-: | :-:|  :-:| 
| Mip-NeRF |20h| 70s| 22.42| 
|Mip-NeRF-360 | 14h | 42s | 24.46| 
|Instant-NGP | 30min | 0.35s |23.84|
|S-NeRF | 15h | 80s | 24.89|
|Zip-NeRF | 2h | 2s | 26.21|
|Ours | 3h | 3.2s | 28.13 |

Reviewer comment (Novelty):
The first two contributions, i.e., Layer-based Color Correction and Virtual Warping, are kind of trivial and have limited novelty. They are constructed based on existing methods like the pretrained segmentation model, the MVS model, and a geometric consistent check approach. The procedures of these two parts perform a preprocessing-like role in the proposed method. The authors are suggested to give more clarifications and highlight their specific contributions.

Author response:
**3） The highlight of the first two contributions.**

The first two modules are embedded in the NeRF framework for end-to-end training. They respectively address a specific challenge of color correction during training of NeRF in multi-camera systems, i.e., **(i)** the way to model color correction and **(ii)** color correction in limited observations.
 
**(i)** Existing methods do not perform well in color correction in our setting, as shown in Fig. 11 and Tab. 5 (in Appendix A.3.2) of our revised paper. They generally alleviate the inconsistent color supervision by modeling image-dependent appearance with a global latent code for each image. However, the lack of explicit modeling for color correction from the latent code can lead to the disentanglement of attributes unrelated to color inconsistency. Moreover, the capacity of a global latent code to uniformly correct colors in different regions of an image is limited, especially when different regions correspond to different color transformations. We propose the concept of layer-wise color correction. 
We approximate color corrections in foreground and sky regions as different affine transformations. This strategy not only avoids the disentanglement of attributes unrelated to color inconsistency when the latent code implicitly models the image-dependent appearance, but also ensures the ability to model different color transformations in different regions of a single image.

**(ii)** The limited observations used for color correction. Learning one color correction for each image leads to overfitting when the training images lack color and viewpoint diversity, especially for areas observed by side-view cameras, which have fewer observations and limited overlapping with front-view areas. Virtual warping provides a novel way to improve color correction. It provides diverse color observations from more viewpoints of different cameras for each 3D region, alleviating the overfitting of color correction and improving the 3D recovery.

Reviewer comment (Experiments):
For the color correction part, it seems that the accuracy of the correction performance highly depends on the sky segmentation. However, the cases shown in the paper only contain clean skies and sunny weather. I am wondering how this work performs under diverse weather conditions. Because this work aims at multi-camera systems that are widely used in outdoor scenes (such as autonomous driving), the real-world application would be preferred over the method itself.

Author response:
**4) The performance under diverse weather conditions.**

Thanks for your suggestion. Our datasets have included sunny and overcast weather, and we add rainy and night conditions here. As shown in Tab. 2 below, our approach notably improves across these scenarios. Please check the rendered images in the Fig. 13 (in Appendix A.3.2) of the revised paper and the video in the supplementary material.

Table 2. Comparison of diverse weather conditions. Tested on Waymo Segment-100170 (Rainy) and Waymo Segment-150908 (Night).
| Method  | PSNR (Rainy) ↑ | SSIM (Rainy) ↑ | LPIPS (Rainy) ↓ | PSNR (Night) ↑ | SSIM (Night) ↑ | LPIPS (Night) ↓ |
|-------|:-: | :-:|  :-:| :-: | :-:|  :-:|
| Zip-NeRF |27.65 | 0.831 | 0.434 | 30.69 | 0.864 | 0.512 |
| UC-NeRF (Ours) | **30.03** | **0.866** | **0.387** | **31.32** | **0.869** | **0.491**|

Reviewer comment (Novelty):
For the proposed Spatiotemporally Constrained Pose Refinement, please clarify its relationship and difference to the bundle adjustment.

Author response:
**5) The relationship and difference to the bundle adjustment.**

Bundle adjustment (BA) is a widely used technique that involves simultaneously adjusting pose parameters and 3D point locations.
In our framework, we perform BA to optimize poses, and additionally incorporate the spatiotemporal constraint between cameras to eliminate errors of the under-constrained pose estimation of small-overlapping cameras in a multi-camera system. 

---

Thanks for your feedback. We greatly appreciate your questions and suggestions concerning our pose optimization module. First, we would like to emphasize that our UC-NeRF is designed to address the problem of color inconsistency in NeRF training under a multi-camera system, and pose refinement is only a part of this. It works in conjunction with other modules to tackle the challenges of color inconsistency. 

In our spatiotemporally constrained pose refinement module, we optimize the poses using bundle adjustment (BA). However, BA is a general pose optimization technique that relies on different energy functions to optimize poses. Many works enhance pose accuracy through the design of energy functions, such as Pixel Perfect SFM[1] rewrites reprojection loss as pixel's feature loss, and NIM-SLAM [2] rewrites pixel's photometric loss as multi-scale patch’s photometric loss. For our multi-camera setting, we also redesign the energy function. We consider the prior of the cameras, i.e., their spatiotemporal relationships, and add additional constraints to the poses. We design the energy function in the form of Eq.1 below:

$$
L=\sum_{((i, k),(j, l)) \in \mathcal{E}}\left\|\mathbf{p}_l^j-\Pi_l\left(\left(\mathbf{T}^j \Delta \mathbf{T}_l\right)^{-1} \mathbf{T}^i \Delta \mathbf{T}_k \Pi_k^{-1}\left(\mathbf{q}_k^i\right)\right)\right\|^2. \tag{1}
$$

$\mathbf{T}^{i}_{k}$, which is the kth camera's pose at time $i$, is modeled as  $\mathbf{T}^i \Delta \mathbf{T}_k$ with two parts: the ego pose $\mathbf{T}^i$, and the temporally consistent and optimizable relative transformation of the cameras $\Delta \mathbf{T}_k$. The inclusion of this constraint significantly improves the rendering results, as shown in Tab. 1 below and Fig.12 in the appendix of the paper. 

Table 1. Effectiveness of spatiotemporal constraint tested on ten Waymo scenes.
| Method  | PSNR ↑   | SSIM ↑   | LPIPS ↓ |
|-------|-------|-------|-------|
| W/O Spatiotemporal Constraint | 27.89 | 0.835 | 0.368|
| W Spatiotemporal Constraint  | **28.13** | **0.842** |**0.356**|

Otherwise, independently modeling camera poses without the proposed spatiotemporal constraint is feasible, but it may lead to less accurate pose estimation. This is primarily because images captured by a moving vehicle share limited overlaps, especially among the images captured by different cameras. This could potentially result in insufficient connections between images taken by different cameras, making the pose optimization relatively less constrained and unstable. 

In addition, we want to state the motivation for using BA. We have tried various strategies for joint optimization of NeRF and pose in multi-camera systems. However, due to the color differences between images, joint optimization of NeRF, pose, and color correction through photometric loss only might not provide sufficient constraints. Tab. 2 below (also shown as Tab. 6 in our paper) also demonstrates that explicitly establishing pixel correspondence to optimize poses shows more significant improvement than jointly learning poses, NeRF, and color correction.

Table 2. Comparison of our pose refinement method with nerf-pose joint refinement methods BARF, NeRF-- (the same pose refinemment strategy in Block-NeRF). Tested on a randomly selected Waymo scene.

| Method  | PSNR ↑   | SSIM ↑   | LPIPS ↓ |
|-------|-------|-------|-------|
|NeRF$--$ | 28.48 | 0.851 | 0.383 |
|BARF|  28.65 | 0.856 |  0.373 |
|UC-NeRF (Ours) | **29.14** | **0.867** | **0.355** | 
---
[1] Lindenberger P, Sarlin P E, Larsson V, et al. Pixel-perfect structure-from-motion with featuremetric refinement[C]//Proceedings of the IEEE/CVF international conference on computer vision (ICCV). 2021: 5987-5997. 

[2] Li H, Gu X, Yuan W, et al. DENSE RGB SLAM WITH NEURAL IMPLICIT MAPS[C]//The Eleventh International Conference on Learning Representations (ICLR). 2022.

Reviewer comment (Presentation):
First, I would like the title to be more specific. Its very misleading as the paper does use automotive setting (sky+fg) as necessary input. The title seems to suggest that its a generic method for uncalibrated cameras. Also all results are on automotive. Please correct it.

Author response:
**1) The title needs to be more specific.**

Thanks for your suggestion. Our method aims at novel view rendering with under-calibrated multi-camera systems, where autonomous driving is one of the typical scenarios. Moreover, public autonomous driving datasets like Waymo are widely used benchmarks, making it convincing to compare our method with other works using the datasets. We are also open to incorporating ''autonomous driving'' in the title if needed.

Thanks for your valuable feedback. We greatly appreciate your suggestion regarding the title and agree with it. Accordingly, we have incorporated the concept of 'autonomous driving' into the title in the most recent revision of our paper.

Reviewer comment (Writing):
In Eq. 5, what is b and d. I think the authors missed defining those params.

Author response:
**2) The missing definition of ‘b’ and ‘d’.**

Thanks for pointing this out. **b** and **d** should be **x** and **y** as the translation term in the color correction matrix. Please check our general response 2 about the typos and symbols.

Reviewer comment (Theory):
In Eq.7, isn't it better to reduce reprojection error as a function of direct 4x4 transformation between the cameras across time. For example directly modeling the transformation between cameras labeled T^i*delta_T1 and T^j * delta_T1 in Fig4. The reason for this is that at time t=0, its convenient to bring all the three cameras in Fig4 to the car's coordinate system, but later assuming that this transformation is fixed and won't perturb due to camera shake, bad roads, bumps etc. is an unrealistic assumption. Then propagating this incorrect assumption across time from T^i to T^j can lead to erroneous extrinsics estimation in Eq. 7.

Author response:
**3) Isn't it better to reduce reprojection error as a function of direct 4x4 transformation between the cameras across time?**

Thanks for your suggestion. As you pointed out, the relative pose between cameras may change during driving, but such changes are typically caused by accumulated shakes over a vehicle's long-term driving like several months. NeRF models generally operate on very short video sequences lasting from minutes to hours, during which the changes among the relative poses of the multiple cameras are subtle. On the contrary, directly modeling the transformation between cameras without the proposed spatiotemporal constraint is feasible, but it may lead to less accurate pose estimation. This is primarily due to the fact that images captured by a moving vehicle share limited overlaps, especially among the images captured by different cameras. This could potentially result in insufficient connections between images taken by different cameras, making the pose optimization relatively less constrained and unstable. 
Considering both factors, we propose to leverage the spatiotemporal constraint to enhance the pose refinement.

We add an experiment in Tab. 1 below to analyze the spatiotemporal constraint. The experiment is conducted in ten commonly used self-driving sequences without any special selection. Compared to directly modeling the transformation without any constraint, our spatiotemporal constrained pose refinement leads to higher rendering quality of the reconstructed NeRF with more accurate camera poses. Fig. 12 of our revised paper also illustrates the improvement of rendering from this constraint.

Table 1. Effectiveness of spatiotemporal constraint tested on ten Waymo scenes.
| Method  | PSNR ↑   | SSIM ↑   | LPIPS ↓ |
|-------|-------|-------|-------|
| W/O Spatiotemporal Constraint | 27.89 | 0.835 | 0.368|
| W Spatiotemporal Constraint  | **28.13** | **0.842** |**0.356**|

Reviewer comment (Theory):
When virtual views are created and added to the training set as discussed in Section 3.3, it has holes either due to low confidence or occlusion as shown in Fig 9, then how does the sky segmentation from Yin 2022 perform in these images. What happens to the mask value in the missing image regions in virtual view and how does in impact Equation 3.

Author response:
**4) How does the segmentation model perform in virtual views?**

We do not employ any segmentation in virtual views. Both the sky mask and color of a pixel in the virtual view are obtained from the corresponding pixel in the original frames of the input sequence based on the warping by depth.

Reviewer comment (Reproducibility):
The training strategy is not very clear. So, you train using Eq.4 upto some convergence, Then you get A and C color correction matrices. Then again you apply the corrected A and C matrices to virtual views in Section 3.3. Then you get color corrected virutal views. Then you again train the original set of spatial+temporal data but this time include virtual views? I think Section 3.5 needs more explanation because it joins all your individual modules and is critical to understand how the complete system is working.

Author response:
**5) The training strategy is not very clear. Sec. 4.4 is redundant. Explain Sec. 3.5 in detail.**

Thanks for your suggestion. We revise the training strategy in Sec. 3.5 (detailed in general response 1) and move Sec. 4.4 to the appendix.

Reviewer comment (Presentation):
Section 4.4 is redundant I think. It has nothing to do with the main goal of your paper and that space could have been used to explain your main parts e.g. Section 3.5 in detail.

Author response:
**5) The training strategy is not very clear. Sec. 4.4 is redundant. Explain Sec. 3.5 in detail.**

Thanks for your suggestion. We revise the training strategy in Sec. 3.5 (detailed in general response 1) and move Sec. 4.4 to the appendix.