Paper: RoDyn-SLAM: Robust Dynamic Dense RGB-D SLAM with Neural Radiance Fields (ICLR-2024)

Reviewer comment (Novelty):
Doesn't introduce substantial architectural changes or novelty for NeRF-based SLAM. Such as multi-resolution hash encoding from instant ngp, joint optimization and store a subset of pixels to represent each keyframe from Co-SLAM, etc.

Author response:
**W1: More substantial architectural changes details**.   
Thanks for the suggestion. We have added the more substantial architectural details for our SLAM system in Appendix A.1 in the revised paper.

Reviewer comment (Experiments):
Enhancing additional losses (e.g., Edge reprojection loss) and masks to improve accuracy in dynamic object scenarios. How does the system's performance compare in terms of pose estimation and reconstruction in static scenes? Is it competitive with other SLAM systems?

Author response:
**W2: More experiments in static scenes**.   
Thanks for the suggestion. To better demonstrate the system performance of pose estimation and reconstruction in a static scene, we choose the real-world RGB-D dataset [1] rather than the synthetic datasets [2,3]. Compared with the current the state of the art neural rgb-d SLAM, we evaluate the pose estimation and reconstruction results on three represented sequences. Additional detailed comparison results are available in Table 9 in Appendix A.5. 

Compared with our baseline methods Co-SLAM [4], `our method does not compromise the performance of the original SLAM methods in terms of tracking and mapping in static scenes`. In fact, it achieves competitive results.Notably, our proposed optimization algorithm is not restricted to a specific slam system. Thus, it can also be applied to other neural rgb-d slam methods to improve the data association between the inter-frame. 

Compared with the E-SLAM [5], our SLAM system requires less storage space and achieves real-time running at 10fps, despite several hours of training time needed for E-SLAM. Consequently, our method strikes a well-balanced compromise between accuracy and speed, demonstrating competitive performance of pose estimation when compared to state-of-the-art methods.


> [1] J ̈urgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A benchmark for the evaluation of rgb-d slam systems. In IROS, 2012. 

> [2] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv preprint, 2019. 

> [3] Dejan Azinovi ́c, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, and Justus Thies. Neural rgb-d surface reconstruction. In CVPR, 2022. 

> [4] Hengyi Wang, Jingwen Wang, and Lourdes Agapito. Co-slam: Joint coordinate and sparse parametric encodings for neural real-time slam. In CVPR, 2023.

> [5] JMohammad Mahdi Johari, Camilla Carta, and Franc ̧ois Fleuret. Eslam: Efficient dense slam system based on hybrid representation of signed distance fields. In CVPR, 2023.

Reviewer comment (Theory):
A key concern in this paper is the insufficiently explained rationale for incorporating neural radiance field representation in dynamic SLAM.

Author response:
**W1: Motivation of neural RGB-D SLAM over dense RGB-D SLAM in dynamic scene**.   
Thanks for the suggestions. We have revised the introduction to motivate the benefits compared to existing dynamic dense RGB-D SLAM methods. All the modified text is highlighted in blue in the revised version.

Compared with the existing dense RGB-D dynamic SLAM methods, the key advantage of neural RGB-D SLAM lies in its neural implicit representation method. The neural scene representations have attractive properties for mapping in dynamic scenes, including `improving noise and outlier handling, geometry estimation capabilities for unobserved scene parts, high-fidelity reconstructions with reduced memory usage, and the ability to generate high-quality static background images from novel views`. Moreover, neural radiance fields also open up possibilities for the subsequent development of dynamic SLAM, such as distilling perceptive information and accomplishing object-based tracking or reconstruction.

Finally, please allow us to briefly introduce our motivation and contribution to building Rodyn-SLAM system. Existing neural RGB-D SLAM methods rely on a static environment assumption and do not work robustly within a dynamic environment due to the inconsistent observation of geometry and photometry. To tackle this problem, we propose a dynamic object identification method and filter the invalid sample ray to recover the static scene map. To further improve the accuracy of pose estimation in dynamic scenes, we propose novel and robust pose optimization methods utilizing edge projection loss to enhance data association between convisible frames.

Reviewer comment (Experiments):
There is a noticeable absence of robust baseline comparisons during the evaluation.

Author response:
**W2: More experiment results with stronger baselines**.    
Thanks for the suggestion. Due to the current limitation of NeRF-based SLAM methods in handling dynamic scenes, we propose a general robust dynamic Nerf-based SLAM method RoDyn-SLAM, which constitutes the innovation of this paper. We have selected state-of-the-art NeRF-based SLAM methods as our benchmark for comparison, serving as robust baselines. Following your suggestions, we have added the results of MID-Fusion [1] and Droid-SLAM [2] in Table 2 and Table 3 in the revised paper, respectively. Due to the advanced factor graph optimization algorithm and extensive training data, Droid-SLAM has achieved superior pose estimation results in dynamic scenes compared to Nerf-based SLAM methods.

`Bundle-SDF [3] and vMAP [4]` are object-based neural SLAM methods rather than a general nerf-based SLAM method. It relies on external conditions, such as the `availability of pose prior` or the requirement for the `camera to remain stationary`. Thus, we do not add the comparison results of these methods in the revised paper. The more detailed analysis is summarized as:

* Bundle-SDF is designed for estimating the object pose and reconstructing the object mesh. Our approach aims to estimate the camera pose in dynamic scenes by leveraging a static background. The objectives of these two methods are distinct. Additionally, the validation datasets used for Bundle-SDF require the camera to remain stationary, allowing for the pose estimation of moving objects. The datasets we employed for evaluation do not meet these conditions. 

* vMAP primarily emphasizes joint reconstruction of objects, relying on estimated poses from other methods i.e. ORB-SLAM3 to provide the initial pose. Therefore, we believe that the direct comparison of pose optimazation is unfair. We attempted to make a comparison of reconstruction performance with vMAP given GT pose on TUM RGB-D dynamic sequence. However, the final reconstruction results for dynamic scenes were notably blurred. Actually, due to the absence of joint optimization between the camera pose and implicit map, the dynamic objects can not be filtered, resulting in significant blurriness and floaters in the reconstructed scene.

> [1] Binbin Xu, Wenbin Li, Dimos Tzoumanikas, Michael Bloesch, Andrew Davison, and Stefan Leutenegger. Mid-fusion: Octree-based object-level multi-instance dynamic slam. In ICRA, 2019.
 
> [2] Zachary Teed and Jia Deng. DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras. In NeurIPS, 2021.

> [3] DBowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas M ̈uller, Alex Evans, Dieter Fox, Jan Kautz, and Stan Birchfield. Bundlesdf: Neural 6-dof tracking and 3d reconstruction of unknown objects. In CVPR, 2023. 

> [4] Xin Kong, Shikun Liu, Marwan Taher, and Andrew J Davison. vmap: Vectorised object mapping for neural field slam. In CVPR, 2023.

Reviewer comment (Experiments):
The evaluation primarily compares the proposed method against static neural SLAM, which may not be entirely fair. It would be beneficial to include related works as stronger baselines, including traditional SLAM methods like MID-Fusion and Droid-SLAM, as well as NeRF SLAM methods like vMAP and BundleSDF.

Author response:
**W2: More experiment results with stronger baselines**.    
Thanks for the suggestion. Due to the current limitation of NeRF-based SLAM methods in handling dynamic scenes, we propose a general robust dynamic Nerf-based SLAM method RoDyn-SLAM, which constitutes the innovation of this paper. We have selected state-of-the-art NeRF-based SLAM methods as our benchmark for comparison, serving as robust baselines. Following your suggestions, we have added the results of MID-Fusion [1] and Droid-SLAM [2] in Table 2 and Table 3 in the revised paper, respectively. Due to the advanced factor graph optimization algorithm and extensive training data, Droid-SLAM has achieved superior pose estimation results in dynamic scenes compared to Nerf-based SLAM methods.

`Bundle-SDF [3] and vMAP [4]` are object-based neural SLAM methods rather than a general nerf-based SLAM method. It relies on external conditions, such as the `availability of pose prior` or the requirement for the `camera to remain stationary`. Thus, we do not add the comparison results of these methods in the revised paper. The more detailed analysis is summarized as:

* Bundle-SDF is designed for estimating the object pose and reconstructing the object mesh. Our approach aims to estimate the camera pose in dynamic scenes by leveraging a static background. The objectives of these two methods are distinct. Additionally, the validation datasets used for Bundle-SDF require the camera to remain stationary, allowing for the pose estimation of moving objects. The datasets we employed for evaluation do not meet these conditions. 

* vMAP primarily emphasizes joint reconstruction of objects, relying on estimated poses from other methods i.e. ORB-SLAM3 to provide the initial pose. Therefore, we believe that the direct comparison of pose optimazation is unfair. We attempted to make a comparison of reconstruction performance with vMAP given GT pose on TUM RGB-D dynamic sequence. However, the final reconstruction results for dynamic scenes were notably blurred. Actually, due to the absence of joint optimization between the camera pose and implicit map, the dynamic objects can not be filtered, resulting in significant blurriness and floaters in the reconstructed scene.

> [1] Binbin Xu, Wenbin Li, Dimos Tzoumanikas, Michael Bloesch, Andrew Davison, and Stefan Leutenegger. Mid-fusion: Octree-based object-level multi-instance dynamic slam. In ICRA, 2019.
 
> [2] Zachary Teed and Jia Deng. DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras. In NeurIPS, 2021.

> [3] DBowen Wen, Jonathan Tremblay, Valts Blukis, Stephen Tyree, Thomas M ̈uller, Alex Evans, Dieter Fox, Jan Kautz, and Stan Birchfield. Bundlesdf: Neural 6-dof tracking and 3d reconstruction of unknown objects. In CVPR, 2023. 

> [4] Xin Kong, Shikun Liu, Marwan Taher, and Andrew J Davison. vmap: Vectorised object mapping for neural field slam. In CVPR, 2023.

Reviewer comment (Experiments):
Although Fig. 3 displays TSDF fusion results, these results are not reported in tables. Comparing the results with and without the motion mask, as proposed in this paper, using TSDF fusion would be valuable and should be considered a baseline method.

Author response:
**W3: TSDF fusion as a baseline**.


Thanks for the suggestion. `TSDF-fusion` is a traditional reconstruction method with `ground truth pose` as input rather than a SLAM method. We have reported the comparison results and analysis of TSDF-fusion in the revised version, available in Table 8 in Appendix A.4, aiming to address your concern. However, directly using this as a baseline method to compare with the other neural RGB-D slam method in dynamic scenes is not entirely fair. Because the `current mapping results` are built `based on an estimated pose rather than the ground truth pose`. The quality of mapping will be significantly affected by the error of pose estimation in the dynamic scene. This differs slightly from the quality evaluation of reconstruction in static environments.

Finally, please allow us to explain why the comparison results for TSDF-Fusion are not included in the tables. The TUM dataset does not provide the ground truth mesh or point cloud for evaluating the reconstruction quality, and the BONN dataset only offers the point cloud captured by lidar scan. Thus, we evaluate the quantitative results in the BONN dataset. To better compare and visualize the reconstruction results, we utilize TSDF-fusion to generate the scene mesh as the ground truth mesh, leveraging the known ground truth pose.

Reviewer comment (Experiments):
The paper's visualization results are impressive; however, for a SLAM system, it would be preferable to include videos or real-world demonstrations as additional supplementary material.

Author response:
**W4: More visualization results on the real-world dataset**.  
Thanks for the suggestion. Following your suggestions, we provide an additional video demo to visualize the pose estimation and reconstruction results. Please refer to the supplementary material. Notably, to better illustrate the effort of our proposed method, we have evaluated the performance of the SLAM system on real-world datasets such as TUM and BONN rgb-d datasets rather than the synthetic datasets in the original paper. These two datasets cover representative dynamic scenes in daily life. Thus, we believe it can demonstrate the real-world performance of our SLAM system.

Reviewer comment (Writing):
While the overall writing is good, there are a few places that worthy of fix: e.g., in section 3.2 eq (4) there is no introduction on $j$ and $k$; also typos such as Camear in Fig 1. A thorough proofread is recommended.

Author response:
**W1: Lack introduction in Section 3.2 eq (4)**.   
Thank you for pointing it out. We have added the introduction of the meaning of variables $j$ and $k$ in Equation 4 to address confusion about the expression‘s significance. $j$ and $k$ stand for the keyframe ID, illustrating the optical flow mask warping process from the k-th to the j-th keyframe. All the modified text is highlighted in blue in the revised version.

**W2: Typos in Fig 1**.   
Thank you for pointing out the spelling mistake in Fig 1. It is a typo and has been corrected. We have reviewed this paper to reduce occurrences of spelling issues in the revised paper.

Reviewer comment (Experiments):
Only 2 datasets are used for testing.

Author response:
**Q1: Testing dataset**.   
Following traditional dynamic RGB-D SLAM methods [1,2,3], we assess the pose estimation results and reconstruction quality on the TUM RGB-D and BONN RGB-D datasets. As indicated in these papers, these datasets are sufficient to evaluate the performance of dynamic SLAM system. Additional evaluation results are presented in Appendix A.2, specifically in Tables 7 and 9 of the revised paper.

Reviewer comment (Writing):
Some acronyms are not provided; what is ATE? in the results section.

Author response:
**Q2: ATE meaning**.   
We have illustrated the meaning of ATE in the original paper. Please refer to the "Metric" part in the Experiment section for more details.

Reviewer comment (Theory):
It does not appear that the authors address the degenerate cases for computing the fundamental matrix, how does their method handle this?

Author response:
**Q4: Degenerate case for computing the fundamental matrix**.   
Actually, the issue of degeneracy in fundamental matrix computation is a fundamental problem in multi-view geometry and should be investigated as a standalone problem. This is not the contribution of our paper. We employ an engineering trick to address this problem, similar to approaches used in ORB-SLAM2 and ORB-SLAM3. We compute the fundamental matrix $\mathbf{F}$ and homography matrix $\mathbf{H}$, utilizing the RANSAC method and reprojection error to score the computed matrices $SF$ and $SH$. Then, we determine the ratio of scores with the equation $rh = SH/(SH+SF)$. If the value of $rh$ is bigger than 0.5, we choose the homography matrix for the pose estimation; otherwise, we choose the fundamental matrix.

Reviewer comment (Experiments):
With the comparisons, you should have compared to orb slam 3 and possibly DVO slam, an indirect and direct traditional method.

Author response:
**Q3: Comparison with ORB-SLAM3 and DVO SLAM**.   
As our method is a neural-based dynamic SLAM method, we selected the traditional dynamic SLAM methods, such as ReFusion [1], DynaSLAM [2], and MID-Fusion [3], as the baseline methods instead of the general static SLAM methods. ORB-SLAM3 [4] and DVO SLAM [5] rely on a static environment assumption and do not work robustly within a dynamic environment due to the inconsistent observation of geometry. 

DynaSLAM has showcased competitive pose estimation results in comparison to DVO-SLAM. We add the comparison results of DVO-SLAM in Table 2 in the revised paper, cited from DynaSLAM. Please refer to the original DynaSLAM paper for more details.

ORB-SLAM3 primarily introduces multiple map fusion and tightly integrated visual-inertial fusion techniques via MAP estimation. However, it still depends on the assumption of a static environment, making it similar to ORB-SLAM2 [6] in this regard. We have included the ORB-SLAM2 pose estimation results in Table 2, cited from ReFusion. To address your concerns, we also evaluated the results of ORB-SLAM3 and included them in Table 10 in Appendix A.6.

Our comparison results with ORB-SLAM3 and DVO SLAM are as follows:

| ATE[cm]      | f3/wk_xyz | f3/wk_half|  f3/wk_static |  f3/st_half| Avg.
| :---        |    :----:   |   :----:   |     :----:   | :----:   |  :----:   | 
|  DVO SLAM      |  59.7   |     52.9   |   21.2  | 6.2 |  35.0 |
|  ORB-SLAM3    |   28.1  | 30.5   | 2.1  | 2.6 | 15.9 |
|  Ours | 8.3  | 5.6 | 1.7  | 4.4 | 5.0 |

Reviewer comment (Reproducibility):
There is no mention of computation times, can this run in real time and what type of computing resources are required for such. With the computation of a radiance field and the extra processes, it is hard to see this functioning in real time.

Author response:
**Q5: Computing resources and Computation times analysis**.   
We have reported the time consumption of the tracking and mapping process in Section 4.4. Please refer to Table 5 in the original paper. Our system outperforms E-SLAM [7] in terms of time consumption and maintains a comparable level with Co-SLAM [8]. Therefore, our system can run in real-time like Co-SLAM, which is reported as a real-time SLAM method in paper [8]. In terms of computing resources, we run our RoDyn-SLAM on an RTX 3090Ti GPU, taking roughly 4GB of memory in total (please see "Implementation detail" part in the Experiment section).