Paper: EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision (ICLR-2024)

Reviewer comment (Novelty):
The authors claim “no prior works in this field have explicitly modeled temporal correspondence for dynamic objects”, which is wrong. Some related works are missing here [A1-3].

Author response:
1. **Missing related work**. We are grateful to the reviewer for highlighting additional relevant literature.  We have included a discussion of these related works in the revised version of our paper. Additionally, we plan to develop a detailed comparison section in the main text of our camera-ready paper to provide a more thorough analysis of how our work relates to these works.

Reviewer comment (Experiments):
For the results in Table 1 and Table 2, it is unclear whether visual feature distillation is being used.

Author response:
- 1. **Feature distillation in Tables 1 & 2**: The feature head is disabled, as stated at the end of the “setup” paragraph in Section 4.1.

Reviewer comment (Presentation):
It would be helpful to show the number of parameters in Table 1.

Author response:
- 2. **Number of parameters**. We have included the number of parameters and training times of different methods in Appendix A.3 for your reference. Specifically, StreetSurf employs a near-field hash encoder (32M) and a far-field encoder (16M). Our approach has
a static hash encoder (30.55M parameters), with the optional inclusion of a dynamic hash encoder
(10.49M parameters) and a flow encoder (9.70M parameters). It’s important to note that the number of parameters in both implicit and hybrid models does not directly correlate with their performance. For instance, as illustrated in Figure 2 of [1], an MLP with 438k parameters can achieve better image fitting than a hybrid model with a total of 33.6M parameters (10k from MLP and 33.6M from the grids). Please refer to the revised paper for more discussions.

| Method         | # Parameters in MLPs | # Parameters in Grids | GPU Hours | # Iterations | Seconds / 1k Iterations | Batch Size |
|----------------|----------------------|-----------------------|-----------|--------------|-------------------------|------------|
| **Static:**    |                      |                       |           |              |                         |            |
| StreetSurf     | 0.055M               | 32M + 16M             | 1.26 hrs  | 12,500       | 362.88s                 | 8192       |
| Ours           | 0.065M               | 30.55M                | 0.58 hrs  | 25,000       | 83.52s                  | 8192       |
| **Dynamic:**   |                      |                       |           |              |                         |            |
| D2NeRF         | 3.03M                | 0                     | 9.68 hrs  | 100,000      | 348.48s                 | 4096       |
| HyperNeRF      | 1.32M                | 0                     | 7.32 hrs  | 100,000      | 263.52s                 | 4096       |
| Ours w/o flow  | 0.065M               | 30.55M + 10.49M       | 0.88 hrs  | 25,000       | 126.72s                 | 8192       |
| Ours w/t flow  | 0.065M               | 30.55M + 10.49M + 9.7M| 2.03 hrs  | 25,000       | 292.32s                 | 8192       |

Reviewer comment (Experiments):
The comparisons with baselines in Table 1 seem problematic. For HyperNeRF and D2NeRF, lidar points are projected to the image and an L2 loss is used for depth regularization, which is worse than directly regularizing on lidar rays due to projection error and possible occlusions. Although the authors claim that these baselines do not support lidar ray sampling, incorporating this functionality is not difficult. A simpler approach would be to apply the same depth regularization method as in the proposed framework. Additionally, StreetSuRF has its own normal supervision and lidar ray preprocessing, and simply disabling them and using the same approach as the proposed method does not seem correct.

Author response:
- 3. **Lidar Projection**. We tested a variant of EmerNeRF that uses projected LiDAR points onto the image plane for depth supervision. This modification only marginally affects performance, indicating it is not a key factor that makes our method outperform other baselines. The results are reported in Table C.2. in Appendix C. We also provide the numbers here for reference:

| Method | Scene Reconstruction: Full Image | Scene Reconstruction: Dynamic-Only | Novel View Synthesis: Full Image | Novel View Synthesis: Dynamic-Only |
| ------ | -------------------------------- | ---------------------------------- | ------------------------------- | ---------------------------------- |
|        | PSNR↑ / SSIM↑                    | PSNR↑ / SSIM↑                      | PSNR↑ / SSIM↑                   | PSNR↑ / SSIM↑                      |
| use projected depth     | 28.874 / 0.8097                 | 26.200 / 0.7275                   | 27.532 / 0.7884                 |  24.321 / 0.6642     |
| default | 28.873/ 0.8096                  | 26.223 / 0.7276                  | 27.547 /  0.7885                  | 24.323 /  0.6655  

- 4. **StreetSurf Results**. The reported PSNR in the StreetSurf paper for the same dataset split is 26.66, achieved with normal supervision and their processed LiDAR rays. Our re-run of their code under the same setting yielded a PSNR of 26.44. Our initial submission reported a PSNR of 26.15 when trained without normals and using our processed data. In contrast, our method achieves a PSNR of 28.54. We believe our reproduction of their results is faithful. The number in our paper has been updated to reflect the 26.66 PSNR as reported in the StreetSurf paper.

Reviewer comment (Experiments):
While not necessary, it would be good to see a comparison on the scene flow estimation task with a SOTA method [A4].

Author response:
- 5. **Comparison to FastNSFF**: We thank the reviewer for pointing out the recent baseline as a potential comparison for our work. However, due to the limited timeframe and an intensive CVPR effort during the rebuttal period, we were unfortunately unable to complete the comparison experiment with FastNSFF by the rebuttal deadline. We recognize the importance of this comparison and are committed to including it in the camera-ready version of our paper. We appreciate your understanding.

Reviewer comment (Experiments):
The main concern is that the core reason for the proposed method’s success is unclear. More extensive ablation analysis or experiments on other datasets, especially monocular ones used for dynamic view synthesis, would help. More reasonable comparisons with baselines are also expected. The missing literature should be incorporated and discussed accordingly.

Author response:
4. **What is the key factor to make EmerNeRF excel at flow estimation?**
TL;DR: The key to our success lies in proper depth supervision and temporal feature aggregation.
- **1. Proper Depth supervision**: Consistent with other NeRF applications in the AD domain, EmerNeRF relies on LiDAR data for accurate scene reconstruction. However, we argue that depth signals, commonly obtained through LiDAR sensors, are readily available in the AD context and are integral to various tasks. Given the widespread availability of LiDAR sensors and the absence of a direct "flow" sensor equivalent, the challenge of learning flows from camera images and LiDAR data emerges as a significant area of interest. We also want to highlight the novelty of our approach in this regard.

Additionally, we provide quantitative results to identify the importance of different forms of depth losses, which are presented in Appendix C. Please feel free to refer to there for detailed settings. We include the results to provide a holistic preview. Specifically, (b) Using an L2 depth loss results in reasonable flow estimations  (i.e., achieving satisfactory Acc5 and Acc10 results), but these
estimates come with a large angle error. (c) Applying a line-of-sight loss yields much better results in terms of angle errors. The line-of-sight loss encourages a concentrated density distribution along a ray towards its termination point—a peaky distribution. This strategy effectively resolves the ambiguity associated with accumulating numerous noisy flows in the 3D scene, as it focuses on the core contributors of these flows: the dynamic objects. (a) Combining these two depth losses provides the optimal performance.

| Settings                                                     | EPE3D (m) ↓ | Acc5(%) ↑ | Acc10(%) ↑ | θ (rad) ↓ |
|--------------------------------------------------------------|-------------|-----------|------------|-----------|
| Combined L2 and line-of-sight loss for depth; feature-ensemble 
| (a) Ours                                             | **0.0130**      | 0.9272    | **0.9599**     | **0.5175**    |
| Depth supervision:                                          |             |           |            |           |
| (b) L2 loss only                                             | 0.0367      | 0.9077    | 0.9211     | 1.1207    |
| (c) Line-of-sight-loss only                                  | 0.0136      | **0.9296**    | 0.9574     | 0.5996    |

Reviewer comment (Experiments):
Ablation analysis is lacking - table C1 addresses some of the of the modeling decisions but there are more aspects I would have loved to see analyzed.

Author response:
**Ablation analysis**. We appreciate your emphasis on the importance of a detailed ablation analysis. In response, we have expanded this section in the appendix to include a more extensive range of studies. These studies are conducted across 5 sequences randomly selected from the Dynamic32 split, specifically for ablation purposes. The additional ablations include a variety of configurations, such as head ablation, depth loss variations, feature field ablation, and flow ablation studies. These extended analyses offer more insights into the impact of each component within our model. Due to the extensiveness of the results, we refer the reviewer to Appendix C in the revised paper for details.

Reviewer comment (Experiments):
There is a lot of focus on DINO/v2 features and their related issues - I am wondering if other SSL methods suffer from similar PE issues (specifically ones that handle motion such VideoMAE / Siamese MAE etc.)

Author response:
**Discussion regarding ViT features**.
This is indeed another great and insightful question, which closely aligns with our current research focus. In fact, we have an immediate follow-up work under review of CVPR where we discovered that these artifacts exist in nearly all Vision Transformers. We traced their origins to the positional embedding added to the input tokens and proposed a method to address this issue across all ViTs. We are also curious about whether 3D video ViTs (that explicitly deal with video data, like VideoMAE / Siamese MAE) exhibit a similar issue. However, our study on Video ViTs is not yet complete, as how to lift video features to 4D remains unexplored within the NeRF community. We are actively pursuing this line of research and are eager to share our findings on our anonymous website as soon as new updates are available. Although these updates might be beyond the scope of this rebuttal, we invite the reviewer and interested readers to check our anonymous website for new visualizations and findings, which we plan to post soon. We seriously consider your question on 3D video ViTs and are more than glad to explore this direction further!

Reviewer comment (Experiments):
How much dependance is there on LiDAR data? would this method work without direct depth supervision signals?

Author response:
**Addressing specific questions:**
1. **Dependence on LiDAR data**: Our method, in line with other NeRF approaches for autonomous driving (AD), relies on LiDAR data to reconstruct high-quality geometry. 

Despite these two mentioned aspects are beyond the scope of this paper, we acknowledge their importance for both research and real-world applications. Recognizing the limitations they may pose, we aim to improve these aspects in future work by exploring ways to reduce or eliminate the dependence on LiDAR and ground truth camera parameters.  

We believe our work will provide an excellent foundation for future research endeavors aimed at tackling these challenges, thanks to the open-sourced nature of our implementation, the comprehensive nature of our task setups, and the inclusion of a diverse range of challenging driving sequences. We anticipate that these elements will greatly facilitate and inspire subsequent studies in this field.

Reviewer comment (Experiments):
It's not written explicitly in the paper - does the data include ground truth camera parameters? (extrinsic and intrinsic, I imagine the answer is yes)

Author response:
**Addressing specific questions:**
2. **Ground truth camera parameters**. Yes, we do use the ground truth camera parameters that are readily available in all AD datasets. 

Despite these two mentioned aspects are beyond the scope of this paper, we acknowledge their importance for both research and real-world applications. Recognizing the limitations they may pose, we aim to improve these aspects in future work by exploring ways to reduce or eliminate the dependence on LiDAR and ground truth camera parameters.  

We believe our work will provide an excellent foundation for future research endeavors aimed at tackling these challenges, thanks to the open-sourced nature of our implementation, the comprehensive nature of our task setups, and the inclusion of a diverse range of challenging driving sequences. We anticipate that these elements will greatly facilitate and inspire subsequent studies in this field.

Reviewer comment (Experiments):
The dynamics regularization term is a bit simplistic as it is a simple minimization of total density for dynamic elements - have you tried other regularization methods? say assuming sparsity and so on?

Author response:
**Addressing specific questions:**  
3. **Other regularization terms**: At the beginning of this project, we experimented with several sparsity regularization terms and converged to the current form. We commit to including this ablation study in the final version.

Reviewer comment (Novelty):
Novelty. Many design components of EMerNeRF have been proposed in previous work, including separated static and dynamic fields [2][3], sky head [1], shadow head [2], flow field [4]. The paper lacks a detailed discussion highlighting the differences compared to these previous works.

Author response:
**Difference from [1-4].** In addition, we acknowledge the contribution of [1-4] and included all these works in our initial submission. We use similar sky head [1] and shadow head [2] in this paper. However, our design of static, dynamic, and flow field is novel in the sense that no prior work successfully learns a decomposition of these three fields completely through self-supervision (D2NeRF only learns static-dynamic decomposition without flow modeling). Moreover, our method provides a simple yet effective pipeline to integrate all these individual contributions to work synergistically.

Reviewer comment (Experiments):
Generalizability. The proposed method lacks verification in existing dynamic scene datasets used in baselines such as Nerfies [5], HyperNeRF [6]. Those datasets contain more complex deformations and a significant proportion of dynamic components. It is necessary to evaluate the robustness and understand the limitations of the proposed design modules, including dynamic density regularization and self-supervised flow field.

Author response:
**New dataset and Baselines.** We acknowledge the concern regarding the generalizability of EmerNeRF. We consider the application of our framework to other dynamic-NeRF datasets as an orthogonal contribution, which lies beyond the scope of this particular study. 
Yet, we still would like to include their results. 
Unfortunately, due to the limited timeframe and an intensive CVPR effort during the rebuttal period, we are unable to finish all suggested experiments on a new dataset and re-implementation of other methods, we are committed to including these results in the anonymous website in the coming weeks and add them into the camera-ready version of our paper. We hope reviewers can consider these results on the website during the discussion period should they be of interest. We also welcome further suggestions and feedback to make this paper stronger. We sincerely thank the reviewer’s efforts in helping get this submission into a better shape! We also thank the reviewer for pointing out the recent baseline as a potential comparison for our work. We have included a discussion of this paper in the related work section, and we commit to incorporating this baseline in the camera-ready version of our paper.

Reviewer comment (Theory):
Ambiguity of equation 6. The meaning and formulation of the expectation of the dynamic density remain unclear.

Author response:
**Ambiguity in Equation 6**, We apologize for the confusion caused by the presentation of Equation 6. This equation represents a regularization term designed to minimize the expectation of dynamic density, i.e., we encourage dynamic density to be as small as possible and prompt the dynamic field to produce density values only as needed. We have revised the paper to include a clearer description and formulation of this aspect.

Reviewer comment (Experiments):
Baseline. The quantitative evaluation lacks a more recent baseline [7].

Author response:
**New dataset and Baselines.** We acknowledge the concern regarding the generalizability of EmerNeRF. We consider the application of our framework to other dynamic-NeRF datasets as an orthogonal contribution, which lies beyond the scope of this particular study. 
Yet, we still would like to include their results. 
Unfortunately, due to the limited timeframe and an intensive CVPR effort during the rebuttal period, we are unable to finish all suggested experiments on a new dataset and re-implementation of other methods, we are committed to including these results in the anonymous website in the coming weeks and add them into the camera-ready version of our paper. We hope reviewers can consider these results on the website during the discussion period should they be of interest. We also welcome further suggestions and feedback to make this paper stronger. We sincerely thank the reviewer’s efforts in helping get this submission into a better shape! We also thank the reviewer for pointing out the recent baseline as a potential comparison for our work. We have included a discussion of this paper in the related work section, and we commit to incorporating this baseline in the camera-ready version of our paper.