Paper: Semantic Flow: Learning Semantic Fields of Dynamic Scenes from Monocular Videos (ICLR-2024)

Reviewer comment (Experiments):
There are some baselines that would be reasonable to try that are missing from the paper. For instance, if we just render the rgb images with any NeRF method (e.g. MonoNeRF) and apply some video object segmentation algorithm (e.g. any top-performing method on the DAVIS dataset) or semantic segmentation method (trained on some dataset with overlapping labels), how well would that perform?

Author response:
Thank you for your question. During the rebuttal phrase, we test the semantic segmentation performance by applying the state-of-the-art tracking method DeAOT [1] and image segmentation method Mask R-CNN [2] to the rendered images from MonoNeRF [3]. The results are shown **in Table 2 and Figure 8**. We also discuss the performance of these methods in the **General Response**. DeAOT lacks semantic understanding of 3D scenes and hence predicts inconsistent semantic labels in different views, leading to a severe performance drop as shown in Table 2. As Mask R-CNN only learns semantic information from single images, Figure 8 shows that it predicts inconsistent semantic labels of the same instances in different frames.

In addition, we also compare more Dynamic NeRF baselines by manually adding semantic segmentation heads to these methods in the **General Response and Table 1 in the paper**.

Reviewer comment (Evaluation):
Since one of the applications mentioned in the paper is scene editing, i.e. removing some specific object, it is necessary to not just render semantics correctly but also rgb. There are no values provided for the standard novel view synthesis metrics (PSNR, SSIM, LPIPS) for rgb on the tested video sequences.

Author response:
Thank you for your question. We present the PSNR, SSIM and LPIPS scores of rendered RGB images **in the following table**. Our model could achieve slightly better RGB color rendering quality compared to MonoNeRF [3] and DynNeRF [4], which demonstrates that our model is able to conduct the instance-level scene editing application. We also add the results to **Section G.1 and Table 11 in the Appendix**. 

|PSNR / SSIM / LPIPS|  Jumping   | Skating  | Average |
|----|----|----|----|
|DynNeRF [4] + head|  21.91 / 0.6856 / 0.174    | 24.68 / 0.7866 / 0.175  |23.30 / 0.7361 / 0.176|
|MonoNeRF [3] + head| 22.41 / 0.7484 / 0.145  | **26.18** / 0.8739 / 0.115 |24.30 / 0.8112 / 0.130|
|Semantic Flow | **22.86** / **0.7658** / **0.138**  | 25.75 / **0.8774** / **0.113** | **24.31** / **0.8216** / **0.126**|

Reviewer comment (Evaluation):
It would have been interesting to somehow visualise or discuss the flow fields. Since the objective is semantic rendering rather than colour rendering it is not clear if we need the scene flow to map to the same specific part of an object, or if it is sufficient or even beneficial to just map to anywhere within the same object. For instance, the consistency loss L_consis only enforces that points along flow trajectories should have the same label.

Author response:
**In Section G.2 and Figure 6 in the Appendix**, we show the visualization of the rendered images, estimated flow fields, predicted semantics and the point correspondence of dynamical foreground in 3 consecutive frames. We also discuss these results **in the General Response**. We agree that we do not force the flows to map the same parts of an object moving in time. However, Figure 6 shows that our model is able to predict flows that map similar parts of the dynamic foreground.

Reviewer comment (Novelty):
Missing related work: “Panoptic neural fields: A semantic object-aware neural scene representation” (CVPR 2022) also considers novel view synthesis for semantic segmentation from a video, although their method is limited to non-deformable dynamic objects.

Author response:
* *Missing related work*. We have inserted the Panoptic Neural Field [5] into **the Related Work section in the paper, page 3**.

Reviewer comment (Evaluation):
Page 5: Does ground truth flow mean optical flow estimated from RAFT? If that is the case it should not be called ground truth.

Author response:
* *Ground truth flow*. Thank you for your suggestion. we have corrected "the ground truth flows" to "the flows generated from the pretrained model".

Reviewer comment (Writing):
Page 5: The closing parenthesis in eq. (7) is probably incorrect.

Author response:
* *Incorrect parenthesis*. Thanks. We have revised the closing parenthesis of $\boldsymbol{F}_{dy}(I;{\boldsymbol{\Gamma}})$ near the equation (7) in the paper.

Reviewer comment (Writing):
Page 8: “Boundray” typo.

Author response:
* *"Boundray" typo.* Thank you. We have revised "boundray" to "boundary" in page 8.

Reviewer comment (Reproducibility):
When training for semantic completion or tracking, only a subset of the frames are used for semantic supervision. Is the same true for RGB supervision or are all frames used for that?

Author response:
* *RGB frames in semantic completion and tracking settings.* We follow MonoNeRF [3] and use the same subset of frames for RGB supervision.

Reviewer comment (Presentation):
In Fig. 3, what are the indices of the frames that are shown? How far are they from the frames with semantic labels?

Author response:
* *Indices of the frames in Figure 3*. In the completion setting, Figure 3 shows the semantic prediction results from the frame #8, which are 2-frame far from the frame #10 with the semantic labels. For the dynamic scene tracking setting, Figure 3 shows the results from the frame #10, which are 7-frame far from the frame #3 with the semantic labels.

Reviewer comment (Reproducibility):
For the DynNeRF and MonoNeRF baselines, what exactly is the input to the semantic heads that are learned?

Author response:
* *The input to the semantic heads.* For the DynNeRF [4] and MonoNeRF [3] methods, the input is the feature vector with 256 channels obtained from a 8-layer MLP with ReLU activations.

Reviewer comment (Reproducibility):
Paper leverages optical flow output as one of the intermediate steps, but fails to discuss its shortcomings and how exactly do they handle occlusion and disocclusion related to both dynamic and static regions of the frame.

Author response:
To evaluate the performance of our model in the occlusion and disocclusion situations, as shown **in Section G.4 and Figure 7 in Appendix**, we add an occlusion region in both the static background and dynamic foreground in the frame #6, and remove the occlusion in the frames #5 and #7. Concretely, we manually occludes the both RGB colors and semantic labels of the region in the frame #6. As Figure 7 shows, the optical flow estimation method fails to estimate the object movement due to the occlusion. We train the model with the occluded image, wrong optical flow maps and semantic labels. The occlusion may provide wrong information to train the semantics, which leads to the incorrect semantic prediction in the occlusion part. However, since our model learns the semantics from flows, it successfully predicts the semantics in disocclusion parts by incorporating correct information among the flows in non-occluded regions.

Reviewer comment (Experiments):
For the most part of the paper, the authors only compare with two dynamic scene-based works, Considering other related works in dynamic scene reconstruction, Would be great to see comparative baseline results, with a few more of these models with semantic head.

Author response:
Thank you for your suggestion. We have tested other methods related to dynamic scene reconstruction **in Table 1 in the paper** and have a discussion about the performance of these methods in the **General Response**. Besides, we also conduct comparisons with the state-of-the-art video tracking method and the image segmentation method **in Table 2 and Figure 8 in the paper and Appendix.**

Reviewer comment (Evaluation):
Could Authors share some of the shortcomings (a few qualitative results) which may be due to imperfect flow prediction, which results in bad performance during inference?

Author response:
Thank you for your suggestion. We present the shortcoming of our model **in Figure 10 in the Appendix**. Due to the imperfect predictions of flows in the tracking setting, our model fails to estimate precise semantic labels of dynamic parts.

Reviewer comment (Reproducibility):
In Section 3.4: while calculating Semantic Consistency Constraint; Do we generate some sort of valid mask here to enforce the semantic consistency or is it done for all pixels, irrespective of occlusion or uncertainty?

Author response:
We conduct the constraint for all pixels. We visualize the pixel correspondence in consecutive frames **in Section G.2 and Figure 6 in the paper** and discuss the correspondence **in the General Response**.  Figure 6 shows that our model could generate the flows that map the similar parts of the dynamic foreground moving across time. We also discuss the occlusion problem **in Section G.4** and the uncertainty of noisy flows **in Table 13 and Section G.3 in the Appendix**.

Reviewer comment (Experiments):
Outdated baselines. The paper compares their method with a couple of baselines (DynNeRF and MonoNeRF) but those are a bit limited. There are many other baselines for the dynamic NeRF task such as D-NeRF, RoDynRF, NSFF (Neural Scene Flow Field), etc. It would have been great if the paper provided accuracy on more baseline methods to make the comparison much fairer.

Author response:
# Outdated baselines
Thank you for your suggestion. We test the performance of NeRF [1] with time, D-NeRF [2], RoDynRF [3], NSFF [4] and discuss these methods **in Table 1 in the paper and in the General Response**. Besides, we also conduct comparisons with the state-of-the-art video tracking method and the image segmentation method **in the General Response** and **in Table 2 in the paper and Figure 8 in the Appendix.**

Reviewer comment (Presentation):
A bit difficult to follow the equations (from Eq. (4) to Eq. (8)). I am wondering if it's possible to put the mathematic notation from Eq. (4) to Eq. (8) into Fig. 2 for better understanding.

Author response:
# Difficult to follow the equations 4-8
Thank you for your suggestion. We have revised **Figure 2 in the paper** to put the mathematic notations from equations 4-8 into the figure.

Reviewer comment (Presentation):
Clarity. Some parts of the paper have lack of clarity and make it hard to understand clearly. What is the meaning of '25%/50% semantic labels' in Fig. 3? I wonder if the paper can provide more details in the figure captions. How are the 25%/50% determined?

Author response:
# Clarity
Thank you for your suggestion. In Figure 3, "25% semantic labels" in the tracking setting denotes that we use the frames #1, #2 and #3 to train the model and conduct semantic view synthesis on the frames #4-12. "25%" is determined by choosing 3 out of 12 frames to train the models in a scene. "50% semantic labels" in the completion setting denotes that we select the frames #1-3 and #10-12 to train the model and render semantic views in frames frames #4-9. "50%" is determined by choosing 6 out of 12 frames to train the models. We have added these details to the caption **in Figure 3 in the paper**.

Reviewer comment (Evaluation):
Marginal accuracy improvement in Fig. 4. The choice of low displacement seems not so critical for the accuracy gain. Maybe it would be good to have a justification or discussion on the result.

Author response:
# Marginal accuracy improvement
Thank you for your question. We agree that the results in Table 4 cannot distinguish $\boldsymbol{\Gamma}$ \& $\Delta \boldsymbol{\Gamma}$, $\Delta \boldsymbol{\Gamma}$ and $\boldsymbol{\Gamma}$. However, we demonstrate that flow displacements contribute to the semantic prediction performance according to the ablation study in Table 6 in the paper and our Semantic Flow has robustness with different flow displacements according to the Table 4. **In Figure 9 in the Appendix**, we visualize the semantic predictions by using $\boldsymbol{\Gamma}$ \& $\Delta \boldsymbol{\Gamma}$, $\Delta \boldsymbol{\Gamma}$ and $\boldsymbol{\Gamma}$. Although Table 4 shows quantitatively comparable results by using different displacements, Figure 9 presents that the semantic prediction of using $\boldsymbol{\Gamma}$ \& $\Delta \boldsymbol{\Gamma}$ is qualitatively better than using $\Delta \boldsymbol{\Gamma}$ or $\boldsymbol{\Gamma}$.

Reviewer comment (Experiments):
How much does the accuracy of the method depend on the off-the-shelf optical flow methods? Can it be critical?

Author response:
# Relationship between the performance of semantic predictions and optical flows
Thank you for your question. We test how much the performance of our model depends on the predicted flows from two perspectives: choosing different flow estimation methods and manually adding noise to the flow maps. 

Firstly, we choose RAFT [1] and FlowNet [2] to generate the optical maps and test the performance in the setting of semantic learning from multiple scenes. The results are presented **in the following table** and **in Table 12 in the Appendix**. According to the RAFT paper, RAFT outperforms FlowNet by a large margin in various optical flow estimation tasks. Therefore, the results show that there is about 10% performance drop in mIOU matrix by using FlowNet when jointly optimizing Jumping and Skating scenes, where the dynamic foregrounds are drastically changing and FlowNet fails to predict the correct flows. On the other hand, our model reaches comparable results when jointly optimizing Balloon1 and Balloon2 scenes where the foreground variations are relatively small and FlowNet successfuly predicts the correct flows of the foregrounds.

| flow estimation method| year  | Balloon1      | Balloon2      | Jumping       | Skating       |
|------------|------------|---------------|---------------|---------------|---------------|
| RAFT [1]   | 2020| 0.919 / 0.844 | 0.967 / 0.839 | 0.936 / 0.733 | 0.970 / 0.716 |
| FlowNet [2] |2015  | 0.926 / 0.855 | 0.974 / 0.766 | 0.921 / 0.636 | 0.965 / 0.591 |

In addition, we manually add different percentages of noise to the predicted optical flows and test the performance of our model with noisy optical flows. The details are stated **in the Section G.3 in the Appendix** and the results are presented **in the following table** and listed **in Table 13 in the Appendix**. Our model could achieve comparable performance by using the optical flows with small percentages of noise (< 5%). Adding more than 10% noise to the predicted flows leads to a performance decrease in mIOU matrix, which is mainly because our model could not build accurate flow fields from noisy optical flows and hence learns inaccurate semantics from imprecise flows.

| noise scale  | Balloon1      | Balloon2      | Jumping       | Skating       |
|------------|---------------|---------------|---------------|---------------|
| w/o. noise   | 0.919 / 0.844 | 0.967 / 0.839 | 0.936 / 0.733 | 0.970 / 0.716 |
| 1% noise   | 0.923 / 0.844 | 0.963 / 0.820 | 0.937 / 0.715 | 0.970 / 0.707 |
| 5% noise   | 0.924 / 0.808 | 0.964 / 0.813 | 0.937 / 0.656 | 0.969 / 0.714 |
| 10 % noise | 0.922 / 0.777 | 0.964 / 0.666 | 0.936 / 0.690 | 0.969 / 0.670 |