Paper: ViFu: Visible Part Fusion for Multiple Scene Radiance Fields (ICLR-2024)

Reviewer comment (Experiments):
One major concern pertains to the practicality of the proposed setup. Reconstructing multiple radiance fields for different variants of a scene, assuming diverse object positions and poses, may be challenging in real-world applications. The implicit assumption of fixed illumination for different radiance fields can limit the practicality of the approach. If we can control the scene, we can reconstruct each object independently rather than using this setup. It would be beneficial to address this concern and discuss potential solutions or scenarios where this setup could be more applicable.

Author response:
> Please kindly justify the proposed setup is practical in real-world applications.

We have outlined some potential application scenarios for our settings in the **Real-world applications** section in the common responses.

W1 -> R2: - Regarding practicality, we believe that the examples mentioned in the common responses, such as indoor mobile robots or wearable visual AI, represent feasible practical applications of our method.

Reviewer comment (Theory):
The omission of modeling scene illumination and shadows is a notable limitation. Figure 4 shows that some reconstructed objects have darker or shadowed parts, which raises questions about the fidelity of the reconstructions. Considering illumination and shadows in the method is critical to enhance the paper's completeness.

Author response:
> The limitation of not modeling illumination and shadows.

We acknowledge that modeling illumination is beneficial for complex scenes. However, the core focus of this paper is the fusion strategy for given multiple NeRFs, and modeling illumination is a specific improvement for each NeRF reconstruction. Additionally, as our method is a general blending algorithm for neural fields, replacing NeRF with more advanced methods that consider lighting decomposition is a direct solution for more complex scenes. However, some failed experimental results also indicate the importance of lighting conditions for the final results. Therefore, we believe that eliminating the influence of lighting is a valuable future direction. Please also refer to the **Lighting conditions** section in common responses.

W2 -> R2: - Regarding lighting conditions, we created scenes with highly reflective foreground objects/background and validated our method under both unchanged and varying lighting conditions. Please kindly refer to Fig. 4 and the light condition section of the ablation study in our updated paper for details. While we agree that modeling illumination and shadows for complex lighting conditions or glossy objects may be a better solution and a promising direction, we believe that, as indicated by our experimental results, our current simple approach can still yield acceptable results in such challenging scenarios.

Reviewer comment (Novelty):
While the paper introduces a new setup, the technical novelty of some aspects, such as the segmentation and visibility parts, is limited. Traditional point cloud registration methods and straightforward visibility calculations may not sufficiently push the boundaries of the field.

Author response:
> Please justify the technical novelty of the method.

We believe the novelty of our method lies in the core of the proposed approach: the solution and insights into the fusion of multiple neural fields. In recent developments in the 3D field, neural fields have gradually become a mainstream representation alongside point clouds or meshes. When given many 3D assets, fusing them is crucial for practical applications, such as creating new scenes. For point clouds or meshes, their fusion is straightforward (e.g., combining each point cloud). However, for neural fields, due to their implicit nature, how to fuse them is non-trivial. We recognized this fundamental yet unexplored area and proposed a practical method to address it. Therefore, we believe that, despite the simplicity of our approach, it fills a gap in the important direction of neural field fusion, enhances its potential to become a better universal 3D representation, and provides valuable insights for researchers working with neural fields.

Reviewer comment (Evaluation):
The quality of results in real scenes appears to be a concern, with reconstructed objects being described as blurry. It is important to provide further analysis or insights into why this issue occurs and potential strategies for improving the quality of real scene reconstructions.

Author response:
> Results in the real-world scene is not good. 

We speculate that the occurrence of blurriness in real scenes may have two main reasons. Firstly, inaccuracies in camera pose prediction for real scenes may lead to inaccurate parts in the reconstructed NeRF, causing errors in the object alignment method based on extracted point clouds, resulting in a blurry fusion result. As discussed in the **Robustness** section of common responses, using alternative alignment methods might improve this aspect. Secondly, for real-world scenes, we model the entire surrounding environment using NeRF, not just the foreground objects and the table but also extending to the distant background. This results in the foreground objects having a relatively small proportion in the overall NeRF model. In our implementation, we use Instant-NGP to encode the entire scene, which could result in insufficient resolution to capture fine details of foreground objects during modeling, leading to blurry outcomes. For this aspect, we believe that improving the overall model resolution, or focusing on modeling the space near the foreground object, may lead to improvements.

Reviewer comment (Experiments):
Scalability. The paper suggests training a per-video representation and using visibility to compose them. This might not scale well in terms of speed and compute cost if there are many videos (needs MxN NeRFs). Since the geometry/appearance of each NeRF is assumed to be consistent across videos, it seems to make more sense to learn a single representation across all videos for each component (needs M NeRFs).

Author response:
Scalability. Since our method involves NeRF reconstruction for N captures, we indeed need to reconstruct N NeRFs, rather than MxN. For further discussion on the number of scans and required NeRFs, please refer to the **Baseline methods** section in the common responses.

Reviewer comment (Experiments):
There is no quantitative evaluation, comparison with the closest baseline, or ablation study. I think those are important for a paper.

Author response:
We agree with the reviewer that quantitative evaluation is crucial.
- **Evaluation**: As mentioned in the paper, finding a quantitative metric that accurately assesses the rendering results of the reconstructed objects is challenging since we aim for the color of the 360-degree objects to be close to the ambient color rather than the color under specific lighting conditions. However, regarding the accuracy of the reconstructed geometry, we agree with the reviewer that the 3D Chamfer distance is a reasonable metric.
- **Baseline**: As discussed in the **Baseline methods** section in the common responses, in our early experiments, we attempted to separate foreground objects using a segmentation model and individually reconstruct them. However, this process required a significant amount of manual labor for refinement to achieve satisfactory reconstruction results. Nevertheless, we acknowledge that although labor-intensive, using a straightforward method as a baseline is appropriate, and we plan to include a comparison with this approach if possible.
- **Ablation**: We appreciate the reviewer's suggestions. We will incorporate experiments comparing our proposed visibility-based fusion and naive fusion approaches, such as taking the average.

Reviewer comment (Evaluation):
Evaluation: how well can the background and objects be segmented (2D IoU) and reconstructed (PSNR / 3D Chamfer distance)?

Author response:
We agree with the reviewer that quantitative evaluation is crucial.
- **Evaluation**: As mentioned in the paper, finding a quantitative metric that accurately assesses the rendering results of the reconstructed objects is challenging since we aim for the color of the 360-degree objects to be close to the ambient color rather than the color under specific lighting conditions. However, regarding the accuracy of the reconstructed geometry, we agree with the reviewer that the 3D Chamfer distance is a reasonable metric.

Reviewer comment (Experiments):
Baseline: I think the shared NeRFs alternative I mentioned earlier is a valid baseline.

Author response:
- **Baseline**: As discussed in the **Baseline methods** section in the common responses, in our early experiments, we attempted to separate foreground objects using a segmentation model and individually reconstruct them. However, this process required a significant amount of manual labor for refinement to achieve satisfactory reconstruction results. Nevertheless, we acknowledge that although labor-intensive, using a straightforward method as a baseline is appropriate, and we plan to include a comparison with this approach if possible.

Reviewer comment (Experiments):
Ablation: How well does the method work without the proposed visibility composition? How does the selection strategy affect the result (max selection vs weighted)?

Author response:
- **Ablation**: We appreciate the reviewer's suggestions. We will incorporate experiments comparing our proposed visibility-based fusion and naive fusion approaches, such as taking the average.

Reviewer comment (Experiments):
The videos have different illumination conditions as mentioned in the limitation.

Author response:
We have incorporated new experiments on illumination conditions based on your feedback. We created scenes with highly reflective foreground objects/background and validated our method under both unchanged and varying lighting conditions. Please kindly refer to Fig. 4 and the light condition section of the ablation study in our updated paper for details.

Reviewer comment (Experiments):
The appearance of the background changes over time (e.g., adding a table cloth).

Author response:
For the issue of background changing over time, our current method indeed cannot address this, as the current segmentation algorithm assumes that the background remains unchanged across scenes to achieve foreground/background segmentation. However, we agree that this is an important point for practical applications, and improving the generalization of the segmentation step is a feasible direction.

Reviewer comment (Writing):
The term "scene" can be confusing. The underlying scene is the same across all videos, but the paper refers to them as different scenes. This makes the definition of visibility confusing. I think naming them video 1...N would be clearer.

Author response:
- We appreciate the reviewer's suggestions. We agree that changing "scene" to "video" or "object configurations" might be clearer.

Reviewer comment (Presentation):
In Fig 3 bottom right subfigure, should the points near the end of the ray be red, as they are not visible from the camera?

Author response:
- Yes, they should be in red. We will correct that.

Reviewer comment (Writing):
In Eq 4, it might be worth showing the exact equation that controls the peakedness (I assume it is similar to temperature-scaled softmax?).

Author response:
- Eq.4 is the specific formula for calculating the blending weight based on the computed visibility. What do you mean by the "exact equation"? Yes, it is similar to a temperature-scaled softmax.

Reviewer comment (Reproducibility):
In object clustering, do you need to predefined the number of objects.

Author response:
Yes, for robustness, our current method requires specifying the number of objects M. After obtaining the set of foreground point clouds through distance thresholding, we use DBSCAN to obtain multiple point cloud clusters. However, these clusters may include those consisting entirely of noise formed by a small number of points. Therefore, we select the top-M clusters based on the number of points contained in each cluster to obtain the point clouds of each foreground object.

Reviewer comment (Writing):
How to obtain a clean background is not very clear to me? Do you mean if a part is less occlusion, then it belongs to the background?

Author response:
In this work, we define the term “background” as the part that remains unchanged across different captures.. We assume that each part of the background (e.g., a part of the table surface) can be observed in at least one capture. For example, an area with no object occlusion in the first capture (no objects above it, no occlusion), but is occluded by objects in subsequent captures (occlusion), our algorithm selects its scene from the first capture as the background. For all areas, we determine the clean background by comparing visibilities, selecting the one with higher visibility.

Reviewer comment (Experiments):
It would have been better if there was more we could learn from the paper experimentally, including failure cases.

Author response:
Failure cases. We acknowledge the importance of failure cases. In our method, failure cases primarily arise from two aspects: (1) failure in object pose registration and (2) significant appearance discrepancies between different captures. We have added discussions on these two aspects in the section of **Robustness** and **Lightning conditions** in our common responses.

Reviewer comment (Novelty):
Perhaps consider citing FiG-NeRF: "separating foreground objects from their varying backgrounds" (instead of the same static background).

Author response:
We appreciate the reviewer's suggestions. We find its motivation similar to ours, and we will add it in the citation.

Reviewer comment (Theory):
What if the background isn't flat?

Author response:
There shouldn't be any issues. In our assumption, if a part remains unchanged in all scenes, we consider it as part of the background. For example, if a keyboard on a desk doesn't move in each capture, it is considered part of the background, and in this case the separated background will include both the desk and the keyboard. A flat background is not our assumption. (If time permits, we would like to append experimental results under non-flat background conditions.)

Reviewer comment (Experiments):
How robust is this to noise in point-cloud registration?

Author response:
Please refer to our discussion on **Robustness** in the common responses.