Paper: 3D Reconstruction with Generalizable Neural Fields using Scene Priors (ICLR-2024)

Reviewer comment (Experiments):
The model is not scale/rotation/translation invariant. I think the main reason is the use of point position as input to the decoder, which means if the coordinates system is changed, the output of the decoder is also changed. Similarly, the surface normal or view directions should also be in the local coordinates system, otherwise the output will change if the scene was translated. I wonder how sensitive of the current model to random rotation/translation. Would be great to have an ablation study. A simple solution is just don't use point position as input to the network. I am also interested in how this performs.

Author response:
**A**: Thanks for pointing out the drawbacks of using point positions during the first stage of training. The point position is used in the many per-scene optimization approaches[1,2,3], we keep it in our per-scene optimization stage as well. To ensure the decoders can be shared in both prior models and per-scene optimization, we have to keep the point position as inputs during the prior learning stage. However, we have made some efforts to mitigate the effect of scale/rotation/translation invariant. For example, we zero-center the input point cloud first and scale it into a unit sphere before feature extraction and positional encoding during both training and inference to ensure the scale and translation invariant. We don’t need to ensure the rotation is invariant, as the view-dependent features are required.
If we ignore the per-scene optimization stage, the point position is not necessary. 

[1] Barron, Jonathan T., et al. "Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[2] Wang, Peng, et al. "Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction." arXiv preprint arXiv:2106.10689 (2021).

[3] Li, Zhaoshuo, et al. "Neuralangelo: High-Fidelity Neural Surface Reconstruction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Reviewer comment (Novelty):
Would be great to cite and discuss [1] as it's very related to the paper. The paper already included many baselines so this is not a minus point.

Author response:
**A**: Thanks for mentioning this relevant paper, we have cited it in the revised paper. Additionally, **we add a comparison with [1] in Table1**.

[1] Fast Monocular Scene Reconstruction with Global-Sparse Local-Dense Grids

Reviewer comment (Experiments):
One very simple baseline is missing: simply use TSDF fusion with the GT depth map for geometry reconstruction.

Author response:
**A**: In Table 2, we have shown the results of Bundle-Fusion, which is a better approach than TSDF fusion or RGB-D SLAM due to the advanced global pose alignment procedure. The better reconstruction results on 10 synthetic scenes demonstrate the superiority of our approach. In addition, we show the mesh reconstruction of a scene from ScanNet by employing the TSDF fusion under the same setting of ours-prior(the sparse view) in https://drive.google.com/file/d/1lJ54Zwo7OnJvdroCABtusKKYgjmGc-cq/view?usp=sharing. It can be observed that under the same setting of sparse views input, the TSDF fusion can only reconstruct a very coarse mesh even with the ground-truth depth.

Reviewer comment (Experiments):
In Table 3. most of the baseline are not using depth as input, might be good to make it clear.

Author response:
**A**: We first clarify that Go-Surf in Table 3 is trained with ground-truth depth maps and we will emphasize this point in the main paper. 
Most of the methods [1,2,3] using depth maps focus on mesh reconstruction rather than novel view synthesis.

[1] Yu, Zehao, et al. "Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction." Advances in neural information processing systems 35 (2022): 25018-25032.

[2] Azinović, Dejan, et al. "Neural rgb-d surface reconstruction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[3] Williams, Francis, et al. "Neural fields as learnable kernels for 3d reconstruction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

Reviewer comment (Experiments):
In the direct fusion part, points from different views are directly concatenated, will this change the distribution of point density and therefore change the K neighbouring points during inference?

Author response:
**A**: During the direct fusion part, we simply collect all surface points and their feature from each view together. In other words, the operation of KNN and PointConv have already been employed on each view independently, thus the distribution of point density will not be unchanged.  We use this concatenated surface feature and positions to construct the surface implicit representation that could represent the entire scene instead of a single view as described in Section 3.2

Reviewer comment (Writing):
Typo in the first paragraph "Although these results are encouraging, Although these results are encouraging".

Author response:
**A**: Thanks for pointing out this typo. We have fixed it in the revised paper.

Reviewer comment (Evaluation):
The authors claim that learning-based multi-view stereo methods, "their multi-view setting makes it less flexible to scale up and to broad applications". The reviewer is not sure evidence is shown that these methods do not scale up and are less broadly applicable, and suggests that precise pointers to the results or references would help.

Author response:
**Q**: Learning-based multi-view stereo methods, "their multi-view setting makes it less flexible to scale up and to broad applications". Again, I'm not sure evidence is shown that these methods do not scale up and are less broadly applicably. Perhaps precise pointers to the results or references would help.  

**A**: Analogous to how many more RGB images are available on the Internet versus videos, we believe that using single-view RGB-D images will follow a similar pattern versus RGB-D videos. This will allow us to scale up our prior model more easily with diverse RGB-D images in the future as more such content becomes available.  We note that our current framework can be naturally extended to multi-view inputs by integrating all multiple views in the volume space, without requiring re-training of the encoder or decoder.

For example, our NFPs can be trained using a collection of independent RGB-D images from random different scenes, they DO NOT have to be sequences, as against the muti-view stereos approaches. This brings significant benefits: (1). Analogous to how many more RGB images are available on the Internet versus videos, we believe that using single-view RGB-D images will follow a similar pattern versus RGB-D videos. (2). Individual RGB-D images will be much more divergent statistically than RGB-D videos, which contain redundancy between frames. This leads to better generalizability of our network. (3). Networks with single image input is much easier to implement and to train, compared to multi-view setting networks that require complex components, e.g., GRUs, to model the relations between frames.

Reviewer comment (Experiments):
The experiments are restricted to a single dataset (ScanNetV2). While it is certainly a good dataset to evaluate on, as it is based on real-world scenes, it would have been useful to see results on other recently used NVS datasets, even if synthetic.

Author response:
**Q**:”Restricted to a single dataset (ScanNetV2). While it's certainly a good dataset to evaluate on, as it is based on real-world scenes, it would have been useful to see results on other recently used NVS datasets even if synthetic.”  

**A**: Beyond ScanNet, we also evaluate our approach on 10 synthetic scenes proposed by NeuralRGBD as listed in Table 2 and also present a qualitative result of mesh reconstruction of a self-captured room as shown in Figure 10 in supplementary materials.

Reviewer comment (Experiments):
Novel view synthesis experiments are evaluated on a small number of scenes only, with potentially high variance of results.

Author response:
**Q**: “Novel view synthesis experiments are evaluated on a small number of scenes only, with potentially high variance of results.”  

**A**: For a fair comparison, we follow the same setting of NerfingMVS and NeRFfusion using 8 scenes from ScanNet for the evaluation of novel view synthesis. Besides these 8 scenes, we also show some qualitative results of colored mesh of other Scannet scenes reconstructed via our approach in Figure 9. Additionally, we further show a reconstruction of a self-captured room in Figure 10.

Reviewer comment (Presentation):
It is not clear what is the protocol for making a result bold in Table 1, making it difficult to quickly see what performs best on each section of rows.

Author response:
**Q**:”It's not clear what is the protocal for making a result bold in Table 1, making it diffucult to quickly see what performs best on each section of rows.”  

**A**: We highlight the best results column-wise, i.e., top-1 of each matric is in bold. Table 1 is split into three sections. In the first and second sections, we list the RGB- and RGB-D-based state-of-the-art, and the last section is our proposed method.

Reviewer comment (Reproducibility):
It is unclear how exactly importance sampling is used. The reviewer could not find a technical explanation for its use. In particular, questions include: how is it used for novel-view synthesis tasks? Is it only used during training of the networks? For evaluation, the understanding is that rendering of a novel view is done by uniform sampling of the ray.

Author response:
**Q** : How exactly is importance sampling used? I couldn't find a technical explanation for how it's used. In particular, how is it used for novel-view synthesis tasks? Is it only used during training of the networks? In other words, for evaluation, I understand rendering of a novel view is done by uniform sampling of the ray?  

**A**: We use the importance sampling proposed in NeuS, starting from 64 coarse samples (uniformly sampled from near to far), we iteratively add 16 samples each time based on weights computed with previously sampled points. We perform 3 steps of importance sampling during both training and inference. Additionally, we added 16 samples uniformly sampled from the region near to given ground-truth depth maps only during training.

Reviewer comment (Writing):
The term "generalizable" is used extensively, and it is unclear what it refers to in many cases (e.g., generalizable features, generalizable representations, and generalizable losses). The reviewer asks the authors to either reduce the use of the term or be more precise when using it.

Author response:
**Q**: “The term generalizable is used extensively and I'm not sure what they refer to in many cases (e.g. generalizable features, generalizable representations and generalizable losses). I would kindly ask the authors to either reduce the use of the term or be a bit more precise when using it.”  

**A**: Thanks for your suggestion, we will reduce some of “generalizable” and keep a consistent form for better understanding. Similar to Q1, 'generalizable' in this context means that a single learned network can be applied to numerous scenes. It also indicates that the features learned in the latent space effectively capture the common statistical characteristics of geometry and texture patterns prevalent in our visual world. Again, none of these can be obtained from per-scene optimization-based approaches.

Reviewer comment (Novelty):
Missing references on learning-based novel-view synthesis:  
- Trevithick, Alex, and Bo Yang. "Grf: Learning a general radiance field for 3D representation and rendering." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.  
- Yu, Alex, et al. "pixelnerf: Neural radiance fields from one or few images." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

Author response:
**Q**: Some missing references.  
**A**: Thanks for pointing out these missing references, we will add them to the related work section.

Reviewer comment (Novelty):
Missing reference on using GT depth for guiding training geometric and colour scene functions:  
- Stelzner, Karl, Kristian Kersting, and Adam R. Kosiorek. "Decomposing 3d scenes into objects via unsupervised volume segmentation." arXiv preprint arXiv:2104.01148 (2021).

Author response:
**Q**: Some missing references.  
**A**: Thanks for pointing out these missing references, we will add them to the related work section.

Reviewer comment (Novelty):
To me, the novelty of this paper is limited. The key design of the geometry prior module is similar to PointNeRF (also employs a distance-wise feature aggregation from 3D point clouds). Also, learning a geometric prior (SDF) then pruning to facilitate the texture field is applied in previous neural 3D reconstruction methods such as NeRFusion and SparseNeuS.

Author response:
We thank the reviewer for pointing out the relevant references, however, we respectfully disagree on the limited novelty on top of them. Here, we highlight the key differences:

First, Point-Nerf introduced a separately learned geomatric network to reconstruct the explicit point clouds. Once trained, the point cloud is fixed and detached from the succeeding learning process. This limits the performance as errors in the reconstructed point clouds are not directly optimizable through the gradient from the rendering loss or the per-scene optimization losses (they have resolved to point pruning and growing to fix it). Our approach fills the gap by joint learning of both geometry and texture representation, which is much simpler and more effective. Second, we note that both NeRFusion and SparseNeuS learned or optimized the geometry and texture field simultaneously. They either consider performing novel view synthesis(NeRFusion), where the geometry is not evaluated, or perform pure geometric reconstruction(SparseNeuS)

Also, these three mentioned approaches are based on muti-view stereos, while our approach can be trained using a collection of independent RGB-D images from random different scenes, which DO NOT have to be sequences. This bring significant benefits: (1). Analogous to how many more RGB images are available on the Internet versus videos, we believe that using single-view RGB-D images will follow a similar pattern versus RGB-D videos. (2). Individual RGB-D images will be much more divergent statistically than RGB-D videos, which contains redundancy between frames. This leads to better generalizablily of our network. (3). Networks with single image input is much easier to implement and to train, compared to multi-view setting networks that require a complex component, e.g., GRUs, to model the relations between frames.

Reviewer comment (Experiments):
The paper only conducts experiments on RGB-D sequences to demonstrate the generalizability. To me, the technical impact would be much higher if it also works well on RGB sequences, where obtaining precise geometry is challenging. For RGB-D sequences, some classic methods such as BundleFusion and COLMAP-MVS can achieve superior generalizability across different environments without any training.

Author response:
While our approach is potentially applicable for monocular depth maps (e.g., aligned via camera pose), in this paper, we focus on how to fully utilize the depth prior. We will leave the extension to RGB sequences in future work. 
For inputting RGB-D sequences, first, it is NOT true that BundleFusion and COLMAP-MVS do not require training, rather, they do need to perform per-scene optimization for reconstruction. We note that unlike us, neither of them explored any generalizable geometric/texture representation. Our approach fills this gap. 
Secondly, as evidenced in Table 2, our approach outperforms both approaches with only 3% number of frames.

Reviewer comment (Evaluation):
The advantage over Go-Surf is not convincing on both accuracy and speed.

Author response:
We’d like to point out that the main contribution of our approach is to propose a generalizable scene prior that enables fast, large-scale scene reconstruction. As demonstrated in Table 1, without any optimization, our prior network can achieve comparable performance with some per-scene optimization approaches. Therefore, we emphasize our approach facilitates two things Go-Surf doesn’t provide. 1. We have a prior network that can reconstruct scenes by merely inferencing, no optimization is needed. 2. Single-view novel view synthesis as shown in Figure 7 in supplementary material (useful for image input on mobile devices)

Also, our approach using per-scene optimization still outperforms Go-Surf in both aspects, specifically, the speedup is significant (e.g., use less than ½ of the time). And as demonstrated in Table 3 and Figure 4, Go-Surf focuses on geometric reconstruction, the NVS results are obviously less appealing.

Reviewer comment (Reproducibility):
Missing bits of context information - How much does it cost? While indicative timings and thorough implementation details (in supMat) are provided, information regarding the resource usage, model size and complexity are yet underdescribed. A comparative disclosure of such information covering the main experimental baselines that are considered would help the reader better assess its relative positioning throughout the typical criteria. Mentioning where the computation bottlenecks lie in terms of components would also be valuable in order to fully assess the practical usefulness of the proposed sequential pipeline, beyond rough timings (eg, Fig 3).

Author response:
**Q**: Missing bits of context information - How much does it cost?

**A**: The geometric and texture priors network are trained on 8 NVIDIA V100 GPUs for 2 days until convergence. The per-scene optimization step is trained and tested on a single NVIDIA V100 GPU. All baselines reported in our paper are tested using the same computational resources.  As presented in Table 1&6 of our paper, during inference, the prior network reconstructs the entire scene within 5 minutes and the per-scene optimization network initialized by the prior network achieves sota performance within 15 minutes. Our approach yields superior computational efficiency thanks to the generalization representation.

Reviewer comment (Experiments):
Comparative evaluation - Baselines and Benchmarks. While very recent work (eg, CVPR 2023) have been included in the setup, eg, HelixSurf (Liang et al.), there are a few missing players that currently shine by their absence. For example, the RGB based references:  
-- Li, Z., Müller, T., Evans, A., Taylor, R. H., Unberath, M., Liu, M. Y., & Lin, C. H. (2023). Neuralangelo: High-Fidelity Neural Surface Reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 8456-8465).  
-- Darmon, F., Bascle, B., Devaux, J. C., Monasse, P., & Aubry, M. (2022). Improving neural implicit surfaces geometry with patch warping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6260-6269).  
-- Zhang, J., Yao, Y., Li, S., Fang, T., McKinnon, D., Tsin, Y., & Quan, L. (2022). Critical regularizations for neural surface reconstruction in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6270-6279).  
-- Wang, Y., Skorokhodov, I., & Wonka, P. (2022). Hf-neus: Improved surface reconstruction using high-frequency details. Advances in Neural Information Processing Systems, 35, 1966-1978.  
Similarly, the DTU public benchmark could have been envisioned, albeit partially, just to compare other key papers from the state-of-the-art without having to re-run their public implementations. The same goes for the Tanks and Temples public benchmark as well. At least one additional benchmark would have been a reasonable addition. This would help better – and more thoroughly – assess the relative positioning of the proposed contribution, performance-wise.

Author response:
**Q**: Comparative evaluation - Baselines and Benchmarks.

**A**: Thanks for mentioning these baselines, we will cite them in our revised paper. As many of them focus on reconstructing a texture object rather than an indoor scene, we try our best to adapt them on ScanNet and we find that most of them don’t work at all. This is because most of them have difficulty in handling low-textured or less-observed regions which are common in indoor scenes as suggested by [1,2]. Among these methods, only Neuralangelo[3] could work without major modifications. Beyond Neuralangelo, we further include another baseline[4] 

| Method | Training Time(mins) | Acc↓ | Comp↓ | Prec↑ | Recall↑ | F-score↑ |
| --- | :---: | --- | --- | --- | --- | --- |
| Neuralangelo | 150 | 0.084 | 0.071 | 0.611 | 0.605 | 0.609 |
| FastMono | 30 | 0.042 | 0.056 | 0.751 | 0.678 | 0.710 |
|Ours-prior | 5  | 0.084 | 0.057 | 0.695 | 0.764 | 0.737 |
|Ours | 15 | 0.049 | 0.017 | 0.947 | 0.962 | 0.954 |

[1]Yu, Zehao, et al. "Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction." Advances in neural information processing systems 35 (2022): 25018-25032.

[2] Guo, Haoyu, et al. "Neural 3d scene reconstruction with the manhattan-world assumption." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[3] Li, Zhaoshuo, et al. "Neuralangelo: High-Fidelity Neural Surface Reconstruction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[4] Dong, Wei, et al. "Fast Monocular Scene Reconstruction with Global-Sparse Local-Dense Grids." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Reviewer comment (Experiments):
The dependency on depth maps is limiting since such data is not always available. This is also a drawback of MonoSDF and other works that use additional priors for scene reconstruction, beyond just RGB images.

Author response:
**Q**: “The dependency on depth maps is limiting since such data is not always available. This is also a drawback of MonoSDF and other works that use additional priors for scene reconstruction, beyond just RGB images”

**A**: We acknowledge that depth data is not universally available, however, it's important to note that RGBD settings are increasingly common, e.g., in applications in robotics, self-driving cars, augmented reality, etc., where depth sensors are often integrated. Our method uses depth data to create compact surface representations, rather than only as objectives like MonoNerf, offering a more efficient alternative to the usual volumetric approaches.

Reviewer comment (Theory):
The intuition behind the approximation of GT SDF values by observing depth values along a ray is unclear. This may result in erroneous signed distance predictions. An explanation of this is lacking. I am actually interested in this ablation experiment.

Author response:
**Q**: “The intuition behind the approximation of GT SDF values by observing depth values along a ray is unclear. This may result in erroneous signed distance predictions. An explanation of this is lacking. I am actually interested in this ablation experiment”

**A**: Using depth maps to approximate the SDF value has already been applied in previous work, e.g., [1,2], based on the fact that when only depth is provided, computing the exact distance of a location to the surface is infeasible. To resolve it, we have to compute the distance to a single, nearby surface point, which yields an approximated value that is equal or larger than the ground truth, i.e., an up-bound for the SDF value. Thus, to establish a tight bound, we choose to use the distance along the ray between the query point depth and the measured depth, as described in our paper.

[1] Ortiz, Joseph, et al. "isdf: Real-time neural signed distance fields for robot perception." arXiv preprint arXiv:2204.02296 (2022).
[2] Azinović, Dejan, et al. "Neural rgb-d surface reconstruction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

Reviewer comment (Experiments):
Results on images in the wild are missing. This will add value to the work.

Author response:
**Q**: “Results on images in the wild are missing. This will add value to the work”

**A**: Figure 10 in the supplementary material shows the reconstruction results of a self-captured room. It can be considered as the images in the wild

Reviewer comment (Experiments):
Ablation experiments are not thorough, in terms of the different components involved in Geometric and Texture Prior Networks.

Author response:
**Q**: Ablation experiments are not thorough, in terms of the different components involved in Geometric and Texture Prior Networks

**A**: In Sec. 4.3, we did show the ablation studies to evaluate the impact of Geometric and Texture prior networks, providing essential validation for their functionality. We note that loss functions such as $L_{eik}$ is commonly used and thoroughly analyzed in the previous literature[1,2,3]. Removing any loss in training the Geometric prior, such as $L_{depth}$ or $L_{sdf}$ will lead to inconvergence of the network; We conduct an ablation study w./w.o. the refined surface feature as described in the last paragraph of Page 4. The comparisons are listed below,

| Method | Acc↓ | Comp↓ | Prec↑ | Recall↑ | F-score↑ |
| --- | --- | --- | --- | --- | --- | 
| w./o. refine mlp | 0.089 | 0.062 | 0.675  | 0.728 | 0.707 |
| w. Refine mlp | 0.084 | 0.057 | 0.695 | 0.765 | 0.737 |

[1]Yu, Zehao, et al. "Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction." Advances in neural information processing systems 35 (2022): 25018-25032.

[2] Guo, Haoyu, et al. "Neural 3d scene reconstruction with the manhattan-world assumption." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[3] Wang, Peng, et al. "Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction." arXiv preprint arXiv:2106.10689 (2021).

Reviewer comment (Evaluation):
Discussion on limitations is lacking.

Author response:
**Q**:Discussion on limitations is lacking

**A**: The performance of the prior network could be further improved. Currently, we only utilize very shallow encoders, such as PointConv and ResNet50. Some advanced architectures can be utilized as well. Additionally, our prior models are only trained on ScanNetV2 with 1213 scenes. As mentioned in the paper, since the network takes in single images instead of videos, it is easy to scale up the training data. We expect the performance could be further improved with the upgrade of the network architecture and the increase of training data