Paper: H2O-SDF: Two-phase Learning for 3D Indoor Reconstruction using Object Surface Fields (ICLR-2024)

Reviewer comment (Theory):
Clarification on OSF. Despite the good illustration of OSF as mentioned above, extra clarification is urgently needed to explain the motivation of the mathematical form of the 3D OSF loss (Equ. 2), and details in Fig. 4. Specifically, despite Fig. 4 explains how the gradients of the 3 loss drives SDF to form a zero points around surfaces, the paper does not provide intuitive explanation on (a) why the various terms of the loss in Equ. 2 are designed as they are, (b) how \gamma controls the steepness of the function, how it matters and how $\gamma$ is picked (better with illustrations similar to Fig. 4). Additionally, it is not clear that, between Fig. 4 (a) and (c), why different d(x) lead to identical $\sigma_\gamma(x)$. Without clarifying the issues it is difficult to understand why OSF and the losses are designed the way they are, despite being proven effective.

Author response:
Acknowledging the reviewer's feedback, we have prepared in the common response section a more detailed explanations of OSF and its corresponding loss functions.  

Addressing the reviewer's specific questions, 

(a) Equation 2 incorporates a loss term with dual components: one targeting the room layout and another for objects with intricate details. The room layout component tends to converge quickly due to a "smoothness bias". The object component, rich in high-frequency details, evolves differently. The design rationale behind Equation 2 is elaborated in the subsequent paragraphs following its introduction. The object region's loss term initially appears as depicted in Figure 4 (a). However, as training progresses, our Object Surface Field (OSF) evolves into the form shown in Figure 4 (e). Section 3.2, 'Mutual Induction of OSF and SDF', mathematically expounds this transition, focusing on the influence of the gradients of 3d_osf_loss on both OSF and the Signed Distance Function (SDF).

(b) The 3d_osf_loss in our framework is engineered to modulate the OSF from 0 to 1, correlating with the SDF's transition from positive to negative values. The parameter gamma governs the rate at which OSF's value alters. Through experimental evaluations with varying gamma values, we observed that a lower gamma, indicating a gentler slope, enables the OSF to learn a more expansive region on the object. Conversely, a higher gamma, resulting in a steeper slope, restricts the OSF to a narrower region, potentially undermining its effectiveness in guiding the SDF. Based on these experiments, we have optimally selected a gamma value of 1.

(c) Figure 4 (a) illustrates the scenario in which a well-trained SDF directs the OSF to accurately learn the surface field, particularly in general cases such as bulky furniture. On the other hand, Figure 4 (c) shows a situation where the OSF is effectively trained, possibly with additional inputs like an object mask, but the SDF has not yet accurately represented thinner structures. These examples are intended to demonstrate the reciprocal influence of OSF and SDF in varied training contexts.

Acknowledging the reviewer's feedback, we will provide extra clarification to better explain the motivation of the mathematical form of the 3D OSF loss and other related concepts in the final paper submission.

Reviewer comment (Experiments):
Demonstration of applying the proposed OSF and losses to general SDF-like NeRF-based methods. The proposed OSF and losses should theoretically be applicable to all of the baseline methods as a simple drop-in, but somehow the paper decides to compare against its own vanilla baseline. Is it possible to apply to other existing methods to better showcase the general nature of the proposed method, and how effective will it be?

Author response:
The reviewer's observation is valid as our Object Surface Field (OSF) can be seamless integrated with NeuS-based methods. However, we opted to incorporate our OSF into our enhanced baseline model, particularly after the initial stage of Holistic Surface Learning. This design choice is a strategic improvement over traditional baseline models like NeuRIS. While NeuRIS implements a normal prior and selectively utilizes normals based on their reliability, our Holistic Surface Learning goes a step further. It dynamically adjusts the utilization of both color and normal priors, contingent on the confidence level of the normal information. Consequently, we applied our Object Surface Learning to the refined Signed Distance Function (SDF) emerging from Holistic Surface Learning, rather than directly integrating it into the basic baseline models. This methodological choice was aimed at leveraging the advanced features of our enhanced model for more effective learning and integration of the OSF.

Reviewer comment (Experiments):
Limited scenes to evaluate. The main evaluation is done on ScanNet, with limited qualitative results on Replica. However, ScanNet is known to have image quality issues. Why is the method not evaluated and compared on alternative datasets including Replica, Tanks & Temples, etc., as is done in other papers like MonoSDF? Without the additional evaluation, it is difficult to decide the generalization of the method across various indoor scenes.

Author response:
We agree with the reviewer's point. We couldn't not include our experimental results in the original submission due to page limit constraint but we now present additional experiments on the Replica dataset that include a comparision with MonoSDF as shown in the common commentions section. These experimental results indicate that our method is well generalizable across various indoor scenes.

Reviewer comment (Experiments):
Additional comparison. One less important thing to add is potential comparison with I^2-SDF. Despite I^2-SDF being based on additional geometry supervision signals, the goal is aligned with the proposed method, and it showcases similar improvement in fine detailed objects. It would be beneficial to add comparison to I^2-SDF to inspire the discussion on the optimal strategy to improve reconstruction of high frequency signals in indoor geometry.

Author response:
Our experimental findings address the reviewer's point well. The reviewer rightly points out that I^2-SDF aims to refine fine details in objects, albeit with the aid of additional supervision signals. I^2-SDF employs a "bubble loss", designed to minimize the Signed Distance Function (SDF) value to zero for 3D points back-projected from depth maps. This approach parallels the concept of View-aware SDF loss in Geo-NeuS, which similarly endeavors to drive the SDF value of points, derived from Multi-View Stereo (MVS), to zero.

We explored the direct application of the View-aware SDF loss to the SDF derived from the MVS point cloud. However, our findings revealed that our method, which imparts indirect supervision to the SDF via the Object Surface Field (OSF), yielded superior results. This improvement can be attributed to the inherent noise often found in MVS point clouds, which may not align with the ground truth. Directly forcing the SDF values based on these noisy points can inadvertently degrade the quality of the final mesh. Thus, our method, which leverages the OSF as a mediating layer for SDF supervision, proves to be more effective in enhancing mesh quality, especially in scenarios where the input data may be imperfect.

Reviewer comment (Experiments):
In the supplementary material, authors present that they apply the OSF-based Filtering during reconstruction. I am curious about where does the major improvement of OSF comes from, either the proposed osf loss or the filtering. Authors are expected to conduct ablation study about this to make the contribution more convincing.

Author response:
In the supplementary material, the Object Mesh Extraction via OSF-based Filtering serves merely as an example to demonstrate the potential applications of our trained OSF. Note that this filtering process is not a core component of our proposed method. Hence, we do not anticipate a notable enhancement in performance during reconstruction through OSF-based filtering. To avoid any confusion for the reader, we will explicitly clarify this distinction in the final version of the paper.

Reviewer comment (Presentation):
Minor: The presentation can be further improved, and there exists noticeable typos in the submission such as Table 2.

Author response:
We will enhance the readability of our paper by correcting all typos, including those in Table 2.* and elsewhere in the submission.

Reviewer comment (Reproducibility):
Running time: The paper does not provide specific details about the computational complexity or running time of the approach, for both training and inference. It only states that all experiments were conducted on a single NVIDIA RTX 3090Ti GPU.

Author response:
(Q2) *Running time/computational complexity of the proposed method. Please refer to point 2 under Weaknesses.*

(Q\_A2)

Below, we present the average training and inference times obtained from our experiments on the 12 ScanNet scenes:

|  Time                  | ManhattanSDF | NeuRIS | MonoSDF | H2O-SDF |
|--------------------|--------------|--------|---------|---------|
| Training (hours)    | 5.2     | 4.5     | 21.5    | 4.5     |
| Inference (seconds) | 30      | 21      | 35        | 21       |

We plan to include these results in the Appendix of our final paper submission.

Reviewer comment (Evaluation):
Running time/computational complexity of the proposed method. Please refer to point 2 under Weaknesses.

Author response:
(Q2) *Running time/computational complexity of the proposed method. Please refer to point 2 under Weaknesses.*

(Q\_A2)

Below, we present the average training and inference times obtained from our experiments on the 12 ScanNet scenes:

|  Time                  | ManhattanSDF | NeuRIS | MonoSDF | H2O-SDF |
|--------------------|--------------|--------|---------|---------|
| Training (hours)    | 5.2     | 4.5     | 21.5    | 4.5     |
| Inference (seconds) | 30      | 21      | 35        | 21       |

We plan to include these results in the Appendix of our final paper submission.

Reviewer comment (Writing):
2D object surface loss. Could it be explained as the loss between the rendered object masks and the ground truth masks? It would be great to make it clear that the proposed method actually requires object annotations.

Author response:
Thank you for highlighting the lack of clarity in our explanation regarding the non-requirement of object annotations for the 2D object mask. In Section 3.2, Line 6 of "2D Object Surface Loss", we mentioned, "The 2D object mask is determined by a process which can be easily executed using a pre-trained model (Lambert et al., 2020)". However, we realize this could lead to confusion about whether annotations are necessary during the training of the segmentation model. In our final paper submission, we will explicitly state in Section 4.1, "Implementation Details", that "The 2D object mask is obtained using the MSeg segmentation method, employing its officially provided pre-trained model".

The concept of 2D object surface loss is essentially the disparity between the volume-rendered result of the object surface field and the 2D object label (pseudo-ground truth), which is derived from a pre-trained segmentation model, MSeg. This approach enables us to acquire pseudo-ground truth labels without using object annotations during the training process that relies on this loss.

Reviewer comment (Experiments):
It would be great to explain OSF with more details. Based on the description in the paper, it is quite similar to the absolute gradient field of the occupancy values. In particular, Eq. 3 actually enforces the OSF to have large values on the object defined by the 3D points. In addition, 3D points provide strong prior on the details of the shapes. It would be great to provide the ablations study of using the point cloud with MVS images or not.

Author response:
Please refer to the common response section a full detailed explanations of OSF and its corresponding loss functions. 

Quoting what is already mentioned in the common response section:

***"OSF essentially quantifies the likelihood (ranging from 0 to 1) of the object surface OSF(x) at each spatial point, enhancing the SDF’s ability to better capture fine geometric details and high-frequency surfaces, while ensuring a smooth room layout"***

It implies that there are clear differences between OSF and absolute gradient field. That is, the Object Surface Field is bounded between 0 and 1, indicating the probability of an object's surface presence, whereas the absolute gradient field's values can range from 0 to infinity. Furthermore, our Object Surface Field is not merely the gradient field of an imperfectly trained Signed Distance Function (SDF) using color and normal information. Instead, it serves as an independent representation, enhancing the SDF by indicating regions it has yet to encapsulate, thereby acting as a crucial 3D geometric cue.

Moreover, incorporating refine loss (ref_loss), which leverages the point cloud obtained from Multi-View Stereo (MVS), improves the model's capability in capturing intricate shapes. However, it is essential to emphasize that ref_loss plays a complementary role. Our model demonstrates state-of-the-art performance independently of the ref_loss component. The below experimental results support this claim.

| Methods                   | Acc↓   | Comp↓  | Prec↑  | Recall↑ | F-score↑ |
|---------------------------|--------|--------|--------|---------|----------|
| NeuS                      | 0.179  | 0.208  | 0.313  | 0.275   | 0.291    |
| ManhattanSDF              | 0.053  | 0.056  | 0.715  | 0.664   | 0.688    |
| NeuRIS                    | 0.052  | 0.050  | 0.713  | 0.677   | 0.690    |
| MonoSDF                   | 0.035  | 0.048  | 0.799  | 0.681   | 0.733    |
| HelixSurf                 | 0.038  | 0.044  | 0.786  | 0.727   | 0.755    |
|---------------------------|--------|--------|--------|---------|----------|
| H2O-SDF w/o ref_loss   | 0.034  | 0.038  | 0.824  | 0.757   | 0.789    |
| H2O-SDF                   | 0.032  | 0.0373 | 0.834  | 0.769   | 0.799    |

Reviewer comment (Experiments):
Experiments on ablations studies. It is not clear to the reviewer what model A, B, C are. It would be great to provide detailed explanations about those models.

Author response:
We admit that we employed some unclear notations. In the final paper submission, we plan to employ more enhanced notations in the ablation study part. That is, here is a revised summary:

In our ablation study, we compare various model configurations: Model A, Model B, Model C, and H2O-SDF. Model A, our baseline, is an adaptation of NeuS with the integration of normal loss. Model B represents a scenario where the model undergoes training solely through our first stage, Holistic Surface Learning (NeuS + Normal + re-weighting scheme), right up to the final epoch, but does not include the Object Surface Learning phase. Model C and H2O-SDF, on the other hand, adopt a two-phase learning process. In this approach, Object Surface Learning (OSL) is initiated after the completion of Holistic Surface Learning (HSL). Specifically, Model C explores the impact of conducting OSL without employing the OSF-guided sampling strategy (OGS).

Reviewer comment (Experiments):
For the second stage, it would be great to ablate whether all the losses have contributed to the final results. The proposed method adopts more accurate point cloud obtained from MVS images compared with monocular depth estimated from a single image. Those factors should be ablated to demonstrate the performance benefits from OSF and the sampling strategy not from the prior data.

Author response:
We report our ablation study results for each loss used during Object Surface Learning (i.e. 2nd stage of our method). The notations used in the table are as follows. 

- HSL: Holistic Surface Learning, i.e., color and normal reweighting scheme. 
- 2D: 2D object surface loss.
- 3D: 3D object surface loss.
- ref: refinement loss.
- OGS: Object Surface Field guided sampling strategy.

| Model                                 | Acc↓  | Comp↓ | Prec↑ | Recall↑ | F-score↑ |
|---------------------------------------|-------|-------|-------|---------|----------|
| Neus + HSF                            | 0.041 | 0.041 | 0.802 | 0.751   | 0.776    |
| Neus + HSF + 2D                       | 0.040 | 0.041 | 0.807 | 0.755   | 0.779    |
| Neus + HSF + 2D + 3D                  | 0.040 | 0.041 | 0.809 | 0.760   | 0.784    |
| Neus + HSF + 2D + 3D + ref            | 0.039 | 0.041 | 0.817 | 0.762   | 0.787    |
| Neus + HSF + 2D + 3D + ref + OGS      | 0.037 | 0.039 | 0.830 | 0.773   | 0.800    |

Considering the page limit constraint, we will include this table in the Appendix of our final paper submission to clearly demonstrate the effectiveness of each loss.

Reviewer comment (Experiments):
It would be great to elaborate more on the insight of OSF and also compared with existing density functions parameterised with the SDF values.

Author response:
We thank the reviewer for this suggestion. In the final paper submission, to clearly show the insight of OSF, we will include the following explanation:
"It is revealed through several works that models tend to focus on low frequency regions during training, known as "smoothness bias." Therefore, we introduce the Object Surface Field (OSF), a novel representation that delivers three-dimensional spatial information to the model. This concept is interesting as it extends beyond existing 2D priors, functioning as a 3D geometry cue that delivers 3D spatial information to the SDF."

Additionally, we hope that the answer (W_A2) to Weakness 2, which provided us valuable insights, will address the relation between SDF and OSF.

Reviewer comment (Experiments):
The ablations studies are missing. Please demonstrate all the losses introduced in stage two all contributes to the improvement of the reconstruction. In addition the proposed method leverages the point cloud obtained from MVS. It would be great to show how these priors can influence the final performance.

Author response:
We report the ablation study results for each loss used in stage two, Object Surface Learning, in our answer to Weakness 4 (W_A4). In addition, in addressing Weakness2 (W_A2), we include the ablation study results for ref_loss, which utilizes the MVS point cloud.