MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction
Abstract
In recent years, neural implicit surface reconstruction methods have become popular for multi-view 3D reconstruction.
In contrast to traditional multi-view stereo methods, these approaches tend to produce smoother and more complete reconstructions due to the inductive smoothness bias of neural networks.
State-of-the-art neural implicit methods allow for high-quality reconstructions of simple scenes from many input views.
Yet, their performance drops significantly for larger and more complex scenes and scenes captured from sparse viewpoints.
This is caused primarily by the inherent ambiguity in the RGB reconstruction loss that does not provide enough constraints, in particular in less-observed and textureless areas.
Motivated by recent advances in the area of monocular geometry prediction, we systematically explore the utility these cues provide for improving neural implicit surface reconstruction.
We demonstrate that depth and normal cues, predicted by general-purpose monocular estimators, significantly improve reconstruction quality and optimization time.
Further, we analyse and investigate multiple design choices for representing neural implicit surfaces, ranging from monolithic MLP models over single-grid to multi-resolution grid representations.
We observe that geometric monocular priors improve performance both for small-scale single-object as well as large-scale multi-object scenes, independent of the choice of representation.
Introduction
3D reconstruction from multiple RGB images is a fundamental problem in computer vision with various applications in robotics, graphics, animation, virtual reality, and more.
Recently, coordinatebased neural networks have emerged as a powerful tool for representing 3D geometry and appearance.
The key idea is to use compact, memory efficient multi-layer perceptrons (MLPs) to parameterize implicit shape representations such as occupancy or signed distance fields.
While early works [9,42, 50] relied on 3D supervision, several recent works [47,66,82] use differentiable surface rendering to reconstruct scenes from multi-view images.
At the same time, neural radiance fields (NeRFs)
[44] achieved impressive novel view synthesis results with volume rendering techniques.
[49, 76, 81] combine surface and volume rendering for the task of 3D reconstruction by expressing volume density as a function of the underlying 3D surface, which in turn improves scene geometry.
Current neural implicit-based surface reconstruction approaches achieve impressive reconstruction results for simple scenes with dense viewpoint sampling.
Yet, as shown in the first row of Fig. 1, they struggle in the presence of limited input views (DTU with 3 views) or for scenes that contain large textureless regions (walls in ScanNet or Tanks & Temples).
A key reason for this behavior is that these model are optimized using a per-pixel RGB reconstruction loss.
Using only RGB images as input leads to an underconstrained problem as there exist an infinite number of photo-consistent
[Figure]
explanations [5,90].
Previous works address this problem by incorporating priors on the structure of the scene into the optimization process, e.g., depth smoothness [46], surface smoothness [49,91], semantic similarity [30], or Manhattan world assumptions [21].
In this paper, we explore monocular geometric priors as they are readily available and efficient to compute.
We show that using such priors significantly improves 3D reconstruction quality in challenging scenarios (see second row of Fig. 1).
Estimating geometric cues such as depth and normals from a single image has been an active research area for decades.
The seminal work by Eigen et al.
[18,19] showed that learned models based on deep convolutional neural networks (CNNs) significantly improved over early work in this area [24–27,59–61].
Recent work [17,55,56], in particular Omnidata [17], has made significant headway in terms of prediction quality and generalization to new scenes using very large datasets for training.
These strong results on individual images, and the fact that monocular geometric cues can be computed efficiently, naturally lead to the question whether such models are able to provide the additional constraints required by implicit neural surface reconstruction approaches to handle more challenging settings.
This paper describes a framework, called MonoSDF, for integrating monocular geometric priors into neural implicit surface reconstruction methods: given multi-view images, we infer depth and surface normals for each image, and use them as additional supervision signals during optimization together with the RGB image reconstruction loss.
We observe that these priors lead to significant gains in reconstruction quality, especially in textureless and less-observed areas as shown in Fig.
This is due to the fact that the photometric consistency cues used by surface reconstruction methods and the recognition cues used by monocular networks are complementary: while photometric consistency fails in texturless regions such as walls, surface normals can be predicted reliably in these areas due to the structured 3D scene layout.
Conversely, photoconsistency cues allow for establishing globally accurate 3D geometry in textured regions, while normal and (relative) depth cues only provide local geometric information.
Apart from incorporating monocular geometric cues, we provide a systematic study and analysis of state-of-the-art design choices for coordinate-based neural representations in the context of implicit surface reconstruction.
More specifically, we investigate the following architectures: a single, large MLP [49,76,81,82], a dense SDF grid [31], a single feature grid [28,38,53,54] and multi-resolution feature grids [10, 22, 45, 69, 92].
We observe that MLPs act globally and exhibit an inductive smoothness bias while being computationally expensive to optimize and evaluate.
In contrast, grid-based representations benefit from locality during optimization and evaluation, hence they are computationally more efficient.
However, reconstructions are noisier for sparse views or less-observed areas.
Including monocular geometric priors improves neural implicit reconstruction results across different settings with faster convergence times and independent of the underlying representation.
In summary, we make the following contributions:[-
-]
• We introduce MonoSDF , a novel framework which exploits monocular geometric cues to improve multi-view 3D reconstruction quality, efficiency, and scalability for neural implicit surface models.
• We provide a systematic comparison and detailed analysis of design choices of neural implicit surface representations, including vanilla MLP and grid-based approaches.
• We conduct extensive experiments on multiple challenging datasets, ranging from object-level reconstruction on the DTU dataset [1], over room-level reconstruction on Replica [67] and ScanNet [13], to large-scale indoor scene reconstruction on Tanks and Temples [34].
Related Work
Architectures for Neural Implicit Scene Representations.
Neural implicit scene representations or neural fields [78] have recently gained popularity for representing 3D geometry due to their expressiveness and low memory footprint.
Seminal works [9,42,50] use a single MLP as the scene representation and show impressive object-level reconstruction quality, but they do not scale to more complicated or large-scale scenes due to the limited model capacity.
Follow-up works [10,22,41, 45, 54, 69, 92] combine an MLP decoder with one or multi-level voxel grids of low-dimensional features.
Such hybrid representations are able to better represent fine geometric details and can be evaluated fast.
However, they lead to a larger memory footprint with increasing scene size.
In this paper we provide a systematic comparison of four architectural design choices for implicit surface reconstruction .
3D Reconstruction from Multi-view Images.
Reconstructing the underlying 3D geometry from multi-view images is a long-standing goal of computer vision.
Classic multi-view stereo (MVS) methods [2, 6–8, 35, 35, 62, 64, 65] consider either feature matching for depth estimation [6, 62] or represent shapes with voxels [2, 7, 8, 35, 51, 64, 72, 73].
Learning-based MVS methods usually replace some parts of the classic MVS pipeline, e.g., feature matching [23, 36, 40, 74, 88], depth fusion [16,57], or inferring depth from multi-view images [29,79,80,85].
In contrast to the explicit scene representations used by classic MVS algorithms, recent neural approaches [39,48,82] represent surfaces via a single MLP with continuous outputs.
Learned purely from posed 2D images, they show appealing reconstruction results and do not suffer from discretization.
However, accurate object masks are required.
Inspired by the density-based volume rendering in NeRF [44], which demonstrated impressive view synthesis without object masks, several works [49,76,81] use volume rendering for neural implicit surface reconstruction without masks.
However, these methods lead to poor results in large-scale scenes with textureless regions.
In this work, we show that incorporating monocular priors allows these approaches to obtain significantly more detailed reconstructions and to scale to larger and more challenging scenes.
Incorporating Priors into Neural Scene Representations.
Several researchers proposed to incorporate priors such as depth smoothness [46], semantic similarity [30], or sparse MVS point clouds [58] for the task of novel view synthesis from sparse inputs.
In contrast, in this work, our focus is on implicit 3D surface reconstruction.
Concurrently, Manhattan-SDF [21] uses dense MVS depth maps from COLMAP [63] as supervision and adopts Manhattan world priors [11] to handle lowtextured planar regions corresponding to walls, floors, etc.
Our approach is based on the observation that data-driven monocular depth and normal predictions [17] provide high-quality priors for the full scene.
Incorporating these priors into the optimization of neural implicit surfaces not only removes the Manhattan world assumption [11] but also results in improved reconstruction quality and a simpler pipeline. 1 Compared to NeuRIS [75], a concurrent work that proposes to use normal priors for indoor scene reconstruction, we integrate monocular depth cues and further demonstrate the effectiveness of monocular cues on various neural scene representations, ranging from MLP to multi-resolution feature grids.
Method
Our goal is to recover the underlying scene geometry from multiple posed images while utilizing monocular geometric cues to guide the optimization process.
To this end, we first review neural implicit scene representations and various design choices in Section 3.1 and discuss how to perform volume rendering of these representations in Section 3.2.
Next, we introduce the monocular geometric cues we investigate in our study in Section 3.3 and discuss loss functions and the overall optimization process in Section 3.4.
An overview of our framework is provided in Fig.
3.1 Implicit Scene Representations
We represent scene geometry as a signed distance function (SDF).
A signed distance function is a continuous function f that, for a given 3D point, returns the point’s distance to the closest surface:
[Equation]
Here, x is the 3D point and s denotes the corresponding SDF value.
In this work, we parameterize the SDF function with learnable parameters θ and investigate several different design choices for representing the function: explicit as a dense grid of learnable SDF values, implicit as a single MLP, or hybrid using an MLP in combination with single- or multi-resolution feature grids.
Dense SDF Grid.
The most straightforward way of parameterizing an SDF is to directly store SDF values in each cell of a discretized volume G θ with resolution of R H × R W × R D [31].
To query the
SDF value ˆ s for an arbitrary point x from the dense SDF grid, we can use any interpolation operation:
[Equation]
In our experiments, we implement interp as trilinear interpolation.
Single MLP.
The SDF function can also be parameterized by a single MLP [50] f θ :
[Equation]
where ˆ s is the predicted SDF value and γ corresponds to a fixed positional encoding [44,70] mapping x to a higher dimensional space.
After their introduction to novel view synthesis [44], positional encoding functions are now widely used for neural implicit surface reconstruction [49,76,81,82] as they increase the expressiveness of coordinate-based networks [70].
Single-Resolution Feature Grid with MLP Decoder.
We can also combine both parameterizations and use a feature-conditioned MLP f θ together with a feature grid Φ θ with a resolution of R 3 , where each cell of the grid stores a feature vector [28,38,54,69] instead of directly storing SDF values:
[Equation]
Note that the MLP f θ is conditioned on the interpolated local feature vector from the feature grid Φ θ .
Multi-Resolution Feature Grids with MLP Decoder.
Instead of using a single feature grid Φ θ , one can also employ multi-resolution feature grids { Φ lθ } Ll =1 with resolutions R l [10,22,45,69,92].
The resolutions are sampled in geometric space [45] to combine features at different frequencies:
[Equation]
where R min , R max are the coarsest and finest resolution, respectively.
Similarly, we extract the interpolated features at each level and concatenate them together:
[Equation]
As the total number of grid cells grows cubically, we use a fixed number of parameters to store the feature grids and use a spatial hash function to index the feature vector at finer levels [45] (see supplementary for details).
Color Prediction.
In addition to the 3D geometry, we also predict color values such that our model can be optimized with a reconstruction loss.
Following [82], we therefore define a second function c θ
[Equation]
that predicts a RGB color value ˆ c for a 3D point x and a viewing direction v .
The 3D unit normal ˆ n is the analytical gradient of our SDF function.
The feature vector ˆ z is the output of a second linear head of the SDF network as in [82].
We parameterize c θ with a two-layer MLP with network weights θ .
In case of the dense grid SDF parameterization, we similarly optimize a dense feature grid and obtain the feature vector ˆ z via the interpolation function interp .
3.2 Volume Rendering of Implicit Surfaces
Following recent work [49,76,81,82], we optimize the implicit representations described in Section 3. via an image-based reconstruction loss using differentiable volume rendering.
More specifically, to render a pixel, we cast a ray r from the camera center o through the pixel along its view direction v .
We sample M points x i r
= o
+ t
i r v along the ray and predict their SDF ˆ s i r and color values ˆ c i r .
We follow [81] to transform the SDF values ˆ s i r to density values σ i r for volume rendering:
[Equation]
where β is a learnable parameter.
Following NeRF [44], the color ˆ C ( r ) for the current ray r is computed via numerical integration:
[Equation]
where T i r and α i r denote the transmittance and alpha value of sample point i along ray r , respectively, and δ i r is the distance between neighboring sample points.
Similarly, we compute the depth ˆ D ( r ) and normal ˆ N ( r ) of the surface intersecting the current ray as:
[Equation]
3.3 Exploiting Monocular Geometric Cues
Unifying volume rendering with implicit surfaces leads to impressive 3D reconstruction results.
Yet, this approach struggles with more complex scenes especially in textureless and sparsely covered regions.
To overcome this limitation, we use readily available, efficient-to-compute monocular geometric priors thereby improving neural implicit surface methods.
Monocular Depth Cues.
One common monocular geometric cue is a monocular depth map, which can be easily obtained via an off-the-shelf monocular depth predictor.
More specifically, we use a pretrained Omnidata model [17] to predict a depth map ¯ D for each input RGB image.
Note that the absolute scale is difficult to estimate in general scenes, so ¯ D must be considered as a relative cue.
However, this relative depth information is provided also over larger distances in the image.
Monocular Normal Cues.
Another geometric cue we use is the surface normal.
Similar to the depth cues, we apply the same pretrained Omnidata model to acquire a normal map ¯ N for each RGB image.
Unlike depth cues that provide semi-local relative information, normal cues are local and capture geometric detail.
We hence expect that surface normals and depth are complementary to each other.
3.4 Optimization
Reconstruction Loss.
Eq.
(9) provides a linkage from the 3D scene representation to 2D observations.
We can therefore optimize the scene representation with a simple RGB reconstruction loss:
[Equation]
Here R
Eikonal Loss.
Following common practice, we also add an Eikonal term [20] on the sampled points to regularize SDF values in 3D space
[Equation] [Equation]
where X are a set of uniformly sampled points together with near-surface points [81].
Depth Consistency Loss.
Besides L rgb and L eikonal , we also enforce consistency between our rendered expected depth ˆ D and the monocular depth ¯ D :
[Equation]
where w and q are the scale and shift used to align ˆ D and ¯ D since ¯ D is defined only up to scale.
Note that these factors have to be estimated individually per batch as the depth maps predicted for different batches can differ in scale and shift.
Specifically, we solve for w and q with a least-squares criterion [19,56] which has a closed-form solution (see supplementary for details).
Normal Consistency Loss.
Similarly, we impose consistency on the volume-rendered normal ˆ N and the predicted monocular normals ¯ N transformed to the same coordinate system with angular and
L1 losses [17]:
[Equation]
The overall loss we use to optimize our implicit surfaces jointly with the appearance network is:
[Equation]
Implementation Details.
We implement our method in PyTorch [52] and use the Adam optimizer [33] with a learning rate of 5e-4 for neural networks and 1e-2 for feature grids and dense SDF grids.
We set λ 1 , λ 2 , λ 3 to 0.1, 0.1, 0.05, respectively.
We sample 1024 rays per iteration and apply the error-bounded sampling strategy introduced by [81] to sample points along each ray.
For MLPs and feature grids, we adapt the architecture and initialization scheme from [81] and [45], respectively.
For obtaining monocular cues, we first resize each image and center crop it to 384 × 384 , which we then feed as input to the pretrained Omnidata model [17].
See supplementary for more details.
[Figure]
Experiments
We first analyze different architectural design choices and perform ablation studies wrt.
monocular cues and optimization time on a room-level dataset (Replica) with perfect ground truth.
Next, we provide qualitative and quantitative comparisons against state-of-the-art baselines on real-world indoor scenes.
Finally, we evaluate our method on object-level reconstruction for both sparse input and dense input scenarios.
Datasets.
While previous neural implicit-based reconstruction methods mainly focused on singleobject scenes with many input views, in this work, we investigate the importance of monocular geometric cues for scaling to more complex scenes.
Thus we consider: a) Real-world indoor scans:
Replica [67] and ScanNet [13]; b) Real-world large-scale indoor scenes: Tanks and Temples [34] advanced scenes; c) Object-level scenes: DTU [1] in the sparse 3-view setting from [46,84].
Baselines.
We compare against a) state-of-the-art neural implicit surfaces methods: UNISURF [49],
VolSDF [81], NeuS [76], and Manhattan-SDF [21].
b) Classic MVS methods: COLMAP [62] and a state-of-the-art commercial software (RealityCapture 2 ).
c) TSDF-Fusion [12] with predicted monocular depth cues, where GT depth maps are used to recover the scale and shift values (cf. Eq.
This baseline shows the reconstruction quality if only monocular depth cues and no implicit surface model is used.
Evaluation Metrics.
For DTU, we follow the official evaluation protocol and report the Chamfer distance.
For Replica and ScanNet, following [21,42,53,54,68,92], we report the Chamfer Distance, the F-score with a threshold of 5cm, as well as a Normal Consistency measure.
4.1 Ablation Study
We first analyze different scene representation choices on the Replica dataset.
Next, we ablate the impact of our geometric cues on reconstruction quality and convergence time.
[Table]
We compare the four different scene geometry representations introduced in Section 3.1 and report metrics averaged over the Replica dataset in Table 1.
Note that no monocular geometric cues are used here.
We first observe that using a single MLP as the scene geometry representation leads to decent results, but the reconstruction tends to be over-smooth (see Table 1 and Fig. 3).
For grid-based representations, optimizing a dense SDF grid leads to a significantly worse performance compared to all other neural implicit scene representations, even with careful parameter tuning.
The reason is the lack of a smoothness bias: The SDF values in grid cells are all stored and optimized independently of each other, hence there is no local or global smoothness bias.
In contrast, the Single-Res.
Fea.
Grid replaces the SDF value in each grid cell with a low-dimensional latent code, and uses a shallow MLP conditioned on these features to read out SDF values of arbitrary 3D points.
This modification leads to a notable boost in reconstruction quality over the dense grid, performing similarly well as the single MLP.
Using a Multi-Res.
Fea.
Grids as in [45] further increases performance.
We observe that the Multi-Res.
Fea.
Grids is the best-performing grid-based model, and from now on we report results for the single MLP [Figure] [Figure] and the Multi-Res.
Feature Grids.
For simplicity, we will refer to the multi-resolution feature grids as
Multi-Res.
Grids or Grids in the
Ablation of Different Cues.
We now investigate the effectiveness of different monocular geometric cues for the two chosen representations.
Table 2 (a) and Fig.
4 show that, for both representations, using either one or both monocular cues significantly boosts reconstruction quality.
We also find both cues to be complementary, with the best performance being achieved when using both.
Similar behavior can be observed for the other two representations (cf. supplementary material).
It is worth noting that the differences between the two representations become negligible when using monocular cues, indicating that those serve as a general drop-in to improve reconstruction quality.
Optimization Time.
Table 2 (b) shows optimization time for the two scene representations with and without cues.
We see that the Multi-Res.
Grids converge faster than the single MLP model.
Further, adding the monocular cues significantly speeds up the convergence process.
After only 10K iterations, both representations perform better than the converged models without monocular cues.
Note that the overhead required for incorporating the monocular cues into the optimization process is small and can be neglected. An extended version of Table 2 (b) can be found in the supplementary materials.
4.2 Real-world Large-scale Scene Reconstruction
To show the effectiveness of our method for large-scale scene reconstruction, we compare against various baselines on two challenging large-scale indoor datasets.
ScanNet.
On ScanNet, we use the test split from [21] and also follow their evaluation protocol in which depth maps are rendered from input camera poses and then re-fused using TSDF Fusion [12] to evaluate only observed areas.
We observe in Table 3 that our MLP variant outperforms all baselines achieving smoother reconstructions with more fine details. Note that we outperform concurrent work [75].
Further, we find that the MLP variant performs significantly better than using Multi-Res.
Grids.
ScanNet’s RGB images contain motion blur and the camera poses are also noisy.
This can be harmful to the local geometry updates in grid-based representations, while MLPs are more robust to this noise due to their smoothness bias. COLMAP [62] [Table] Manhattan-
[Table] [Table] Tanks & Temples.
To further investigate the scalability of our method to larger-scale scenes [Table] we conduct experiments on the Tanks and Temples advanced sets.
The qualitative results in Fig.
1 show that the monocular cues significantly boost the performance of VolSDF [81], making MonoSDF the first neural implicit model achieving reasonable results on such a large-scale indoor scene.
See the supplementary material for more visual comparisons and discussions
[Table]
We now evaluate our method on another challenging task: reconstructing single objects from sparse input views. We adopt the test split from [81,82] on DTU and choose three input views following [46].
[Table] [Table]
We first observe in Table 4 and Fig.
1 that without the usage of the monocular geometric cues, neither the MLP (VolSDF [81]) nor
the Multi-Res. Grids work well with only 3 input views. When incorporating the cues, the results for both representations are significantly improved.
Interestingly, the grid-based representations perform inferior to a single MLP as they are updated locally and do not benefit from the inductive bias of a monolithic MLP representation.
Comparing against TSDF Fusion [12] that fused predicted depth cues from all views into a TSDF volume without any optimization, we observe that this baseline has difficulties in reconstructing meaningful details due to inconsistencies in the monocular depth cues. Note that this baseline uses the GT depth maps from [16] to compute scale and shift for the depth cues. Classic MVS methods perform well quantitatively, but they heavily rely on dense matching, and in case of three input images, this inevitably leads to incomplete reconstructions (see supplementary material). In contrast, our approach combines neural implicit surface representations with the benefits from monocular geometric cues that are more robust to less-observed regions.
4.4 Object-level Reconstruction from Dense Views
To further investigate the effectiveness and flexibility of our method, we evaluate our approach on the DTU dataset with all input views, which is a common setting in recent work [49,77,81]. In this experiment, we simply resize the low-resolution monocular cues to full resolution (from 384 ×to 1200 × 1200 pixels) while keeping the image ratio.
As the original image is of size 1200 × 1600 , the monocular cues are missing in the left and right part of the image.
Therefore, we only use the monocular cues where they are available. As shown in Table 5, our approach with MLP architecture achieves reconstruction quality similar to state-of-the-art methods [49,77,81].
This is reasonable as the dense input views provide enough constraints and the prior information from monocular cues is negligible. However, our method with multi-resolution feature grid architecture outperforms previous work by a large margin.
We attribute this to the expressiveness of multi-resolution feature grids where monocular cues are still effective to [Table] [Table] noise and therefore can reconstruct smooth and detailed surfaces.
We kindly refer the reader to the supplementary material for additional visual comparisons.
Conclusion
We have presented MonoSDF, a novel framework that systematically explores how monocular geometric cues can be incorporated into the optimization of neural implicit surfaces from multiview images.
We show that such easy-to-obtain monocular cues can significantly improve 3D reconstruction quality, efficiency, and scalability for a variety of neural implicit representations.
When using monocular cues, a simple MLP architecture performs best overall, demonstrating that MLPs in principle are able to represent complex scenes, albeit being slower to converge compared to grid-based representations.
Multi-resolution feature grids in general can converge fast and capture details, but are less robust to noise and ambiguities in the input images.
Limitations.
The performance of our model depends on the quality of the monocular cues.
Filtering strategies to handle failures of the monocular predictor are thus a promising direction to further improve reconstruction quality. We kindly refer the reader to the supplementary material for additional analysis.
While we demonstrated that integrating depth and normal cues significantly improves reconstruction, exploring other cues such as occlusion edges, plane, or curvature [17,87] is an interesting future direction.
We are currently limited by the low-resolution ( 384 × 384 pixels) output of the Omnidata model [17] and plan to explore different ways of using higher-resolution cues. We provide some preliminary results of using high-resolution cues in the supplementary.
Joint optimization of scene representations and camera parameters [4,92] is another interesting direction, especially for multi-resolution grids, in order to better handle noisy camera poses. Acknowledgments and Disclosure of Funding This work was supported by an NVIDIA research gift. We thank the Max Planck ETH Center for
Learning Systems (CLS) for supporting SP and the International Max Planck Research School for
Intelligent Systems (IMPRS-IS) for supporting MN.
ZY is supported by BMWi in the project KI
Delta Learning (project number 19A19013O).
AG is supported by the ERC Starting Grant LEGO3D (850533) and DFG EXC number 2064/1 - project number 390727645.
TS is supported by the EU Horizon 2020 project RICAIP (grant agreeement No.857306), and the European Regional
Development Fund under project IMPACT (No. CZ.02.1.01/0.0/0.0/15_003/0000468).
We thank the authors of Manhattan-SDF and NeuRIS for sharing results on ScanNet.
We also thank Christian
Reiser and Zijian Dong for proofreading.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work? [Yes] We discuss potential negative societal impacts in our supplementary material. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [N/A] (b) Did you include complete proofs of all theoretical results? [N/A] 3. If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Code and data are released. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] We describe details of our computational resources in supplementary material. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [Yes] (b) Did you mention the license of the assets? [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating? [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] 5. If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
Supplementary Material for MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction In this supplementary document , we first discuss architectural and implementation details in Section A. Next, we provide additional ablation studies of our monocular geometric cues for four different scene representations in Section B and report additional quantitative and qualitative results in Section C.
Finally, we discuss potential negative impact of this work in Section D.
A Implementation Details
In this section, we first present an overview of 4 different architectures for neural implicit scene representations and details of Multi-Res.
Grids in Section A.1 and provide details of the depth loss computation in Section A.2.
Next, we describe additional details regarding our parameterizations and optimization in Section A.3 and discuss evaluation metrics in Section A.4.
A.1 Architectures
In the main paper, we investigate four different architectures as our scene representation: Dense SDF
Grid , Single MLP , Single-Res.
Grid , and Multi-Res.
Grids .
See Fig.
5 for an overview over the architectures.
In the following, we provide details for Multi-Res.
Feature Grids.
Multi-Res.
Grids.
Following Instant-NGP [45], we use L levels of feature grids with resolutions sampled in geometric space to combine features at different frequencies:
[Equation]
where R min , R max are the coarsest and finest resolutions, respectively.
As the total number of grid cells grows cubically, we use a fixed number of parameters to store the feature grids and use a spatial hash function to index the feature vector at finer levels.
More specifically, each grid contains up to T feature vectors with dimensionality F .
At the coarse level where R 3 l ≤ T , the feature grid is stored densely.
At the finer level where R 3 l > T , a spatial hash function [71] is used to index the corresponding feature vector:
[Equation]
where (cid:76) is the bit-wise XOR operation and π i are unique, large prime numbers.
We use the default values R min = 16 , R max = 2048 , L = 16 , F = 2 , and T = 2 19 similar to [45] in all experiments.
A.2 Depth Consistency Loss
We enforce consistency between our rendered expected depth ˆ D and the monocular depth ¯ D with a scale invariant loss function:
[Equation]
where w and q are the scale and shift used to align ˆ D and ¯ D since ¯ D is given only up to scale.
Specifically, we solve w and q with a least-squares criterion [19,56]:
[Equation]
w and q can be efficiently computed as follows: Let h = ( w, q ) T and d r = ( ˆ D ( r ) , 1) T , then Eq.
(19) can be rewrite as:
[Equation]
which has the closed-form solution:
[Equation]
Note that we estimate w and q individually at each iteration for a batch of randomly sampled rays within a single image because depth maps predicted by the monocular depth predictor can differ in scale and shift and the underlying scene geometry changes at each iteration.
A.3 Additional Details
For our single MLP architecture, we use an 8-layer MLP with hidden dimension 256.
We use a two-layer MLP with hidden dimension 256 for the SDF prediction for both, Single-Res.
Grid and
Multi-Res.
Grids.
We implement the color network with a two-layer MLP with hidden dimension 256 and use it for all architectures.
We use Softplus activation for geometric network and use ReLU activation for the color network.
We explicitly initialize the SDF grid with a sphere and use the geometric initialization from [3] for other architectures.
For obtaining monocular cues, we first resize each image and center crop it to 384 × 384 , which we then feed as input to the pretrained Omnidata model [17].
The output depth and normal maps have the same resolution of 384 × 384 .
As a result, we use the same resolution for RGB images, depth cues and normal cues and adjust camera intrinsics accordingly for all experiments.
We optimize our model for 200k iterations which takes about 6 hours and 11 hours for our Multi-Res.
Grids and MLP, respectively, on a single NVIDIA RTX3090 GPU.
A.4 Evaluation Metrics
For the DTU dataset [1], we follow the official evaluation protocol and report the reconstruction quality with: Accuracy , Completeness and Chamfer Distance .
Accuracy measures how close the reconstructed points are to the ground truth and is defined as the mean distance of the reconstructed points to the ground truth.
Completeness measures to what extent the ground truth points are recovered and is defined as the mean distance of the ground truth points to the reconstructed points.
Chamfer
Distance is the mean of Accuracy and Completeness .
It measures the overall reconstruction quality.
For efficiency, we use the Python script 3 to compute these evaluation metrics.
[Table]
Replica [67] and ScanNet [13], we report Accuracy , Completeness , Chamfer Distance , Precision ,
Recall , and F-score with a threshold of 5cm following [21, 68, 92].
We further report Normal
Consistency for the Replica dataset following [21,42,53,54,68,92] as near-perfect ground truth is available.
These metrics are defined in Table 6.
For the Tanks and Temples dataset [34], we submit our reconstruction results to the official evaluation server 4 and report the provided F-score.
B Ablation
In this section, we first conduct several ablation studies to verify the effectiveness of our method, including using geometric cues with different scene representations in Section B.1, different architecture configurations in Section B.2, different number of input views in Section B.3, different cues predictors in Section B.4.
Next, we analyze the optimization time of our framework in Section B.5.
[Table]
[Table]
20 Views 30 Views 40 Views
GT Views
M L P 28.
w / C u e s 75.
B.1 Ablation of Different Cues
To evaluate the effectiveness of our monocular geometric cues for different scene representations, we conduct ablation studies on the Replica dataset with our four different scene representations.
Note that as the Replica dataset is part of the training set of Omnidata (making up 0.46% of the entire training data)
[17], we split the evaluation into the train/test split of Omnidata [17].
As shown in Table 2 and Fig. 8, our geometric cues improve reconstruction quality significantly independent of the underlying scene representations.
We observe that using both, depth cues and normal cues, leads to the best results, indicating the complementary nature of the different cues.
We further observe that the reconstruction quality as well as the improvements from adding geometric cues are similar for the train and test split of Omnidata, showing that the monocular predictor did not overfit to the training data.
[Figure]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table]
In order to evaluate the performance with different model capacities
[Figure]
we consider MLPs with a different number of layers and Multi-res. Feature Grids with different sizes of the hash table. We list the number of learnable parameters using different architecture configurations in the Table
[Figure]
and show their performance over the optimization processes in Fig.
Our experiments show that using monocular geometric cues improves reconstruction quality and convergence speed independent of the network configuration. B.3 Ablation of Different Numbers of Input Views We ran experiments with a different number of input images and monocular geometric cues.
As shown in Fig.
6, adding the monocular geometric cues leads to consistent improvements across different numbers of input views. Method [Table] score
MLP 64. w/ MiDaS [56] 68. w/ LeReS [83] 72.
w/ Omnidata [17] 86. Method [Table] score
MLP 64.
w/ Tilted [15] 45.
w/ Omnidata [17] 92.
Method
F-score
MLP 64.
w/ Self-supervised [37,86] 45.
w/ Omnidata [17] 86.
[Figure]
normal improve results.
Similarly, normals predicted by state model leads to the best performance
Using supervised depth estimator degrades performance
We hypothesize that this is due to the weaker performance of the self-supervised model which is also trained with an RGB loss and hence suffers from the under-constrained problem of recovering geometry from multi-view images
B.4 Ablation of Different Monocular Cues Predictors
To further analyze the robustness of our approach to monocular geometric cues of different levels of quality, we further tested our model with different supervised depth predictors [56,83], normal predictors [15], and self-supervised depth predictors [37,86].
The result is shown in Table 9.
We found that using the state-of-the-art Omnidata model leads to the best results, indicating that the development of better geometric cues will further improve the performance of our approach
B.5 Optimization Time
Adding monocular geometric cues to the optimization introduces a small overhead to our overall optimization pipeline.
First, predicting these cues with a pretrained Omnidata model is very efficient (36 FPS with an NVIDIA RTX3090 GPU).
For example, it takes less than 26 seconds to predict both depth maps and normal maps for 464 images for one of the ScanNet scene.
Note that this only needs to be done once and that we measure FPS with a batch size of one; using a larger batch size will result in a speed up.
Second, we volume render depth and normals during optimization in order to apply a loss against these monocular cues.
This overhead is also small and can be neglected since the most expensive part wrt.
compute is the inference of the network.
For our MLP variant, the additional flops for volume rendering depth and normal is only 0.0002% of the MLP inference time.
While adding monocular geometric cues introduce a small overhead, the improvements in terms of reconstruction quality and converge speed are significant.
As shown in Table 2 (b) in the main paper, with only 5k iterations, our Multi-Res.
Grids representation with cues performs better than the converged models without geometric cues, which implies a 40 × speed up (5k vs. 200k).
C Additional Results
In this section, we provide more qualitative and quantitative results for three datasets: ScanNet ( Section C.1), Tanks and Temples ( Section C.2), and DTU ( Section C.4).
C.1 ScanNet
We report quantitative results with all metrics for ScanNet in Table 10 and show more visualizations in Fig.
Compared to state-of-the-art methods, our approach with MLP architecture produces significantly better reconstructions both visually as well as quantitatively.
It’s worth noting that we perform better than concurrent work [75] even though they have some filtering mechanism.
[Table]
[Table]
C.2 Tanks and Temples
We show quantitative results for Tanks and Temples in Table 11.
Qualitative comparisons of with or without monocular cues of our MLP variant are shown in Fig.
10 and Fig.
Fig.
12 and Fig.show qualitative comparison of our Mulit-Res.
Grids.
Our monocular geometric cues significantly improve the reconstruction quality.
We further show an additional comparison against state-of-the-art MVS methods in Fig.
We use a pretrained Vis-MVSNet [89] to predict depth maps for the input images and fuse them to point clouds follow the official code.
Next, we use Meshlab’s screened Poisson reconstruction [32] to reconstruct a mesh from point clouds with default parameters.
We observe that our reconstructions are more complete which is useful for many applications.
Further, reconstructing a mesh from point clouds involves lossy post-processing, leading to floating artifacts and bloated areas in less-observed areas.
C.3 Preliminary Results of Using High-resolution Monocular Cues
In the main paper, we center-crop each image and resize it to 384 × 384 .
Then, we use a pretrained
Omnidata model to predict depth maps and normal maps which are also of size 384 × 384 .
While we have shown that training at a resolution of 384 × 384 produces impressive results, we believe that exploring different ways to generate and integrate higher resolution cues could further improve reconstruction quality.
Here, we provide a proof-of-concept experiment for generating higher resolution monocular cues and integrating them into our model.
We use a divide-and-conquer method for generating high-resolution cues.
First, we partition a high-resolution image to multiple overlapping sub-images, and we predict monocular depth and normal for each sub-image.
Next, we merge these predictions.
We use Eq. 21 to align the depth maps and solve the rotation for the normal [Table] scan
[Table] [Table] [Table] [Table]
[Table]
[Table] [Table]
scan [Table] [Table] [Table]
[Table]
[Table]
[Table] [Table]
scan [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
scan
[Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
scan [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
scan
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] scan
[Table] [Table] [Table] [Table]
scan
[Table]
[Table] [Table] [Table] [Table]
[Table]
scan [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
scan
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] scan
[Table] [Table]
[Table]
[Table]
[Table] [Table] [Table] scan
[Table] N/A [Table] [Table]
[Table]
[Table]
[Table]
scan 4.
[Table] [Table] [Table] [Table]
scan 4. [Table] [Table] [Table]
[Table] [Table] scan 4. [Table]
[Table]
[Table]
maps.
An example of the resulting high-resolution monocular cues is shown in Fig.
We found that our high-resolution cues contain more fine details compared to low-resolution cues. Note that using other methods for generating high-resolution depth maps is also possible, e.g., [43]. We then use the high-resolution cues to train our model, and the results are shown in Fig. We observe significant improvements when using high-resolution monocular cues.
Geometry. We show per-scene quantitative results on the DTU dataset with 3 input-views in Tableand more qualitative results in Fig. We find that without the monocular geometric cues, both
MLP and Multi-Res. Grids fail to produce satisfying reconstructions, while with our monocular cues, both methods are improved and are able to reconstruct high-quality meshes. We further show more visualizations on the DTU dataset using all input views in Fig. 19. Compared to state-of-the-art methods, our approach with multi-resolution feature grids produces more accurate reconstructions.
PSNR [Table] [Table] [Table]
Novel View Synthesis.
We further compare our novel view synthesis results on the DTU dataset with three input views.
As shown in Tableand Fig.
18, using monocular geometric cues improves novel view synthesis results significantly.
Weight Annealing.
As the monocular depth and normal predictor is not perfect, we exponentially anneal the loss weight for the monocular depth consistency and normal consistency loss, λ 2 and λ 3 , to 0 during the first 200 epochs of optimization.
Qualitative comparison in Fig. 20 verifies the importance of weight annealing. Failure cases. We show a failure case on DTU with 3 input views in Fig. 21.
The reconstructed mesh duplicates the object in front of each camera frustum. One reason is that the monocular depth cues that we use are only up to scale so they do not guarantee multi-view consistency. Therefore, the optimization is still underconstrained since the input RGB images and monocular cues can be explained by individual objects in front of the image plane. One possible solution would be incorporating explicit multi-view constraints such as using sparse point clouds from COLMAP [62] as an additional supervision [14].
Our method can faithfully reconstruct a 3D scene which can be used for application ranging from virtual reality to robotics. However, it can also have potential negative societal impact. First, our method relies on a general purpose monocular geometric predictor that needs to be trained on large amounts of data and with large computational resources, which potentially has a negative impact on global climate change.
Second, accurate reconstruction of a scene may raise privacy concerns that need to be addressed carefully.
Finally, accurate geometry reconstructed by our method can potentially be used for malicious purposes. [Figure] Depth + Normal + Both [Figure]
Both
[Figure] [Figure] [Figure] [Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure] [Figure] [Figure] [Figure]