Learning Consistency-Aware Unsigned Distance Functions Progressively from Raw Point Clouds
Abstract
Surface reconstruction for point clouds is an important task in 3D computer vision.
Most of the latest methods resolve this problem by learning signed distance functions (SDF) from point clouds, which are limited to reconstructing shapes or scenes with closed surfaces.
Some other methods tried to represent shapes or scenes with open surfaces using unsigned distance functions (UDF) which are learned from large scale ground truth unsigned distances.
However, the learned
UDF is hard to provide smooth distance fields near the surface due to the noncontinuous character of point clouds.
In this paper, we propose a novel method to learn consistency-aware unsigned distance functions directly from raw point clouds.
We achieve this by learning to move 3D queries to reach the surface with a field consistency constraint, where we also enable to progressively estimate a more accurate surface.
Specifically, we train a neural network to gradually infer the relationship between 3D queries and the approximated surface by searching for the moving target of queries in a dynamic way, which results in a consistent field around the surface.
Meanwhile, we introduce a polygonization algorithm to extract surfaces directly from the gradient field of the learned UDF.
The experimental results in surface reconstruction for synthetic and real scan data show significant improvements over the state-of-the-art under the widely used benchmarks.
Project page: https://junshengzhou.github.io/CAP-UDF .
Introduction
Reconstructing surfaces from 3D point clouds is vital in 3D vision, robotics and graphics.
It bridges the gap between raw point clouds that can be captured by 3D sensors and the editable surfaces for various downstream applications.
Recently, Neural Implicit Functions (NIFs) have achieved promising results by training deep networks to learn Signed Distance Functions (SDFs) [44, 26, 42, 14] or occupancies [40, 46, 41, 10], and then extract a polygon mesh of a continuous iso-surface from a discrete scalar field using the marching cubes algorithm [35].
However, the NIFs approaches based on learning internal and external relations can only reconstruct closed surfaces.
The limitation prevents NIFs from representing most real-world objects such as cars with inner structures, clothes with unsealed ends or 3D scenes with open walls and holes.
As a remedy, state-of-the-art methods [12, 61, 48] learn Unsigned Distance Functions (UDFs) as a more general representation to reconstruct surfaces from point clouds.
However, these methods can not learn UDFs with smooth distance fields near surfaces, due to the noncontinuous character of point clouds, even using ground truth distance values or large scale meshes during training.
Moreover, most UDF approaches failed to extract surfaces directly from unsigned distance fields.
Particularly, they rely on post-processing such as Ball-Pivoting-Algorithm (BPA) [3] to extract surfaces based on the dense point clouds generated from the learned UDF, which is very time-consuming and also leads to surfaces with discontinuity and poor quality.
To solve these issues, we propose a novel method to learn consistency-aware UDFs directly from raw point clouds.
We learn to move 3D queries to reach the approximated surface aggressively with a field consistency constraint, and introduce a polygonization algorithm to extract surfaces from the learned unsigned distance functions in a new perspective.
Our method can learn UDFs from a single point cloud without requiring ground truth distances, point normals, or a large scale training set.
Specifically, given query locations sampled in 3D space as input, We learn to move them to the approximated surface according to the predicted unsigned distances and the gradient at the query locations.
More appealing solutions [1, 2, 20, 36] have been proposed to learn SDFs from raw point clouds by optimizing the relationship between the query point and its closest point in raw data as a surface prior.
However, since the raw point cloud is a highly discrete approximation of the surface, the closest point to the query location is always inaccurate and ambiguous, which makes the network difficult to converge to an accurate UDF due to the inconsistent or even conflicting optimization directions in the distance field.
Therefore, in order to encourage the network to learn a consistency-aware and accurate unsigned distance field, we propose to dynamically search the optimization target with a specially designed loss function containing field consistency to mimic the conflict optimizations.
We also progressively infer the mapping between 3D queries and the approximated zero iso-surface by using well-moved queries as additional priors for promoting further convergence.
To extract a surface in a direct way, we propose to use the gradient field of the learned UDFs to determine whether two queries are on the same side of the approximated surface or not.
In contrast to NDF [12] which also learns UDFs but takes dense point clouds as output and depends on BPA [3] to generate meshes, our method shows great advantages in efficiency and accuracy due to the straightforward surface extraction.
Our main contributions can be summarized as:
• We propose a novel neural network that learns consistent-aware UDFs directly from raw point clouds.
Our method gradually infers the relationship between 3D query locations and the approximated surface with a field consistent loss.
• We introduce an algorithm for directly extracting high-fidelity iso-surfaces with arbitrary topology from the gradient field of the learned unsigned distance functions.
• We obtain state-of-the-art results in surface reconstruction from synthetic and real scan point clouds under the widely used benchmarks.
Related Works
Surface reconstruction from 3D point clouds has been studied for decades.
Classic optimizationbased methods [15, 3, 28, 29] tried to resolve this problem by inferring continuous surfaces from the geometry of point clouds.
With the rapid development of deep learning [62, 33, 55, 60, 27, 24, 53, 54, 57, 52], the neural networks have shown great potential in reconstructing 3D surfaces [30, 9, 8, 22, 13, 17, 50, 43, 31, 51].
In the following, we will briefly review the studies of deep learning based methods.
2.1 Neural Implicit Surface Reconstruction
In the past few years, a lot of advances have been made in 3D surface reconstruction with Neural
Implicit Functions (NIFs).
The NIFs approaches [7, 25, 46, 40, 16, 26, 32, 41, 34, 44, 39] use either binary occupancies [40, 46, 41, 10] or signed distance functions (SDFs) [44, 26, 42, 14] to represent 3D shapes or scenes, and then use marching cubes [35] algorithm to reconstruct the learned implicit functions into surfaces.
Earlier works [40, 44, 25, 10] use an encoder [40, 10] or an optimization based method [44] to embed the shape into a global latent code, and then use a decoder to reconstruct the shape.
To obtain more detailed geometry, some methods [18, 19, 47, 7, 11, 26, 41, 38, 37] proposed to leverage more latent codes to capture local shape priors.
To achieve this, the point cloud is first split into different uniform grids [7, 11, 26] or local patches [18, 19, 47], and a neural network is then used to extract a latent code for each grid/patch.
Some recent works propose to learn NIFs from a new perspective, such as implicit moving least-squares surfaces [32], differentiable poisson solver [45], iso-points [59] , point convolution [6] or predictive context learning [38].
However, the
NIFs approaches can only represent closed shapes due to the characters of occupancies and SDFs.
2.2 Learning Unsigned Distance Functions
To model general shapes with open and multi-layer surfaces, NDF [12] learns unsigned distance functions to represent shapes by predicting the unsigned distance from a query location to the continuous surface.
However, NDF merely predicts dense point clouds as the output, which requires time-consuming post-processing for mesh generation and also struggles to retain high-quality details of shapes.
In contrast, our method is able to extract surfaces directly from the gradient field of the learned UDFs.
Following works use image features [61] or query side relations [58] as additional constraints to improve reconstruction accuracy, some other works advance UDFs for normal estimation [48] or semantic segmentation [49].
However, these methods require ground truth distance values or even a large scale meshes during training and are hard to provide smooth distance fields near the surface due to the noncontinuous character of point clouds.
While our method does not require any additional supervision but raw point clouds during training, which allows us to reconstruct surfaces for real point cloud scans.
In a differential manner, a concurrent work named MeshUDF [21] meshes UDFs from the dynamic gradients during training with a voting schema.
On the contrary, we learn a consistancy-aware UDF first and extract the surface from stable gradients during testing.
Moreover, our surface extraction algorithm is simpler to use, which is implemented in the marching cube algorithm.
2.3 Surface Reconstruction from Raw Point Clouds
Learning implicit functions directly from raw point clouds without ground truth signed/unsigned distance values or occupancy values is more challenge.
Current works introduce sign agnostic learning with a specially designed network initialization [1], constraints on gradients [2] or geometric regularization [20] for learning SDFs from raw data.
Neural-Pull [36] uses a new way of learning
SDFs by pulling nearby space onto the surface.
However, they aim to learn signed distances and hence can not reconstruct complex shapes with open or multi-layer surfaces.
In contrast, our method is able to learn a continuous unsigned distance function from point clouds, which allows us to reconstruct surfaces for shapes and scenes with arbitrary typology.
Method
Problem statement.
We design a neural network to learn UDFs that represents 3D shapes.
Given a 3D query location q =
[ x, y, z ] , a learned UDF f predicts the unsigned distance value s = f ( q ) ∈ R .
Current methods depend on ground truth distance values generated from continuous surfaces and employ a neural network to learn f as a regression problem.
Different from these methods, we aim to learn f directly from a raw point cloud P = { p i , i ∈ [1 , N ] } .
Furthermore, these methods require post-processing [12] or additional supervision [58] to generate meshes.
On the contrary, we introduce an algorithm to extract surfaces directly from f using the gradient field ∇ f .
The overview of our method is shown in Fig.
3.1 Learn UDFs from Raw Point Clouds
We introduce a novel neural network to learn a continuous UDF f from a raw point cloud.
We demonstrate our idea using a 2D point cloud S in Fig. 1(a), where S indicates some discrete points of a continuous surface.
Specifically, given a set of query locations Q = { q i , i ∈ [1 , M ] } which is randomly sampled around S , the network moves q i against the direction of the gradient g i at q i with a stride of predicted unsigned distance value f ( q i ) .
The gradient g i is a vector that presents the partial derivative of f at q i
=
[ x i , y i , z i ] , which can be formulated as g i = ∇ f ( q i ) = [Figure] UDF
Gradient ℒ [Figure] + [Figure]
ℒ + 𝑁𝑒𝑢𝑟𝑎𝑙
[Figure]
𝑀 𝑀 i. Stage
[Figure]
Stage𝑁 iii.
Surface extraction algorithm ii.
Stage
UDFsandGradientfields
Zeroiso−surfaces 3Dgrids ℒ Field consistency loss + 𝑀 Moving operation
Union operator
Query locations
Raw point clouds 𝒇(𝒒
[Figure]
) 𝜵𝒇(𝒒 𝒊 [Figure] [ ∂f/∂x, ∂f/∂y, ∂f/∂z ] . The direction of g i indicates the orientation of where the unsigned distance increases the fastest in 3D space, which points the direction away from the surface, therefore moving q i against the direction of g i will find a path to reach the surface of S .
The moving operation can be formulated as:
[Equation]
[Equation]
) where z i is the location of the moved query q i , and ∇ f ( q i ) /
||∇ f ( q i )
|| 2 is the normalized gradient g i , which indicates the direction of g i .
The moving operation is differentiable in both the unsigned distance value and the gradient, which allows us to optimize them simultaneously during training.
[Equation]
[Equation]
b) SAL, (c) NDF, (d) Ours.
The color of blue and red represent positive or negative distance.
The darker the color, the closer it is to the approximated surface.
The four examples in Fig.
2 show the distance fields learned by Neural-Pull [36], SAL [1], NDF [12] and our method for a sparse 2D point cloud P which only contains 13 points.
One main branch to learn signed or unsigned distance functions for point clouds is to directly minimize the mean squared error between the predicted distance value f ( q i ) and the euclidean distance between q i and its nearest neighbour in P , as proposed in NDF and SAL.
However, as shown in Fig. 2(c), NDF leads to an extremely discrete distance field.
To learn a continuous distance field, NDF introduces ground truth distance values extracted from the continuous surface as extra supervision, which prevents it from learning directly from raw point clouds.
SAL shows a great capacity in learning SDFs for watertight shapes using a carefully designed initialization.
However, as shown in
Fig. 2(b), SAL fails to converge to a multi-structure shape since the network is initialized as a single layer shape prior.
Neural-Pull uses a similar way as ours to pull queries onto the surface, thus also learns a continuous signed distance field as shown in Fig. 2(a).
However, the nature of SDF prevents Neural-Pull from reconstructing open surfaces like the “1” on the left of Fig. 2(a).
As shown in Fig. 2(d), our method can learn a continuous level set of distance field and can also represent open surfaces.
One way to extend Neural-Pull directly to learn UDFs is to predict a positive distance value for each query and pull it to the nearest neighbour in P .
However, for shapes with complex topology, this optimization is often ambiguous due to the noncontinuous character of raw point clouds.
We solve this problem by introducing consistency-aware field learning.
3.2 Consistency-Aware Field Learning
[Figure]
Neural-Pull leverages a mean squared error to minimize the distance between the moved query
z i and the nearest neighbour n i of q i in P :
[Equation]
However, the direct optimization of the loss in
Eq.
(2) will form a distorted field and lead some queries to get stuck due to the conflict optimization which makes the network difficult to converge.
We show a 2D case of learning UDFs for a double-deck wall using the loss Eq.
(2) as in Fig.
3(b) and using our loss as in Fig. 3(c).
Assume p 1 and p 2 as two discrete points in two different decks of the wall, q 1 and q 2 are two queries whose closest neighbours are p 1 and p 2 , respectively.
Optimizing the network using q 1 and q 2 by minimizing Eq.
(2) or our proposed loss in Eq.
(3) will lead to an unsigned distance field as in Fig. 3(a).
Assuming in the next training batch, q 3 and q 4 are two queries whose closest neighbours are p 3 and p 4 .
If we use the loss in Eq.
(2), the optimization target of q 3 is to minimize L = ||
z 3 − p 3 ||
Notice that the target point p 3 is located on the lower surface, however the opposite direction of gradient around q 3 is upward at this moment.
Therefore, the partial derivative ∂ L ∂z 3 leads to a decrease in the unsigned distance value f ( q 3 ) predicted by the network.
The case of q 4 is optimized similarly.
An immediate consequence is that the inconsistent optimization directions will form a distorted fields that has local minima of unsigned distance values at q 3 and q 4 as in Fig. 3(b).
However, this situation causes other query points around point q 3 or q 4 to get stuck in the distorted fields and unable to move to the correct location, thus making the network hard to converge.
To address this issue, we propose a loss function which can keep the consistency of unsigned distance fields to avoid the conflicting optimization directions.
Specifically, instead of strictly constraining the convergence target before forward propagation as Eq.
(2), we first predict the moving path of a query location q i and move it using Eq.
(1) to z i , then look for the surface point p i in P which is the closest to z i and minimize the distance between z i and p i .
As shown in Fig. 3(c), after moving q 3 against the gradient direction with a stride of f ( q 3 ) to z 3 , the closest surface point of z 3 lies on the upper deck, so the distance fields remain continuous and are optimized correctly.
In practical, we use chamfer distance as a suitable loss implementation, formulated as:
[Equation]
𝒇)
We also use toy examples as shown in Fig.
4 to show the advantage of our proposed field consistency loss.
We learn UDFs for a raw point cloud of a double-deck wall as shown in Fig. 4(a).
Fig. 4(b) denotes the randomly sampled query locations between two decks of the wall where the different colors mean the queries are closer to the upper or lower deck of the wall.
Fig. 4(c) and Fig. 4(d) indicate the moved queries by loss in Eq.
(2) and Eq.
It can be seen that our proposed loss can move most of the queries to the correct surface position, and Neural-Pull loss stops moving in many places or moves queries to the wrong places due to the field inconsistency in optimization.
Fig.
4(e) and Fig. 4(f)
show the learned distance field of a car with inner structure by the loss in Eq.
(2) and our loss.
[Figure]
3.3 Progressive Surface Approximation
Moreover, in order to predict unsigned distance values more accurately and learn more local details, we propose a progressive learning strategy by taking the intermediate results of moved queries as additional priors.
Given a raw point cloud which is a discrete representation of the surface, we have made a reasonable assumption: the closer the query location is to the given point cloud, the smaller the error of searching the target point on the given point cloud.
We provide the Proof of this assumption in the appendix.
Based on this assumption, we set up two regions: the high confidence region with small error and the low confidence region with large error.
We sample query points in the high confidence region to help train the network and sample auxiliary points in the low confidence region to move to the estimated surface position by network gradient after network convergence at current stage, where the moved auxiliary points are regarded as the surface prior for the next stage.
Notably, the auxiliary points do not participate in network training since these points with low confidence will lead to a large error and affect network training.
Since the low confidence regions which are not optimized explicitly during training are distributed interspersed between the high confidence regions, according to the integral Monotone Convergence Theorem [5], the UDFs and gradient predicted by the low confidence region are a smooth expression of the trained high confidence region.
We save the moved queries and auxiliary points and use them to update S .
According to the updated point cloud, we re-divide the regions with high confidence and low confidence and re-sample the query points and auxiliary points for the next stage.
We demonstrate our idea using a 2D case in Fig.
5.(a).
We divided the regions with high confidence (red region) and low confidence (yellow region) based on the given raw point cloud S 1 (black dots) and then sample query points Q 1 = { q i , i ∈ [1 , M ] } (blue dots) and auxiliary points(green dots)
A 1 = { a i , i ∈ [1 , M ] } .
(b) We train the network to learn UDFs by moving the query locations Qusing Eq.
(1), and optimize the network by minimizing Eq.
(c) After network convergence at current stage, we move query points Q and auxiliary points A to the estimated surface position by the gradient of network, S ′
= p − f ( p ) × ∇ f ( p ) / ||∇ f ( p ) || 2 , p ⊂ Q 1 ∪ A 1 .
(d) We save the moved points S ′ 1 and use them to update S , S
= S 1 ∪ S ′ 1 .
According to the updated point cloud S 2 , we re-divided the regions with high confidence and low confidence and re-sampled the query points
Q 2 and auxiliary points
A 2 .
(e) We continue to train the network by moving query points Q 2 to the updated S 2 , and then update S by combining the moved Q 2 and A 2 with S 2 .
(f) Because of the more continuous surface prior information, the network will learn more accurately and learn more local details of the UDFs.
3.4 Surface Extraction Algorithm
[Figure]
Unlike SDFs, UDFs fail to extract surfaces by marching cubes since UDFs cannot perform inside/outside tests on 3D grids.
To address this issue, we propose to use the gradient field ∇ f to determine whether two 3D grid locations are on the same side or the opposite side of the surface approximated by the point clouds P .
We make an assumption that on a micro-scale of the surface, the space can always be divided into two sides, where the 3D query locations of different sides denoted as Q in = { q in i , i ∈
[1 , L ] } and
Q out =
{ q outi , i ∈
[1 , I ] } .
For two queries q ini and q outj in different sides of the surface, the included angle between the directions of the gradients ∇ f ( q ini ) and ∇ f ( q outj ) are always more than 90 degrees, which can be formulated as ∇ f ( q ini ) · ∇ f ( q outj ) < 0 .
On the contrary, for two queries q ini and q inj in the same side, the formula ∇ f ( q ini ) · ∇ f ( q inj ) > 0 holds true.
So, we can classify whether two points
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
are in the same or the opposite side using dot product of gradients, cls ( q i , q j ) = ∇ f ( q i ) ·
∇ f ( q j ) .
Based on that, we divide the space into 3D grids (e.g. 256 3 ), and perform gradient discrimination on the 8 vertices v i in each cell grid according to the gradient field ∇ f using cls ( q i , q j ) .
As shown in
Fig. 6(a), the gradient field separates the vertices into two sets, where we can further adapt marching cubes algorithm [35] to create triangles for the grid using the lookup table.
The complete surface is generated by grouping triangles of each grid together.
To accelerate the surface extraction process and avoid extracting unexpected triangles in the multi-layers structures, we set a threshold θ to stop surface extraction on grids where f ( g i ) >
θ, i ∈ [0 , 7] .
Mesh refinement.
The initial surface extracted by marching cubes is only a discrete approximation of the zero iso-surface.
To achieve a more detailed mesh, we propose to refine it using the UDF values.
As shown in Fig. 6(b), given the predicted UDF values UDF A and UDF B of grid vertex v A and v b , the mesh vertex v M can be moved to a finer position where l A → M : l M → B =
UDF
A : UDF B .
Experiments
We evaluate our method on the task of surface reconstruction from raw point clouds.
We first demonstrate the ability of our method to reconstruct general shapes with open and multi-layer surfaces in Sec.4.1.
Next, we apply our method to reconstruct surfaces for real scanned raw data including 3D objects in Sec.4.2 and complex scenes in Sec.4.3.
Ablation studies are shown in Sec.4.4.
Implementation details.
To learn UDFs for raw point clouds P , we adopt a neural network similar to OccNet [40] to predict the unsigned distance given 3D queries as input.
Our network containslayers of MLP where each layer has 256 nodes.
Similar to Neural-Pull and SAL, given the single point cloud P as input, we do not leverage any condition and overfit the network to approximate the surface of P by minimizing the loss of Eq.
Therefore, we do not need to train our network on large scale training dataset in contrast to previous methods [12, 58, 26].
In addition, we use the same strategy as Neural-Pull to sample 60 queries around each point p
i on P as training data.
A Gaussian function N ( µ, σ 2 ) is adopt to calculate the sampling probability where µ = p i and σ is the distance between p i and its 50-th nearest points on P .
For sampling auxiliary points in the low confidence region, the standard deviation is set to 1 .
1 σ .
And we train our network for two stages in practice.
[Figure]
4.1 Surface Reconstruction for Synthetic Shapes
Dataset and metrics.
For the experiments on synthetic shapes, we follow NDF [12] to choose the “Car" category of the ShapeNet dataset which contains the greatest amount of multi-layer shapes and non-closed shapes.
And 10k points is sampled from the surface of each shape as the input.
Besides, we employ the MGD dataset [4] to show the advantage of our method in open surfaces.
To measure the reconstruction quality, we follow GIFS [58] to sample 100k points from the reconstructed surfaces and adopt the Chamfer distance ( × 10 4 ), Normal Consistency (NC) [40] and F-Score with a threshold of 0.005/0.01 as evaluation metrics.
[Figure]
[Table]
Comparison.
We compare our method with the state-of-the-art works NDF [12] and GIFS [58].
We quantitatively evaluate our method with NDF and
GIFS in Tab.
We also report the results of points sampled from the watertight ground truth (watertight GT in table) as the upper bound of the traditional SDF-based or Occupancy-based implicit functions.
To show the superior limit of this dataset, we sample two different sets of points from the ground truth mesh and report their results (GT in table).
For a comprehensive comparison with NDF, we transfer our gradient-based reconstruction algorithm to extract surfaces from the learned distance field of NDF, and report three metrics of NDF and our method including generated point cloud ( ∗ PC ), mesh generated using BPA ( ∗ BPA ) and mesh generated using our gradient-based reconstruction algorithm ( ∗ gradRA ).
As shown in Tab. 1, we achieve the best results in terms of all the metrics.
Moreover, our gradient-based reconstruction algorithm shows great generality in transferring to the learned gradient field of other method (e.g. NDF) by achieving significant improvement over the traditional method (BPA).
We also provide the results of surface reconstruction on MGD [4] dataset as shown in Tab. 2, where we largely outperform other methods.
We further present a visual comparison with SAL and NDF in Fig.
Previous methods (e.g. SAL) take SDF as output and are therefore limited to single-layer shapes where the inner-structure is lost.
NDF learns UDFs and is able to represent general shapes, but it outputs a dense point cloud and requires BPA to generate meshes, which leads to an uneven surface.
On the contrary, we can extract surfaces directly from the learned UDFs, which are continuous surfaces with high fidelity.
We also provide a visual comparison with Neural-Pull in MGD dataset as shown in Fig.
8, where we accurately reconstruct the open surfaces but Neural-Pull fails to keep the original geometry.
4.2 Surface Reconstruction for Real Scans
Dataset and metrics.
For surface reconstruction of real point cloud scans, we follow SAP to evaluate our methods under the Surface Reconstruction Benchmarks (SRB) [56].
We use Chamfer distance and F-Score with a threshold of 1% for evaluation.
Note that the ground truth is dense point clouds.
Comparison.
We compare our method with state-of-the-art classic and data-driven surface reconstruction methods in the real scanned SRB dataset, including IGR [20], Point2Mesh [23], Screened
Poisson Surface Reconstruction (SPSR) [29], Shape
As Points (SAP) [45], Neural-Pull [36] and NDF [12].
The numerical comparison is shown in Tab. 4, where we achieve the best accuracy.
The visual comparisons in Fig.
9 demonstrate that our method is able to reconstruct a continuous surface with local geometry consistence while other methods struggle to reveal the geometry details.
For example,
IGR, Neural-Pull and SAP mistakenly mended or failed to reconstruct the hole of the anchor while our method is able to keep the correct geometry.
[Table]
[Table] [Table]
Dataset and metrics.
To further demonstrate the advantage of our method in surface reconstruction of real scene scans, we follow OnSurf [37] to conduct experiments under the 3D Scene dataset [63].
Note that the 3D Scene dataset is a challenging real-world dataset with complex topology and noisy open surfaces.
We uniformly sample 100, 500 and 1000 points per m 2 at the original scale of scenes as the input and follow OnSurf to sample 1M points on both the reconstructed and the ground truth surfaces.
We leverage L1 and L2 Chamfer distance to evaluate the reconstruction quality.
Comparison.
We compare our method with the state-of-the-arts scene reconstruction methods
ConvONet [46], LIG [26], DeepLS [7], OnSurf [37] and NDF [12].
The numerical comparisons in Tab.
3 show that our method significantly outperform the other methods under different point densities.
The visual comparisons in Fig.
10 further shows that our reconstructions present more geometry details in complex real scene scans.
Note that all the other methods have been trained in a large scale dataset, from which they gain additional prior information.
On the contrary, our method does not leverage any additional priors or large scale training datasets, and learns to reconstruct surfaces directly from the raw point cloud, but still yields a non-trivial performance.
4.4 Ablation Study
We conduct ablation studies to justify the effectiveness of each design in our method and the effect of some important parameters.
We report the performance in terms of L2-CD under a subset of the
ShapeNet Car dataset.
By default, all the experimental settings are kept the same as in Sec. 4.1, except for modified part described in each ablation experiment below.
[Table]
Framework design.
We first justify the effectiveness of each design of our framework in Tab. 5.
We first directly use the loss proposed in Neural-Pull and find that the performance degenerates dramatically as shown by “NP loss".
We also use g ( x ) = 1 − e ( − x ) to replace g ( x ) =
| x | on the last layer of the network for f before output, but found no improvement as shown by “Exponent".
We train the second stage from scratch as shown by “Scratch" and prove that an end-to-end training strategy is more effective.
[Table]
The effect of stage numbers.
The number of stages during progressive surface approximation is also a crucial factor in the network training.
We report the performance of training our network in different number of stages St
=
[1 , 2 , 3 , 4] in Tab.
We start the training of next stage after the previous one converges.
We found that two stages training brings great improvement than training a single stage, and the improvements with 3rd and 4th stages are subtle.
[Table]
The effect of low confidence range.
We further explore the range of confidence region sample.
Assume σ as the range of high confidence region, we use 0.9 σ , 1.0 σ , 1.1 σ and 1.2 σ as the range of low confidence region.
The results in Tab.
7 show that a too small or too large range will degenerate the performance.
[Table]
Surface extraction.
We evaluate the effect of mesh refinement and the performance of different 3D grid resolutions.
Tab. 8 shows the accuracy and efficiency of different resolutions.
We observe that the mesh refinement highly improves the accuracy and higher resolutions leads to better reconstructions at a cost of speed.
Conclusion
We propose a novel method to learn continuous UDFs directly from raw point clouds by learning to move 3D queries to reach the approximated surface progressively.
Our introduced reconstruction algorithm can extract surfaces directly from the gradient fields of the learned UDFs.
Our method does not require ground truth distance values or point normals, and can reconstruct surfaces with arbitrary topology.
One limitation of our method is that we use uniformly divided grids to extract surface, which can be improved with a coarse-to-fine paradigm.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] We use public datasets, i.e. ShapeNet and 3D Scene, which can be dowmloaded from the websites.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]