Non-rigid Point Cloud Registration with Neural Deformation Pyramid
Abstract
Non-rigid point cloud registration is a key component in many computer vision and computer graphics applications.
The high complexity of the unknown non-rigid motion make this task a challenging problem.
In this paper, we break down this problem via hierarchical motion decomposition.
Our method called Neural Deformation Pyramid (NDP) represents non-rigid motion using a pyramid architecture.
Each pyramid level, denoted by a Multi-Layer Perception (MLP), takes as input a sinusoidally encoded 3D point and outputs its motion increments from the previous level.
The sinusoidal function starts with a low input frequency and gradually increases when the pyramid level goes down.
This allows a multi-level rigid to nonrigid motion decomposition and also speeds up the solving by 50 times compared to the existing MLP-based approach.
Our method achieves advanced partialto-partial non-rigid point cloud registration results on the 4DMatch/4DLoMatch benchmark under both no-learned and supervised settings. Code is available at https://github.com/rabbityl/DeformationPyramid.
Non-rigid point cloud registration is a key component in many computer vision and computer graphics applications.
The goal of non-rigid registration is to ﬁnd the transformation that maps one point cloud to another.
With the availability of consumer range sensors that can measure time-varying surface points, non-rigid registration has been applied to dynamic shape reconstruction problems such as human performance capture, enabling a wide range of applications in XR and robotics.
Non-rigid registration is a challenging problem.
First, 3D sensor measurements often contain noise, outliers, and occlusions.
Occlusions often lead to disconnection of point cloud geometry.
Point clouds may also have very low overlap ratios with each other due to the scene deformation and sensor’s viewpoint change.
The most challenging thing is, unlike rigid registration that only needs to determine the rotation and translation parameters, non-rigid registration needs to estimate the unknown movement of all points, making this problem especially complex to solve.
In this paper, we alleviate this complexity through motion decomposition.
We observe that natural non-rigid motion usually forms a hierarchical structure: with higher hierarchies representing the global movements and lower hierarchies representing the local deformation.
For instance, a walking person can be roughly approximated by three levels: 1) global location and orientation change, 2) local articulated movements from arms, legs, etc, and 3) ﬁne-grained cloth deformation caused by exterior forces.
Each level represents motion at a different scale and they have top-down dependencies.
Based on this observation, we propose a hierarchical motion representation called Neural Deformation
Pyramid (NDP) for non-rigid registration.
NDP has a pyramid architecture.
Each pyramid level contains a Multi-Layer Perception (MLP) that takes as input a sinusoidally encoded 3D point and outputs its motion increments from the previous level.
We found that the frequency of the sinusoidal function controls MLP’s capacity of representing non-rigidity: low frequencies yield smooth signals that are suitable for ﬁtting relatively rigid motion; high frequencies produce more ﬂuctuations that are capable of representing highly non-rigid motion.
We start the sinusoidal function at the ﬁrst pyramid level with a low frequency and gradually increase it when the pyramid level goes down.
This allows a multi-level rigid to non-rigid motion decomposition and also achieves over 50 times faster solving than the existing MLP-based approach.
The paper is about developing an MLP-based hierarchical deformation model for non-rigid point cloud registration.
The proposed method achieves state-of-the-art partial-to-partial non-rigid registration results on the challenging 4DMatch/4DLoMatch [23] benchmark under both no-learned and supervised settings.
We also demonstrate the application for shape transfer.
Related Work
Non-rigid point cloud registration.
Non-rigid point cloud registration is about estimating the deformation ﬁeld or assigning point-to-point mapping from one point cloud to another.
The simplest way is to estimate the point-wise parameters, such as afﬁne transform, under motion smoothness regularization [26].
Optimal transport-based registration method [14] ﬁnds displacement for each point using a global bijective-matching constraint.
Coherent Point Drift (CPD) [32, 18] constructs a 3D displacement ﬁeld using Gaussian Mixture Model (GMM) and encourages coherent motion of nearby points.
Deformation graph [39] represents the scene using a sparsely sub-sampled graph from the surface and propagates deformation from node to surface via “skinning”.
The Non-rigid Iterative
Closest Point (NICP) [20] achieve efﬁcient registration by optimizing the alignment energy and deformation graph regularization cost [38], and has been adopted in many real-time 4D reconstruction systems [33].
Li et al.
[22] and Bozic et al.
[5] learn dense feature alignment or correspondence re-weighting by differentiating through deformation graph-based non-rigid optimization.
Learningbased scene ﬂow estimation methods, e.g. FlowNet3D [11], use 3D siamese networks to regress the 3D displacement ﬁeld between two point clouds and reach real-time inference but are not robust under large deformations and ambiguities.
Lepard [23] use Transformer [42] to learn global pointto-point mapping and use it as landmark to guide global non-rigid registration.
Functional map approaches [34] estimate the correspondence in the spectral domain between computed LaplacianBeltrami basis functions, but usually assume connected surfaces thus not suitable for real-world partial point cloud data.
To alleviate this problem, Synorim [19] uses 3D CNN networks to estimate the basis functions and derive scene ﬂow from the functional mapping.
While existing methods mainly represent non-rigid motion or mapping at a single level, this paper propose a multi-level deformation model for non-rigid point cloud registration.
Pyramid for motion estimation.
Bouguet et al.
[4] show that the pyramid implementation of the Lucas-Kanade [7] method improves feature tracking.
PWCNet [40] estimates optical ﬂow on a pyramid of CNN feature maps.
PointPWC-net [43] adopts similar idea for scene ﬂow estimation from point cloud inputs.
Pyramid-based camera 6-Dof pose estimation can be found in SLAM systems [12].
ZoomOut [28] adopts upsampling in the spectral domain for shape correspondence estimation.
DynamicFusion [33] constructs a tree of deformation graph for non-rigid tracking, and shows that it stabilizes tracking and reduces computation cost.
DeepCap [16] combines an inner body pose with an outer cloth deformation for human performance capture.
This paper is about using MLP to create a hierarchical deformation pyramid for non-rigid point cloud registration of general scenes.
Motion ﬁeld with coordinate-MLP.
Coordinate-MLP uses an MLP to map input coordinates to signal values.
Coordinate-MLP is continuous and memory-efﬁcient.
It has shown promising results for representing 1D sound wave [37], 2D images [37], 3D shape [29], and radiance ﬁeld [30], etc.
Tancik et al.
[41] show that tuning the sinusoidal positional encoding of the input coordinate bias the network to ﬁtting low- or high-frequency signals.
Coordinate-MLP is also suitable for representing the deformation ﬁeld.
Close to our work, Li et al.
[25] and Li et al.
[21] use MLP to estimate scene ﬂow, Nerﬁes [35] and SAPE [17] propose a coarse-to-ﬁne motion ﬁeld MLP optimization technique by progressively expanding the frequency bandwidth of the input positional encoding.
However, the aforementioned motion ﬁeld MLPs are black box models that are designed to represent signals at a single scale, and usually need a large network to ﬁt complex motion, as a result their optimizations are usually time-consuming.
This paper decomposes the motion ﬁeld using a sequence of smaller
MLPs, achieving more interpretable and controllable motion representation and faster optimization.
Non-rigid Point Cloud Registration Notation
Given a source point cloud S
= { x i | x i ∈ R 3 , i = 1 , ..., n 1 } and a target point cloud T =
{ y j | y j ∈ R 3 , j = 1 , ..., n 2 } , where n 1 , n 2 are the number of points, our goal is to recover the non-rigid warp function W : R 3 (cid:55)→ R 3 that transforms points from S to T .
The simplest form of the warp function is a dense R 3 vector ﬁeld, which is also known as scene ﬂow.
Scene ﬂow is in theory sufﬁcient to represent any continuous deformation, but in practice, it can not ﬁt non-linear motions very well, such as 3D rotations.
We therefore formulate the non-rigid warp function using the SE(3) ﬁeld.
Dense SE(3) warp ﬁeld .
Given a globally non-rigidly deforming point cloud, we consider each individual point locally undergoes 3D rigid body movement.
A 3D rigid body transform (cid:18) R t 0 1 (cid:19) ∈
SE(3) denotes rotation and translation in 3D, with R ∈ SO(3) and t ∈ R 3 .
We parameterize rotations with a 3-dimensional axis-angle vector ω ∈ R 3 .
We use the exponential map exp : so (3) →
SO(3) ,
(cid:98) ω
(cid:55)→ e (cid:98) ω
=
R to convert from axis-angle to matrix rotation form, where the (cid:98) · -operator creates a 3 × 3 skew-symmetric matrix from a 3-dimensional vector.
The resulting 3D motion parameterization for a point x i ∈ R 3 is therefore denoted by ξ i
= ( ω i , t i ) ∈ R 6 , i.e. each point has 6 degrees of freedom (Dof).
The warp function reads
[Equation]
Dense Sim(3) warp ﬁeld .
This paper mainly focuses on registering point clouds captured from the same scene with the same scale.
For scale variant tasks such as inter-shape registration (c.f. Sec. 4.3), we extend the above warp function by incorporating a scaling factor s ∈ R + , resulting in a 3D similarity transform (cid:18) s R t (cid:19) ∈ Sim(3) .
With the parameterization ξ i
= ( s i , ω
i , t i ) ∈
R 7 , the warp function reads W ( x i , ξ i ) =
s i e (cid:98) ω
i x i + t i .
We shows ablation study of the dense warp ﬁeld types including R 3 , SE(3) , and Sim(3) , and rotations representations including Axis-angle, Euler-angle, Quanterion, and 6D [45] in Sec.
Neural Deformation Pyramid
As denoted Sec. 2, we aim to ﬁnd the motion parameter ξ i for each 3D point x
i in the source point cloud.
However, due to the high complexity and non-convexity of the non-rigid registration, directly estimating ξ i is usually difﬁcult and time-consuming.
Therefore, we want to decompose ξ i into a sequence of sub-transformations { ξ 1 i , ξ 2 i , ..., ξ mi } , such that each sub-transformation is easier to estimate, and when combining them we can get the same registration effect as ξ i .
To this end, we introduce Neural Deformation Pyramid (NDP) which allows hierarchical motion decomposition using a pyramid architecture.
3.1 Hierarchical motion decomposition.
As shown in Fig.
3, NDP is a pyramid of functions ∆ = { (Γ k , Θ k ) | k
= 1 , .., m } , where each pyramid level contains a pair of continuous functions (Γ k , Θ k ) , and m is the total number of levels.
At level k of the pyramid, Γ k is the positional encoding function that maps the input point from previous level x k − 1 i ∈ R 3 using the sinusoidal encoding
[Equation]
where k 0 is a constant that controls the initial frequency at the ﬁrst level of the pyramid.
The frequency of the sinusoidal function is enssential for motion decomposition: low frequencies yield smooth signals that are suitable for ﬁtting relatively rigid motion; high frequencies produce more ﬂuctuations that are capable of representing highly non-rigid motion.
By gradually increasing frequency with pyramid level k , we can achieve a hierarchical rigid-to-nonrigid motion decomposition.
Fig.
2 shows an example of hierarchical registration of of multiple scans of a Dinosaur.
Θ k is an optimizable MLP network.
It takes as input the encoded coordinate Γ k ( x k − 1 i ) and estimates the transformation increments at the current pyramid level.
Formally, we have
[Equation]
where ξ ki ∈ R 6 is the 6-Dof transformation parameter that is obtained from a liner output head of the
MLP.
α ki ∈ [0 , 1] is a scalar that represents the network’s conﬁdence in if the motion estimates are successful at the current level, it is obtained via a Sigmoid output head of the MLP.
Given the 6-Dof estimates ξ ki and conﬁdence α ki , we compose the transformed coordinate at the k -th level by
[Equation]
where W is the warp function as deﬁned in Eqn.
Note that α ki control the degree of deviation w.r.t to the previous more rigid pyramid level, thus we regard it as the level-wise deformability .
To encourage as-rigid-as-possible movement, we apply regularization terms on α ki which is shown in the next.
Merits of hierarchical motion decomposition.
1) it provides a more controllable and interpretable coordinate MLP-based deformation representation.
2) it simpliﬁes the task and speedup the optimization: we can use a small MLP for each level, because each level only needs to estimate the motion increments at a single frequency band, we also found that the overall convergence of the pyramid is faster than using a single coordinate-MLP with the full frequency band, this signiﬁcantly cut off the total optimization time compared to the existing MLP-based approaches, c.f. discussion in Sec. 4.2.
3.2 Cost function.
At level k , we denote the transformed source point cloud as S k = { x ki | i = 1 , ..., n 1 } .
The target point cloud is T =
{ y j | j = 1 , ..., n 2 } .
The cost functions at level k are:
Chamfer distance term.
Chamfer distance ﬁnds the nearest point in the other point cloud, and sums the square of distance.
Formally it is deﬁned as
[Equation]
where ρ ( ., . ) is the distance function, for which we use L 1 or L 2 norm.
We found that the robust Lnorm is more suitable for handling partial-to-partial registration, see ablation study in Section.
Correspondence term.
Given a putative correspondence set M , we minimize
[Equation]
where ( u, v ) ∈ M are the indices of the matched points in S k and T .
To obtain correspondence, we leverage the learning-based point cloud matching method Lepard [23], which predicts sparse point-to-point matches.
Lepard’s prediction contains a certain amount of outlier correspondences, which may lead to erroneous registration.
To alleviate this, we design a Transformer-based outlier rejection method, which takes as input matched coordinates pairs ( u, v ) ∈ R 6 and estimates their outlier probabilities.
The details can be seen in the supplemental material.
Deformability regularization term.
Given the deformability score α ki in Eqn. 4, we encourages zero predictions by minimizing the negative log-likelihood
[Equation]
This is to encourage as-rigid-as-possible movement.
We found this regularization help preserve the geometry of the point cloud given high frequency input signals.
Total cost function.
The total cost function E ktotal combines the above terms with the weighting factors λ cd , λ cor , and λ reg to balance them :
[Equation]
In the case that we use the correspondence term E cor , we denote our method as LNDP where ‘L’ indicates the Learned correspondences.
3.3 Non-rigid registration algorithm
Non-rigid registration using NDP is performed in a top-down way: the top-level MLP is ﬁrstly optimized, once it converges, proceeds to a lower level, we repeat this till all MLPs are optimized.
We use gradient descent as the optimizer.
Optimization of an MLP stops if 1) the max _ iter = 500 is reached, 2) a given registration cost threshold γ = 0 . 0001 is reached, or 3) the registration cost does not change for more than σ = 15 iterations.
Alg.
1 shows the pseudocode of the algorithm.
Algorithm
1: function R EGISTRATION ( S 0 , T ) (cid:46) S 0 denotes the raw source point cloud 2: for k ← 1 to m do 3: Θ k ← XavierUniform() (cid:46) MLP initialization 4: for iter ← 1 to max _ iter do 5:
S k ← TransformSourcePointCloud( S k − 1 , Γ k , Θ k ) 6:
E ktotal ← ComputeRegistrationCost( S k , T ) 7: if ConvergenceConditionSatisﬁed( E k total , iter ) then (cid:46) Early stop 8: break 9: Θ k ← GradientDecentSolver(Θ k , E ktotal )
(cid:46) Update MLP weights 10: return S m
Experiments 4.1 Benchmarking partial-to-partial non-rigid point cloud registration
4DMatch/4DLoMatch benchamrk.
4DMatch/4DLoMatch [23] is a benchmark for non-rigid point cloud registration.
It is constructed using animation sequences from DeformingThings4D [24].
This benchmark is extremely challenging due to the partial overlap, occlusions [Table] and large motion present in the data.
Point cloud pairs in this benchmark have a wide range of overlap ratios: 45% − 92% in 4DMatch and 15% − 45% in 4DLoMatch.
We found that the original benchmark contains a certain amount of examples that are dominated by rigid movement.
For fair evaluation, we remove data with near-rigid movements, please check the supplementary for details. [Table] [Table] NSFP
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
Nerﬁes [35]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
PointPWC
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
FLOT [36]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
GeomFmaps [9]
[Table]
[Table]
[Table]
[Table]
[Table]
Synorim
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Lepard
[Table]
NICP
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] Metrics. The metrics for evaluating non-rigid registration quality are 1) End-Point Error (EPE), i.e., the average norm of the 3D warp error vectors over all points, 2) 3D Accuracy Strict (AccS), the percentage of points whose relative error < 2.5% or < 2.5 cm, 3) 3D Accuracy Relaxed (AccR), the percentage of points whose relative error < 5% or < 5 cm, and 4) Outlier Ratio, the percentage of points whose relative error > 30%.
We consider AccS and AccR as the most important metrics as they exactly measure the ratio of accurately registered points.
Baselines.
We benchmark non-rigid registration using a number of baselines under both the nolearned and supervised settings.
We use the underlined names for brevity:
• No-Learned.
Point-to-point Iterative Closest Point (ICP)
[3] implemented in Open3D [44], Coherent Point Drift (CPD)[32] and its Bayesian formulation BCPD [18], ZoomOut [28], Sinkhorn optimal transport method implemented in Geomloss [14] and Keops [13], point-to-point Nonrigid ICP (NICP) [33], coordinate-MLP based approaches including Neural Scene Flow Prior (NSFP) [21] and Nerﬁes [35].
• Supervised.
Procrustes approach using Lepard [23]’s feature matching followed by SVD solver [1] (Lepard+SVD), Deep Geometric Maps (GeomFmaps)
[10], Synorim [19] in the pair-wise setting (Synorim-pw), scene ﬂow estimation methods including FLOT [36] and PointPWC [43], feature matching enhanced NICP as in [23] (Lepard+NICP).
All supervised models are re-trained on 4DMatch’s training split before evaluation.
Benchmarking results.
Tab. 1 shows the quantitative non-rigid registration results.
The coordinateMLP-based methods, including NSFP, Nerﬁes, and our NDP get clearly better results than other no-learned baselines.
This indicates the advantages of using the continuous coordinate-MLP to represent motion.
A major drawback of Nerﬁes and NSFP is that their optimization is usually time-consuming.
With hierarchical motion decomposition, our NDP runs around 50 times faster than
Nerﬁes and still be equally or more accurate.
Note that on 4DMatch, NDP even outperforms the supervised methods FLOT and PointPWC on the AccS/AccR metrics by a signiﬁcant margin.
On 4DLoMatch, due to the small point cloud overlap ( 15% − 45% ), none of the no-learned methods can produce a reasonable result.
LNDP obtains signiﬁcantly better non-rigid registration results than other supervised baselines.
Fig.
[Figure] and 4 shows the qualitative non-rigid registration results.
[Figure] [Figure] [Table]
4DMatch 4DLoMatch
[Table]
4.2 Ablation study
Pyramid level.
We test 9 pyramid levels with the initial frequency parameters in Eqn.
2 set [-tok 
-]to k 0 = − 8 .
Fig.
5 shows that, 1) the ﬁrst level only slightly surpasses the rigid registration baseline, 2) the registration accuracies gradually increase with the pyramid level, i.e., the capacity of representing non-rigid motion gradually grows, and 3) lower levels tend to need more iterations to converge.
Why is NDP faster than Nerﬁes?
Summing up the iterations from all levels (c.f. Fig. 5), NDP needs a total of 738 gradient decent iterations to converge.
The coarse-to-ﬁne optimization in Nerﬁes uses a single MLP with the full frequency band as input.
As a comparison, Nerﬁes needs 3792 iterations for convergence.
This indicates that our motion decomposition method speeds up the overall convergence.
In addition, to represent complex motions, Nerﬁes needs a large MLP of ( witdth, depth ) =
As a comparison, for each pyramid level we use a small MLP of ( witdth, depth ) =
(128 , 3) , because each level only needs to estimate the motion increments at a single frequency band.
Note that NDP only queries a single level MLP at an iteration (c.f.
Algorihtm. 1).
As a result, the overall computation overhead of NDP is much smaller than Nerﬁes.
L 1 norm vs L 2 norm for partial registration.
We show in Table. 2 that the L 1 norm is more suitable for partial-to-partial registration, because it is more robust to large errors.
For example, the chamber distance term is based on the sum of the squared distance from each point to the other model. Minimizing this loss would attempt to reduce the distance for every point, including those that do not correspond to any point on the other point cloud due to the partial overlap.
Compared to the L 2 norm, the L 1 norm allow for large distances on some points.
Similarly, the L 1 norm is also more tolerant to outliers in the correspondence term.
Motion ﬁeld type.
Tab.
3 shows that, 1) SE(3) warp ﬁeld gets better results than Sim(3) ﬁeld and Rvector ﬁeld, and 2) Sim(3) ﬁeld requires the most iterations to converge, possibly because it needs to estimate the extra scale factor.
Rotation representation.
Tab.
3 shows that Axis-angle and Euler angle get similar results and are better and converge faster than Quanterion and 6D.
4.3 Scale variant registration with Sim(3) warp ﬁeld
In Fig.
6, we provide examples of the “shape transfer” as an application of our non-rigid registration method.
To handle scale change between different shapes, we employ the dense Sim(3) warp ﬁeld (c.f.
Sec. 2).
The dense formulation allows NDP to reﬂect different scale changes at different regions of the body, e.g. character A ’s stomach is expanding while the legs are shrinking.
The original mesh vertices of the shapes are unevenly distributed, which is not suitable for computing the chamfer distance cost as in Eqn.
5, therefore we use uniformly sub-sampled point clouds from the mesh’s surface as input points.
Since NDP is a continuous function, the deformed mesh can be obtained by querying NDP using the original mesh’s vertices.
We use the registration parameters ( m, k 0 ) =
(9 , − 8) , this allows the transformed shapes roughly match the target shapes while retaining the geometrical details of the source shapes.
Conclusion
We show that non-rigid point cloud registration can be decomposed into a hierarchical motion estimation schema by stacking coordinate networks with growing input frequency.
Our method demonstrates superior non-rigid registration results on the 4DMatch partial-to-partial non-rigid registration benchmark under both no-learned and supervised settings.
Our method runs overtimes faster than the existing coordinate-networks-based approach.
Limitations.
1) NDP uses input coordinates that are deﬁned over the 3D Euclidean space, therefore it can not handle topological changes very well, extending NDP to the manifold surface space would be interesting. Common to the unsupervised approach, NDP does not handle non-isometric cases.
A failure case could be seen in Figure.
6: i.e. A’s chest is mapped to B’s belly. 3) Though NDP runs about 50 times faster than Nerﬁes, the current implementation still does not run at a real-time rate, further speedup could leverage the tiny CUDA neural network framework [31].
4) Finally, non-rigid registration in the low-overlap cases, such as examples in 4DLoMatch, is still challenging, our method cannot solve all the cases.
Broader Impact.
Our paper presents non-rigid registration, which is needed for a variety of applications ranging from XR to robotics.
In the former, a precise modeling of dynamic and deformable objects is of major importance to provide an immersive experience to the user.
In the later, localizing and understanding dynamic objects such as humans and animals in the envoriment using 3D sensor is essential for safe and intellegent robot operation.
On the other hand, as a low-level building block, our work has no direct negative outcome, other than what could arise from the aforementioned applications.
Acknowledgments and Disclosure of Funding
This work was partially supported by JST AIP Acceleration Research JPMJCR20U3, Moonshot
R&D Grant Number JPMJPS2011, CREST Grant Number JPMJCR2015, JSPS KAKENHI Grant
Number JP19H01115 and Basic Research Grant (Super AI) of Institute for AI and Beyond of the University of Tokyo.
I. Introduction
Supplemental document includes
• Additional ablation studies, see.
II.. • Results on real-world benchmarks: DeepDeform and KITTI, see III.
• Details of our outlier rejection approach, see.
IV.. • Details of data ﬁlter in 4DMatch benchmark, see V..
II. Ablation Study
Ablation study of the network weights initialization. Changing the initialization strategy could greatly affects the results of registration.
Table.
I shows an ablation study for network initialization via Pytorch built-in functions.
With .ones_ initialization, the model does not converge at all. .kaiming_uniform _ and .xavier_uniform_ initialization produce similar results. [Table]
In addition, we employ a trick to constraint the output of MLP: we apply a small scaling factor of 0 . 0001 on the output of MLP, this encourages the MLP to produce a near-identity SE (3) matrix at the beginning of optimization. We found this trick is crucial for NDP (no-learned) but not necessary for LNDP (supervised). Computation time at a different number of input points.
The time complexity of our registration algorithm is sub-linear or O (1) with the number of points. The dominant computation overhead is from the network optimization, the time complexity is O ( n × m ) , where n is the number of points and m is the number of iterations required for convergence. If m is ﬁxed, the time grows in of the number of points.
However, we found that when increasing training point n , the total number of iteration m decreases. Therefore, if we use all points for optimization, the registration time grows sub-linearly.
For faster registration, we can optimize only sub-sampled points. This keeps a constant registration time regardless of the input size. Table. II proves this argument
[Table] [Table]
Registration result at different level of noise. Table. III shows the registration accuracy of NDP under different ratios of point cloud noise. A noisy point is created via uniform perturbation of a clean point inside a ball with a radius=0.5m (the size of objects in 4DMatch range from 0.6m to 2.1m).
We do not observe a signiﬁcant performance drop until the introduction of 25% noise, while real-world range sensors such as the Kinect1 camera only produces 4% - 6% noise, see [27]. This experiment, combined with the results on DeepDeform and
KITTI, proves that our method is robust to noisy data in real-world scenarios.
[Table]
Experiment
different level of Overlap/Partiality/Occlusion. We want to clarify that, in the pair wise setting, the three terms: overlap, partiality, and occlusion are connected.
Given Overlap ratio = θ , we can obtain the others by Partiality = θ , and Occlusion Ratio = (1 − θ ) . Because overlap ratio is deﬁned by the the percentage of co-visible point between a pair of point clouds, by this deﬁnition, overlap ratio exactly represents the relative partiality between two point clouds.
Occlusion ratio denotes the ratio of points that are invisible from another point cloud, therefore it is can be computed by (1 − θ ) .
Table. II. shows the stats of in 4DMatch/4DLoMatch. Note that our method is state-of-the-art on this benchmark, c.f. Table. Overlap ratio / Partiality ( θ )
Occlusion ratio ( 1 − θ ) 4DMatch 45% ∼ 92% 8% ∼ 55% 4DLoMatch 15% ∼ 45%
[Table]
As shown in Table.
IV [Table] we further re-group the results on 4DMatch/4DLoMatch based on different ranges of with 20% interval. [Table]
III. Results on DeepDeform [6] and KITTI [15] benchmark
We report results on two real-world benchmarks: DeepDeform [6], and KITTI Scene Flow [15]. DeepDeform contains real-world partial RGB-D scans of dynamic objects, including humans, animals, cloth, etc. KITTI Scene Flow dataset contains Lidar scans in dynamic autonomous driving scenes. Table.
VI and V shows the quantitative registration results and Figure. III. and II show the qualitative results. Our NDP produces advanced results on both benchmarks.
In particular, the Lidar scans in KITTI capture running vehicles, walking pedestrians, and static trees/buildings on the road, i.e., the data is cluttered, and breaks the isometric deformation assumption. The point density of Lidar scans also changes drastically from near to far. NDP is robust to the above factors and produces competitive results.
[Table] [Table] IV. Outlier rejection using Transformer It is necessary to reject poor correspondences for robust non-rigid registration.
Existing learning-based approaches, such as DGR [8] and PointDSC [2], formulate outlier rejection as a binary classiﬁcation problem, i.e. they use a network to predict the outlier probability of each corresponding point pair. Inspired by these works, we use Transformer to reject outliers. As discussed in DGR [8], inlier correspondences between two rigid scans should form a manifold geometry in 6D space. Similarly, inlier correspondences in non-rigid scenes should also form a submanifold in 6D, because natural deformations are usually locally rigid.
Therefore, the motivation is to learn the geometry of inliers in 6D space using transformers. Fig. III shows the overview of our outlier rejection method. It takes as input an array of 6D coordinates, denoting the corresponding point pairs, and predicts the inlier probability of each correspondence.
We obtain the input correspondence using Lepard [23]. The network consists of a sequence of multi-head self-attention (SA) layers [42], with the head number set to 8. To leverage the spatial consistency of the correspondence, we modify the self-attention layer using the spatial-consistencyguided attention as proposed in PointDSC [2]. The output of the last SA layer is activated through the Sigmoid function that maps the output to range [0 , 1] as the per-correspondence conﬁdence score.
We use weighted binary cross-entropy (BCE) as the loss function for training. The network is trained, validated, and tested on the 4DMatch [23] split. When testing, we treat correspondence with a conﬁdence score below the threshold θ = 0 . 3 as outliers.
Fig. IV shows the inﬂuence of the threshold θ for non-rigid registration. Number of SA layer. Tab.
VII shows the ablation study of number of self-attention (SA) layer. Precision grows with the number of the SA layer. The recall rate becomes stable after using 3 or more layers. By default, our method uses
[Table] [Table]
Outlier rejection conﬁdence. The precision and recall rate in Tab. VII can not reﬂect the end-toend performance for non-rigid registration. We want to ﬁnd a proper outlier rejection threshold for non-rigid registration.
Fig. IV shows the ablation study of the conﬁdence threshold for non-rigid registration accuracy. The optimal conﬁdence threshold lies around 20%-30%. A threshold higher than 60% leads to worse results than not rejecting outliers.
Outlier rejection for registration. Rejecting correspondence outliers gains +6 . 83% AccS on 4DMatch/4DLoMatch. See the supplementary for details of outlier rejection.
[Table] [Figure] 4DLoMatch
[Figure]
We observe that the original 4DMatch/4DLoMatch benchmarks contain a certain amount of entries that are dominated by rigid motion, i.e. the best rigid ﬁtting already registers most of the points. For fair benchmarking of non-rigid registration, we probabilistically remove point cloud pairs that have near-rigid movements.
In the end, we removed around 50% of the entries.
Fig.
V shows the histogram of the benchmark before and after ﬁltering.