Non-rigid Point Cloud Registration with Neural Deformation Pyramid
Abstract
Non-rigid point cloud registration is a key component in many computer vision and computer graphics applications.
The high complexity of the unknown non-rigid motion make this task a challenging problem.
In this paper, we break down this problem via hierarchical motion decomposition.
Our method called Neural Deformation Pyramid (NDP) represents non-rigid motion using a pyramid architecture.
Each pyramid level, denoted by a Multi-Layer Perception (MLP), takes as input a sinusoidally encoded 3D point and outputs its motion increments from the previous level.
The sinusoidal function starts with a low input frequency and gradually increases when the pyramid level goes down.
This allows a multi-level rigid to nonrigid motion decomposition and also speeds up the solving by 50 times compared to the existing MLP-based approach.
Our method achieves advanced partialto-partial non-rigid point cloud registration results on the 4DMatch/4DLoMatch benchmark under both no-learned and supervised settings.
Code is available at https://github.com/rabbityl/DeformationPyramid.
Non-rigid point cloud registration is a key component in many computer vision and computer graphics applications.
The goal of non-rigid registration is to find the transformation that maps one point cloud to another.
With the availability of consumer range sensors that can measure time-varying surface points, non-rigid registration has been applied to dynamic shape reconstruction problems such as human performance capture, enabling a wide range of applications in XR and robotics.
Non-rigid registration is a challenging problem.
First, 3D sensor measurements often contain noise, outliers, and occlusions.
Occlusions often lead to disconnection of point cloud geometry.
Point clouds may also have very low overlap ratios with each other due to the scene deformation and sensor’s viewpoint change.
The most challenging thing is, unlike rigid registration that only needs to determine the rotation and translation parameters, non-rigid registration needs to estimate the unknown movement of all points, making this problem especially complex to solve
In this paper, we alleviate this complexity through motion decomposition.
We observe that natural non-rigid motion usually forms a hierarchical structure: with higher hierarchies representing the global movements and lower hierarchies representing the local deformation.
For instance, a walking person can be roughly approximated by three levels: 1) global location and orientation change, 2) local articulated movements from arms, legs, etc, and 3) fine-grained cloth deformation caused by exterior forces.
Each level represents motion at a different scale and they have top-down dependencies.
Based on this observation, we propose a hierarchical motion representation called Neural Deformation
Pyramid (NDP) for non-rigid registration.
NDP has a pyramid architecture.
Each pyramid level contains a Multi-Layer Perception (MLP) that takes as input a sinusoidally encoded 3D point and outputs its motion increments from the previous level.
We found that the frequency of the sinusoidal function controls MLP’s capacity of representing non-rigidity: low frequencies yield smooth signals that are suitable for fitting relatively rigid motion; high frequencies produce more fluctuations that are capable of representing highly non-rigid motion.
We start the sinusoidal function at the first pyramid level with a low frequency and gradually increase it when the pyramid level goes down.
This allows a multi-level rigid to non-rigid motion decomposition and also achieves over 50 times faster solving than the existing MLP-based approach.
The paper is about developing an MLP-based hierarchical deformation model for non-rigid point cloud registration.
The proposed method achieves state-of-the-art partial-to-partial non-rigid registration results on the challenging 4DMatch/4DLoMatch [19] benchmark under both no-learned and supervised settings.
We also demonstrate the application for shape transfer.
Related Work
Non-rigid point cloud registration.
Non-rigid point cloud registration is about estimating the deformation field or assigning point-to-point mapping from one point cloud to another.
The simplest way is to estimate the point-wise parameters, such as affine transform, under motion smoothness regularization [22].
Optimal transport-based registration method [11] finds displacement for each point using a global bijective-matching constraint.
Coherent Point Drift (CPD) [27, 14] constructs a 3D displacement field using Gaussian Mixture Model (GMM) and encourages coherent motion of nearby points.
Deformation graph [34] represents the scene using a sparsely sub-sampled graph from the surface and propagates deformation from node to surface via “skinning”.
The Non-rigid Iterative
Closest Point (NICP) [16] achieve efficient registration by optimizing the alignment energy and deformation graph regularization cost [33], and has been adopted in many real-time 4D reconstruction systems [28].
Li et al.
[18] and Bozic et al.
[4] learn dense feature alignment or correspondence re-weighting by differentiating through deformation graph-based non-rigid optimization.
Learningbased scene flow estimation methods, e.g. FlowNet3D [8], use 3D siamese networks to regress the 3D displacement field between two point clouds and reach real-time inference but are not robust under large deformations and ambiguities.
Lepard [19] use Transformer [37] to learn global pointto-point mapping and use it as landmark to guide global non-rigid registration.
Functional map approaches [29] estimate the correspondence in the spectral domain between computed LaplacianBeltrami basis functions, but usually assume connected surfaces thus not suitable for real-world partial point cloud data.
To alleviate this problem, Synorim [15] uses 3D CNN networks to estimate the basis functions and derive scene flow from the functional mapping.
While existing methods mainly represent non-rigid motion or mapping at a single level, this paper propose a multi-level deformation model for non-rigid point cloud registration.
Pyramid for motion estimation.
Bouguet et al.
[3] show that the pyramid implementation of the Lucas-Kanade [5] method improves feature tracking.
PWCNet [35] estimates optical flow on a pyramid of CNN feature maps.
PointPWC-net [38] adopts similar idea for scene flow estimation from point cloud inputs.
Pyramid-based camera 6-Dof pose estimation can be found in SLAM systems [9].
ZoomOut [23] adopts upsampling in the spectral domain for shape correspondence estimation.
DynamicFusion [28] constructs a tree of deformation graph for non-rigid tracking, and shows that it stabilizes tracking and reduces computation cost.
DeepCap [12] combines an inner body pose with an outer cloth deformation for human performance capture.
This paper is about using MLP to create a hierarchical deformation pyramid for non-rigid point cloud registration of general scenes.
Motion field with coordinate-MLP.
Coordinate-MLP uses an MLP to map input coordinates to signal values.
Coordinate-MLP is continuous and memory-efficient.
It has shown promising results for representing 1D sound wave [32], 2D images [32], 3D shape [24], and radiance field [25], etc.
Tancik et al.
[36] show that tuning the sinusoidal positional encoding of the input coordinate bias the network to fitting low- or high-frequency signals.
Coordinate-MLP is also suitable for representing the deformation field.
Close to our work, Li et al.
[21] and Li et al.
[17] use MLP to estimate scene flow, Nerfies [30] and SAPE [13] propose a coarse-to-fine motion field MLP optimization technique by progressively expanding the frequency bandwidth of the input positional encoding.
However, the aforementioned motion field MLPs are black box models that are designed to represent signals at a single scale, and usually need a large network to fit complex motion, as a result their optimizations are usually time-consuming.
This paper decomposes the motion field using a sequence of smaller
MLPs, achieving more interpretable and controllable motion representation and faster optimization.
Non-rigid Point Cloud Registration Notation
Given a source point cloud S
= { x i | x i ∈ R 3 , i = 1 , ..., n 1 } and a target point cloud T =
{ y j | y j ∈ R 3 , j = 1 , ..., n 2 } , where n 1 , n 2 are the number of points, our goal is to recover the non-rigid warp function W : R 3 (cid:55)→ R 3 that transforms points from S to T .
The simplest form of the warp function is a dense R 3 vector field, which is also known as scene flow.
Scene flow is in theory sufficient to represent any continuous deformation, but in practice, it can not fit non-linear motions very well, such as 3D rotations.
We therefore formulate the non-rigid warp function using the SE(3) field.
Dense SE(3) warp field .
Given a globally non-rigidly deforming point cloud, we consider each individual point locally undergoes 3D rigid body movement.
A 3D rigid body transform (cid:18) R t 0 1 (cid:19) ∈
SE(3) denotes rotation and translation in 3D, with R ∈ SO(3) and t ∈ R 3 .
We parameterize rotations with a 3-dimensional axis-angle vector ω ∈ R 3 .
We use the exponential map exp : so (3) →
SO(3) ,
(cid:98) ω
(cid:55)→ e (cid:98) ω
=
R to convert from axis-angle to matrix rotation form, where the (cid:98) · -operator creates a 3 × 3 skew-symmetric matrix from a 3-dimensional vector.
The resulting 3D motion parameterization for a point x i ∈ R 3 is therefore denoted by ξ i
= ( ω i , t i ) ∈ R 6 , i.e. each point has 6 degrees of freedom (Dof).
The warp function reads
[Equation]
Dense Sim(3) warp field .
This paper mainly focuses on registering point clouds captured from the same scene with the same scale.
For scale variant tasks such as inter-shape registration (c.f. Sec. 4.3), we extend the above warp function by incorporating a scaling factor s ∈ R + , resulting in a 3D similarity transform (cid:18) s R t (cid:19) ∈ Sim(3) .
With the parameterization ξ i
= ( s i , ω
i , t i ) ∈
R 7 , the warp function reads W ( x i , ξ i ) =
s i e (cid:98) ω
i x i + t i .
We shows ablation study of the dense warp field types including R 3 , SE(3) , and Sim(3) , and rotations representations including Axis-angle, Euler-angle, Quanterion, and 6D [40] in Sec.
Neural Deformation Pyramid
As denoted Sec. 2, we aim to find the motion parameter ξ i for each 3D point x
i in the source point cloud.
However, due to the high complexity and non-convexity of the non-rigid registration, directly estimating ξ i is usually difficult and time-consuming.
Therefore, we want to decompose ξ i into a sequence of sub-transformations { ξ 1 i , ξ 2 i , ..., ξ mi } , such that each sub-transformation is easier to estimate, and when combining them we can get the same registration effect as ξ i .
To this end, we introduce Neural Deformation Pyramid (NDP) which allows hierarchical motion decomposition using a pyramid architecture.
3.1 Hierarchical motion decomposition.
As shown in Fig.
3, NDP is a pyramid of functions ∆ = { (Γ k , Θ k ) | k
= 1 , .., m } , where each pyramid level contains a pair of continuous functions (Γ k , Θ k ) , and m is the total number of levels.
At level k of the pyramid, Γ k is the positional encoding function that maps the input point from previous level x k − 1 i ∈ R 3 using the sinusoidal encoding
[Equation]
where k 0 is a constant that controls the initial frequency at the first level of the pyramid.
The frequency of the sinusoidal function is enssential for motion decomposition: low frequencies yield smooth signals that are suitable for fitting relatively rigid motion; high frequencies produce more fluctuations that are capable of representing highly non-rigid motion.
By gradually increasing frequency with pyramid level k , we can achieve a hierarchical rigid-to-nonrigid motion decomposition.
Fig.
2 shows an example of hierarchical registration of of multiple scans of a Dinosaur.
Θ k is an optimizable MLP network.
It takes as input the encoded coordinate Γ k ( x k − 1 i ) and estimates the transformation increments at the current pyramid level.
Formally, we have
[Equation]
where ξ ki ∈ R 6 is the 6-Dof transformation parameter that is obtained from a liner output head of the
MLP.
α ki ∈ [0 , 1] is a scalar that represents the network’s confidence in if the motion estimates are successful at the current level, it is obtained via a Sigmoid output head of the MLP.
Given the 6-Dof estimates ξ ki and confidence α ki , we compose the transformed coordinate at the k -th level by
[Equation]
where W is the warp function as defined in Eqn.
Note that α ki control the degree of deviation w.r.t to the previous more rigid pyramid level, thus we regard it as the level-wise deformability .
To encourage as-rigid-as-possible movement, we apply regularization terms on α ki which is shown in the next.
Merits of hierarchical motion decomposition.
1) it provides a more controllable and interpretable coordinate MLP-based deformation representation.
2) it simplifies the task and speedup the optimization: we can use a small MLP for each level, because each level only needs to estimate the motion increments at a single frequency band, we also found that the overall convergence of the pyramid is faster than using a single coordinate-MLP with the full frequency band, this significantly cut off the total optimization time compared to the existing MLP-based approaches, c.f. discussion in Sec. 4.2.
3.2 Cost function.
At level k , we denote the transformed source point cloud as S k = { x ki | i = 1 , ..., n 1 } .
The target point cloud is T =
{ y j | j = 1 , ..., n 2 } .
The cost functions at level k are:
Chamfer distance term.
Chamfer distance finds the nearest point in the other point cloud, and sums the square of distance.
Formally it is defined as
[Equation]
where ρ ( ., . ) is the distance function, for which we use L 1 or L 2 norm.
We found that the robust Lnorm is more suitable for handling partial-to-partial registration, see ablation study in Section.
Correspondence term.
Given a putative correspondence set M , we minimize
[Equation]
where ( u, v ) ∈ M are the indices of the matched points in S k and T .
To obtain correspondence, we leverage the learning-based point cloud matching method Lepard [19], which predicts sparse point-to-point matches.
Lepard’s prediction contains a certain amount of outlier correspondences, which may lead to erroneous registration.
To alleviate this, we design a Transformer-based outlier rejection method, which takes as input matched coordinates pairs ( u, v ) ∈ R 6 and estimates their outlier probabilities.
The details can be seen in the supplemental material.
Deformability regularization term.
Given the deformability score α ki in Eqn. 4, we encourages zero predictions by minimizing the negative log-likelihood
[Equation]
This is to encourage as-rigid-as-possible movement.
We found this regularization help preserve the geometry of the point cloud given high frequency input signals.
Total cost function.
The total cost function E ktotal combines the above terms with the weighting factors λ cd , λ cor , and λ reg to balance them :
[Equation]
In the case that we use the correspondence term E cor , we denote our method as LNDP where ‘L’ indicates the Learned correspondences.
3.3 Non-rigid registration algorithm
Non-rigid registration using NDP is performed in a top-down way: the top-level MLP is firstly optimized, once it converges, proceeds to a lower level, we repeat this till all MLPs are optimized.
We use gradient descent as the optimizer.
Optimization of an MLP stops if 1) the max _ iter = 500 is reached, 2) a given registration cost threshold γ = 0 . 0001 is reached, or 3) the registration cost does not change for more than σ = 15 iterations.
Alg.
1 shows the pseudocode of the algorithm.
Algorithm
1: function R EGISTRATION ( S 0 , T ) ▷ S 0 denotes the raw source point cloud 2: for k ← 1 to m do 3: Θ k ← XavierUniform() ▷ MLP initialization 4: for iter ← 1 to max _ iter do 5:
S k ← TransformSourcePointCloud( S k − 1 , Γ k , Θ k ) 6:
E ktotal ← ComputeRegistrationCost( S k , T ) 7: if ConvergenceConditionSatisfied( E k total , iter ) then ▷ Early stop 8: break 9: Θ k ← GradientDecentSolver(Θ k , E ktotal )
▷ Update MLP weights 10: return S m
Experiments 4.1 Benchmarking partial-to-partial non-rigid point cloud registration
4DMatch/4DLoMatch benchamrk.
4DMatch/4DLoMatch [19] is a benchmark for non-rigid point cloud registration.
It is constructed using animation sequences from DeformingThings4D [20].
This benchmark is extremely challenging due to the partial overlap, occlusions
[Table]
and large motion present in the data.
Point cloud pairs in this benchmark have a wide range of overlap ratios: 45% − 92% in 4DMatch and 15% − 45% in 4DLoMatch.
We found that the original benchmark contains a certain amount of examples that are dominated by rigid movement.
For fair evaluation, we remove data with near-rigid movements, please check the supplementary for details.
[Table] [Table]
NSFP
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table]
[Table]
[Table]
Nerfies
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
FLOT
[Table] [Table]
[Table]
[Table] [Table]
[Table] [Table]
[Table]
[Table]
[Table]
Synorim
[Table]
[Table]
[Table] [Table]
Lepard
[Table]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Metrics.
The metrics for evaluating non-rigid registration quality are 1) End-Point Error (EPE), i.e., the average norm of the 3D warp error vectors over all points, 2) 3D Accuracy Strict (AccS), the percentage of points whose relative error < 2.5% or < 2.5 cm, 3) 3D Accuracy Relaxed (AccR), the percentage of points whose relative error < 5% or < 5 cm, and 4) Outlier Ratio, the percentage of points whose relative error > 30%.
We consider AccS and AccR as the most important metrics as they exactly measure the ratio of accurately registered points.
Baselines.
We benchmark non-rigid registration using a number of baselines under both the nolearned and supervised settings.
We use the underlined names for brevity:
• No-Learned.
Point-to-point Iterative Closest Point (ICP)
[2] implemented in Open3D [39], Coherent Point Drift (CPD)[27] and its Bayesian formulation BCPD [14], ZoomOut [23], Sinkhorn optimal transport method implemented in Geomloss [11] and Keops [10], point-to-point Nonrigid ICP (NICP) [28], coordinate-MLP based approaches including Neural Scene Flow Prior (NSFP) [17] and Nerfies [30].
• Supervised.
Procrustes approach using Lepard [19]’s feature matching followed by SVD solver [1] (Lepard+SVD), Deep Geometric Maps (GeomFmaps)
[7], Synorim [15] in the pair-wise setting (Synorim-pw), scene flow estimation methods including FLOT [31] and PointPWC [38], feature matching enhanced NICP as in [19] (Lepard+NICP).
All supervised models are re-trained on training split evaluation.
Benchmarking results.
Tab. 1 shows the quantitative non-rigid registration results.
The coordinateMLP-based methods, including NSFP, Nerfies [Figure] and our NDP get clearly better results than other no-learned baselines.
This indicates the advantages of using the continuous coordinate-MLP to represent motion.
A major drawback of Nerfies and NSFP is that their optimization is usually time-consuming.
With hierarchical motion decomposition, our NDP runs around 50 times faster than
Nerfies and still be equally or more accurate.
Note that on 4DMatch, NDP even outperforms the supervised methods FLOT and PointPWC on the AccS/AccR metrics by a significant margin.
On 4DLoMatch, due to the small point cloud overlap ( 15% − 45% ), none of the no-learned methods can produce a reasonable result.
LNDP obtains significantly better non-rigid registration results than other supervised baselines.
Fig.
[Figure]
and 4 shows the qualitative non-rigid registration results.
[Figure] [Figure]
[Table]
[Table]
4.2 Ablation study
Pyramid level.
We test 9 pyramid levels with the initial frequency parameters in Eqn.
2 set to k 0 = − 8 .
Fig.
5 shows that, 1) the first level only slightly surpasses the rigid registration baseline, 2) the registration accuracies gradually increase with the pyramid level, i.e., the capacity of representing non-rigid motion gradually grows, and 3) lower levels tend to need more iterations to converge.
Why is NDP faster than Nerfies?
Summing up the iterations from all levels (c.f. Fig. 5), NDP needs a total of 738 gradient decent iterations to converge.
The coarse-to-fine optimization in Nerfies uses a single MLP with the full frequency band as input.
As a comparison, Nerfies needs 3792 iterations for convergence.
This indicates that our motion decomposition method speeds up the overall convergence.
In addition, to represent complex motions, Nerfies needs a large MLP of ( witdth, depth ) =
As a comparison, for each pyramid level we use a small MLP of ( witdth, depth ) =
(128 , 3) , because each level only needs to estimate the motion increments at a single frequency band.
Note that NDP only queries a single level MLP at an iteration (c.f.
Algorihtm. 1).
As a result, the overall computation overhead of NDP is much smaller than Nerfies.
L 1 norm vs L 2 norm for partial registration.
We show in Table.
2 that the L 1 norm is more suitable for partial-to-partial registration, because it is more robust to large errors.
For example, the chamber distance term is based on the sum of the squared distance from each point to the other model.
Minimizing this loss would attempt to reduce the distance for every point, including those that do not correspond to any point on the other point cloud due to the partial overlap.
Compared to the L 2 norm, the L 1 norm allow for large distances on some points.
Similarly, the L 1 norm is also more tolerant to outliers in the correspondence term.
Motion field type.
Tab.
3 shows that, 1) SE(3) warp field gets better results than Sim(3) field and Rvector field, and 2) Sim(3) field requires the most iterations to converge, possibly because it needs to estimate the extra scale factor.
Rotation representation.
Tab.
3 shows that Axis-angle and Euler angle get similar results and are better and converge faster than Quanterion and 6D.
4.3 Scale variant registration with Sim(3) warp field
In Fig.
6, we provide examples of the “shape transfer” as an application of our non-rigid registration method.
To handle scale change between different shapes, we employ the dense Sim(3) warp field (c.f.
Sec. 2).
The dense formulation allows NDP to reflect different scale changes at different regions of the body, e.g. character A ’s stomach is expanding while the legs are shrinking.
The original mesh vertices of the shapes are unevenly distributed, which is not suitable for computing the chamfer distance cost as in Eqn.
5, therefore we use uniformly sub-sampled point clouds from the mesh’s surface as input points.
Since NDP is a continuous function, the deformed mesh can be obtained by querying NDP using the original mesh’s vertices.
We use the registration parameters ( m, k 0 ) =
(9 , − 8) , this allows the transformed shapes roughly match the target shapes while retaining the geometrical details of the source shapes.
Conclusion
We show that non-rigid point cloud registration can be decomposed into a hierarchical motion estimation schema by stacking coordinate networks with growing input frequency.
Our method demonstrates superior non-rigid registration results on the 4DMatch partial-to-partial non-rigid registration benchmark under both no-learned and supervised settings.
Our method runs overtimes faster than the existing coordinate-networks-based approach.
Limitations.
1) NDP uses input coordinates that are defined over the 3D Euclidean space, therefore it can not handle topological changes very well, extending NDP to the manifold surface space would be interesting.
Common to the unsupervised approach, NDP does not handle non-isometric cases.
A failure case could be seen in Figure.
6: i.e. A’s chest is mapped to B’s belly.
3) Though NDP runs about 50 times faster than Nerfies, the current implementation still does not run at a real-time rate, further speedup could leverage the tiny CUDA neural network framework [26].
Finally, non-rigid registration in the low-overlap cases, such as examples in 4DLoMatch, is still challenging, our method cannot solve all the cases.
Broader Impact.
Our paper presents non-rigid registration, which is needed for a variety of applications ranging from XR to robotics.
In the former, a precise modeling of dynamic and deformable objects is of major importance to provide an immersive experience to the user.
In the later, localizing and understanding dynamic objects such as humans and animals in the envoriment using 3D sensor is essential for safe and intellegent robot operation.
On the other hand, as a low-level building block, our work has no direct negative outcome, other than what could arise from the aforementioned applications.
Acknowledgments and Disclosure of Funding
This work was partially supported by JST AIP Acceleration Research JPMJCR20U3, Moonshot
R&D Grant Number JPMJPS2011, CREST Grant Number JPMJCR2015, JSPS KAKENHI Grant
Number JP19H01115 and Basic Research Grant (Super AI) of Institute for AI and Beyond of the
University of Tokyo.