SNAKE: Shape-aware Neural 3D Keypoint Field
Abstract
Detecting 3D keypoints from point clouds is important for shape reconstruction, while this work investigates the dual question: can shape reconstruction benefit 3D keypoint detection?
Existing methods either seek salient features according to statistics of different orders or learn to predict keypoints that are invariant to transformation.
Nevertheless, the idea of incorporating shape reconstruction into 3D keypoint detection is under-explored.
We argue that this is restricted by former problem formulations.
To this end, a novel unsupervised paradigm named SNAKE is proposed, which is short for s hape-aware n eur a l 3D ke ypoint field.
Similar to recent coordinate-based radiance or distance field, our network takes 3D coordinates as inputs and predicts implicit shape indicators and keypoint saliency simultaneously, thus naturally entangling 3D keypoint detection and shape reconstruction.
We achieve superior performance on various public benchmarks, including standalone object datasets ModelNet40, KeypointNet, SMPL meshes and scene-level datasets 3DMatch and Redwood.
Intrinsic shape awareness brings several advantages as follows.
(1) SNAKE generates 3D keypoints consistent with human semantic annotation, even without such supervision.
(2) SNAKE outperforms counterparts in terms of repeatability, especially when the input point clouds are down-sampled.
(3) the generated keypoints allow accurate geometric registration, notably in a zero-shot setting.
Codes are available at https://github.com/zhongcl-thu/SNAKE.
Introduction
2D sparse keypoints play a vital role in reconstruction [34], recognition [23] and pose estimation [45], with scale invariant feature transform (SIFT) [20] being arguably the most important preDeep Learning (DL) computer vision algorithm.
Altough dense alignment using photometric or featuremetric losses is also successful in various domains [2, 38, 8], sparse keypoints are usually preferred due to compactness in storage/computation and robustness to illumination/rotation.
Just like their 2D counterparts, 3D keypoints have also drawn a lot of attention from the community in both pre-DL [13, 37] and DL [16, 1, 40] literature, with various applications in reconstruction [47, 43] and recognition[28, 36].
However, detecting 3D keypoints from raw point cloud data is very challenging due to sampling sparsity.
No matter how we obtain raw point clouds (e.g., through RGB-D cameras [42], stereo [4], or LIDAR [10]), they are only a discrete representation of the underlying 3D shape.
This fact drives us to explore the question of whether jointly reconstructing underlying 3D shapes helps 3D [Figure] keypoint detection .
To our knowledge, former methods have seldom visited this idea.
Traditional 3D keypoint detection methods are built upon some forms of first-order (e.g., density in intrinsic shape signature [46]) or second-order (e.g., curvature in mesh saliency [15]) statistics, including sophisticated reformulation like heat diffusion [35].
Modern learning-based methods rely upon the idea of consistency under geometric transformations, which can be imposed on either coordinate like
USIP [16] or saliency value like D3Feat [1].
The most related method that studies joint reconstruction and 3D keypoint detection is a recent one named UKPGAN [40], yet it reconstructs input point cloud coordinates using an auxiliary decoder instead of the underlying shape manifold.
Why is this promising idea under-explored in the literature?
We argue the reason is that former problem formulations are not naturally applicable for reconstructing the underlying shape surface.
Existing paradigms are conceptually illustrated in Fig.
USIP-like methods directly output keypoint coordinates while UKPGAN-like methods generate saliency values for input point clouds.
In both cases, the representations are based upon discrete point clouds.
By contrast, we reformulate the problem using coordinate-based networks, as inspired by the recent success of neural radiance fields [22, 18, 31] and neural distance fields [24, 33].
As shown in Fig.
1-c, our model predicts a keypoint saliency value for each continuous input query point coordinate q ( x, y, z ) .
A direct advantage of this new paradigm is the possibility of tightly entangling shape reconstruction and 3D keypoint detection.
As shown in Fig.
1-c, besides the keypoint saliency decoder, we attach a parallel shape indicator decoder that predicts whether the query point q is occupied.
The input to decoders is feature embedding generated by trilinearly sampling representations conditioned on input point clouds P .
Imagine a feature embedding at the wing tip of an airplane, if it can be used to reconstruct the sharp curvature of the wing tip, it can be naturally detected as a keypoint with high repeatability.
As such, our method is named as s hape-aware n eur a l 3D ke ypoint field, or SNAKE.
Shape awareness, as the core feature of our new formulation, brings several advantages.
(1) High repeatability.
Repeatability is the most important metric for keypoint detection, i.e. , an algorithm should detect the same keypoint locations in two-view point clouds.
If the feature embedding can successfully reconstruct the same chair junction from two-view point clouds, they are expected to generate similar saliency scores.
(2) Robustness to down-sampling.
When input point clouds are sparse, UKPGAN-like frameworks can only achieve reconstruction up to the density of inputs.
In contrast, our SNAKE formulation can naturally reconstruct the underlying surface up to any resolution because it exploits coordinate-based networks.
(3) Semantic consistency.
SNAKE reconstructs the shape across instances of the same category, thus naturally encouraging semantic consistency although no semantic annotation is used.
For example, intermediate representations need to be similar for successfully reconstructing different human bodies because human shapes are intrinsically similar.
To summarize, this study has the following two contributions:
• We propose a new network for joint surface reconstruction and 3D keypoint detection based upon implicit neural representations.
During training, we develop several self-supervised losses that exploit the mutual relationship between two decoders.
During testing, we design a gradient-based optimization strategy for maximizing the saliency of keypoints.
• Via extensive quantitative and qualitative evaluations on standalone object datasets ModelNet40, KeypointNet, SMPL meshes, and scene-level datasets 3DMatch and Redwood, we demonstrate that our shape-aware formulation achieves state-of-the-art performance under three settings: (1) semantic consistency; (2) repeatability; (3) geometric registration.
Related Work
3D Keypoint Detector As discussed in the introduction, 3D keypoint detection methods can be mainly categorized into hand-crafted and learning-based.
Popular hand-crafted approaches [46, 32, 30] employ local geometric statistics to generate keypoints.
These methods usually fail to detect consistent keypoints due to the lack of global context, especially under real-world disturbances, such as density variations and noise.
USIP [16] is a pioneering learning-based 3D keypoint detector that outperforms traditional methods by a large margin.
However, the detected keypoints are not semantically salient, and the number of keypoints is fixed.
Fernandez et al.
[9] exploit the symmetry prior to generate semantically consistent keypoints.
But this method is category-specific, limiting the generalization to unseen categories and scenes.
Recently, UKPGAN [40] makes use of reconstruction to find semanticsaware 3D keypoints.
Yet, it recovers explicit coordinates instead of implicit shape indicators.
As shown in Fig.
1, different from these explicit keypoint detection methods, we propose a new detection framework using implicit neural fields, which naturally incorporates shape reconstruction.
Implicit Neural Representation Our method exploits implicit neural representations to parameterize a continuous 3D keypoint field, which is inspired by recent studies of neural radiance fields [18, 22, 31] and neural distance fields [24, 33, 17, 44].
Unlike explicit 3D representations such as point clouds, voxels, or meshes, implicit neural functions can decode shapes continuously and learn complex shape topologies.
To obtain fine geometry, ConvONet [26] proposes to use volumetric embeddings to get local instead of global features [21] of the input.
Recently, similar local geometry preserving networks show a great success for the grasp pose generation [12] and articulated model estimation [11].
They utilize the synergies between their main tasks and 3D reconstruction using shared local representations and implicit functions.
Unlike [11, 12] that learn geometry as an auxiliary task, our novel losses tightly couple surface occupancy and keypoint saliency estimates.
Method
This section presents SNAKE, a shape-aware implicit network for 3D keypoint detection.
SNAKE conditions two implicit decoders (for shape and keypoint saliency) on shared volumetric feature embeddings, which is shown in Fig.
2-framework.
To encourage repeatable, uniformly scattered, and sparse keypoints, we employ several self-supervised loss functions which entangle the predicted surface occupancy and keypoint saliency, as depicted in the middle panel of Fig.
During inference, query points with high saliency are further refined by gradient-based optimization since the implicit keypoint field is continuous and differentiable, which is displayed in Fig.
2-inference.
3.1 Network Architecture
Point Cloud Encoder
As fine geometry is essential to local keypoint detection, we adopt the
ConvONets [26], which can obtain local details and scale to large scenes, as the point cloud encoder denoted f θ en for SNAKE.
Given an input point cloud P ∈ R N × 3 , our encoder firstly processes it with the PointNet++ [27] or alternatives like [49]) to get a feature embedding Z ∈ R N × C 1 , where N and C 1 are respectively the number of points and the dimension of the features.
Then, these features are projected and aggregated into structured volume Z ′ ∈ R C 1 × H × W × D , where H , W and D are the number of voxels in three orthogonal axes.
The volumetric embeddings serve as input to the 3D
UNet [6] to further integrate local and global information, resulting in the output G ∈ R C 2 × H × W × D , where C 2 is the output feature dimension.
More details can be found in the Appendix A.
Shape Implicit Decoder As shown in the top panel of Fig.
2, each point q ∈ R 3 from a query set Q is encoded into a C e -dimensional vector q e via a multi-layer perceptron that is denoted the positional
SurfaceOccupancyDecoder
[Figure]
Trilinear
[Figure]
𝑷𝒓𝒐𝒃 𝒐 (𝑸|𝑷)
𝑷𝒓𝒐𝒃 𝒔 (𝑸|𝑷) (a
[Figure]
Similarity 𝑷𝒓𝒐𝒃 𝒔 (𝑸|𝑷)
𝑷𝒓𝒐𝒃 𝒔 (𝑻𝑸|𝑻𝑷)
[Figure]
MarchingCubes
Positional Encoding
Sampling
[Figure]
Encoded [Figure]
𝐓∈ SE(3)
[Figure]
𝑷𝒓𝒐𝒃 𝒔 (𝑸 𝟎 |𝑷) 𝑸
[Figure]
{𝒒| [Figure] 𝒐 [-(𝒒|𝑷)→𝟎} 
-](𝒒|𝑷)→
[Figure] [Figure] [Figure]
[Figure]
𝒆
[Figure] 𝒒 𝒆 [Figure] 𝑮 𝑸
[Figure]
𝒒
[Figure] [Figure] encoder f θ pos , i.e. q e = f θ pos ( q ) . Then, the local feature G q is retrieved from the feature volume G according to the coordinate of q via trilinear interpolation.
The generated q e and G q are concatenated and mapped to the surface occupancy probability Prob o ( q | P ) ∈
[0 , 1] by the occupancy decoder f θ o , as given in Eq.
If q is on the input surface, the Prob o ( q | P ) would be 1, otherwise be 0.
In our formulation, the points inside the surface are also considered unoccupied.
[Equation]
Keypoint Implicit Decoder Most of the process here is the same as in shape implicit decoder, except for the last mapping function.
The goal of keypoint implicit decoder f θ s is to estimate the saliency of the query point q conditioned on input points P , which is denoted as Prob s ( q | P ) ∈ [0 , 1] and formulated by:
[Equation]
Here, saliency of the query point q is the likelihood that it is a keypoint.
3.2 Implicit Field Training
The implicit field is jointly optimized for surface occupancy and saliency estimation by several selfsupervised losses.
In contrast to former arts [12, 11] with a similar architecture that learn multiple tasks separately, we leverage the geometry knowledge from shape field to enhance the performance of keypoint field, as shown in the green arrows of Fig.
Specifically, the total loss is given by:
[Equation]
where L o encourages the model to learn the shape from the sparse input, L r , L m and L s respectively help the predicted keypoint to be repeatable, located on the underlying surface and sparse.
Surface Occupancy Loss
The binary cross-entropy loss l BCE between the predicted surface occupancy Prob o ( q | P ) and the ground-truth label Prob gto is used for shape recovery.
The queries Q are randomly sampled from the whole volume size H × W × D .
The average over all queries is as follows:
[Equation]
where | Q | is the number of queries Q .
Repeatability Loss Detecting keypoints with high repeatability is essential for downstream tasks like registration between two-view point clouds.
That indicates the positions of keypoint are covariant to the rigid transformation of the input.
To achieve a similar goal, 2D keypoint detection methods [29, 7, 45] enforce the similarity of corresponding local salient patches from multiple views.
Inspired by them, we enforce the similarity of local overlapped saliency fields from two-view point clouds.
Since the implicit field is continuous, we uniformly sample some values from a local field to represent the local saliency distribution.
Specifically, as shown in the top and the middle part of Fig. 2, we build several local 3D Cartesian grids { Q i } ni =1 with resolution of H l × W l × D l and size of 1 /U .
We empirically set the resolution of Q i to be almost the same as the feature volume G .
As non-occupied regions are uninformative, the center of Q i is randomly sampled from the input.
Then, we perform random rigid transformation T on the P and Q i to generate TP and TQ i .
Similar to [29], the cosine similarity, denoted as cosim , is exploited for the corresponding saliency grids of Q i and TQ i :
[Equation]
Surface Constraint Loss As discussed in [16], 3D keypoints are encouraged to close to the input.
They propose a loss to constrain the distance between the keypoint and its nearest neighbor from the input.
Yet, the generated keypoints are inconsistent when given the same input but with a different density.
Thanks to the shape decoder, SNAKE can reconstruct the underlying surface of the input, which is robust to the resolution change.
Hence, we use the surface occupancy probability to represent the inverse distance between the query and the input.
As can be seen in Fig.
2-(surface constraint), we enforce the saliency of the query that is far from input P close to 0, which is defined as
[Equation]
Sparsity Loss Similar to 2D keypoint detection methods [29], we design a sparsity loss to avoid the trivial solution ( Prob s ( Q | P ) =0) in Eq.
As can be seen in Fig.
2, the goal is to maximize the local peakiness of the local saliency grids.
As the sailency values of non-occupied points are enforced to 0 by L m , we only impose the sparsity loss on the points with high surface occupancy probability.
Hence, we derive the sparsity loss with the help of decoded geometry by
[Equation]
where Q 1 i = { q | q ∈ Q
i , Prob o ( q | P ) > 1 − thr o } , thr o ∈ (0 , 0 . 5] is a constant, and n is the number of grids.
It is noted that the spatial frequency of local peakiness is dependent on the grid size 1 /U , see section 4.4.
Since the network is not only required to find sparse keypoints, but also expected to recover the object shape, it would generate high saliency at the critical parts of the input, like joint points of a desk and corners of a house, as shown in the Fig.
2-result.
3.3 Explicit Keypoint Extraction
The query point q whose saliency is above a predefined threshold thr s ∈ (0 , 1) would be selected as a keypoint at the inference stage.
Although SNAKE can obtain the saliency of any query point, a higher resolution query set results in a high computational cost.
Hence, as shown in Fig.
2-inference, we build a relatively low-resolution query sets Q infer which are evenly distributed in the input space and further refine the coordinates of Q infer by gradient-based optimization on this energy function: [Equation]
[Equation]
Specifically, details of the explicit keypoint extraction algorithm are summarized in Alg. 1.
Require: P, Q infer , f θ en , f θ pos , f θ o , f θ s .
Hyper-parameters: λ , J , thr o , thr s .
Get initial Prob o ( Q infer | P ) according to Eq.
Filter to get new query set Q infer ′ = { q | q ∈ Q infer , Prob o ( q | P ) > 1 − thr o } . for 1 to J do Evaluate energy function E ( Q infer ′ , P ) .
Update coordinates with gradient descent: Q infer ′ =
Q infer ′ − λ ∇ Q infer ′
E ( Q infer ′ , P ) .
end for
Sample final keypoints Q k = { q | q ∈ Q infer ′ , Prob s ( q | P ) > thr s } .
In this section, we evaluate SNAKE under three settings.
First, we compare keypoint semantic consistency across different instances of the same category, using both rigid and deformable objects.
Next, keypoint repeatability of the same instance under disturbances such as SE(3) transformation, noise and downsample is evaluated.
Finally, we inspect the point cloud registration task on the 3DMatch benchmark, notably in a zero-shot generalization setting.
Besides, an ablation study is done to verify the effect of each design choice in SNAKE.
The implementation details and hyper-parameters for SNAKE in three settings can be found in the Appendix B.
4.1 Semantic Consistency
Datasets The KeypointNet [41] dataset and meshes generated with the SMPL model [19] are utilized.
KeypointNet has numerous human-annotated 3D keypoints for 16 object categories from ShapeNet [3].
The training set covers all categories that contain 5500 instances.
Following [40], we evaluateunseen instances from airplanes, chairs, and tables.
SMPL is a skinned vertex-based deformable model that accurately captures body shape variations in natural human poses.
We use the same strategy in [40] to generate both training and testing data.
Metric Mean Intersection over Union (mIoU) is adopted to show whether the keypoints across intra-class instances have the same semantics or not.
For KeypointNet, a predicted keypoint is considered the same as a human-annotated semantic point if the geodesic distance between them is under some threshold.
Due to the lack of human-labeled keypoints on SMPL, we compare the keypoint consistency in a pair of human models.
A keypoint in the first model is regarded semantically consistent if the distance between its corresponding point and the nearest keypoint in the second model is below some threshold.
[Figure]
Evaluation and Results We compare SNAKE with random detection, hand-crafted detectors:
ISS [46], Harris-3D [32] and SIFT-3D [30], and
DL-based unsupervised detectors: USIP [16] and UKPGAN [40].
As USIP has not performed semantic consistency evaluations, we train the model with the code they provided.
We follow the same protocols in [40] to filter the keypoints via NMS with a Euclidean radius of 0.1.
Quantitative results are provided in Fig.
5-(a,e).
SNAKE obtains higher mIoU than other methods under most thresholds on KeypointNet and SMPL.
Qualitative results in Fig.
3 show our keypoints make good alignment with human annotations.
Fig.
4 provides qualitative comparisons of semantically consistent keypoints on rigid and deformable objects.
Owing to entangling shape reconstruction and keypoint detection, SNAKE can extract aligned representation for intra-class instances.
Thus, our keypoints better outline the object shapes and are more semantically consistent under large shape variations.
As shown in the saliency field projected slices, we can get symmetrical keypoints, although without any explicit constraint like the one used in [40]. Here, a projected slice is obtained by taking the maximum value of a given field along the projection direction. [Figure] [Figure]
4.2 Repeatability
Datasets ModelNet40 [39] is a synthetic object-level dataset that contains 12,311 pre-aligned shapes from 40 categories, such as plane, guitar and table.
We adopt the official dataset split strategy. 3DMatch [43] and Redwood [5] are RGB-D reconstruction datasets for indoor scenes.
Following [16], we train the model on 3DMatch and test it on Redwood to show the generalization performance.
The training set contains around 19k samples and the test set consists of 207 point clouds.
Metric We adopt the relative repeatability proposed in USIP [16] as the evaluation metric.
Given two point clouds captured from different viewpoints, a keypoint in the first point cloud is repeatable if its [Figure] distance to the nearest keypoint in the is below threshold means the
Random detection, traditional methods and USIP are chosen as our baselines.
Since UKPGAN does not provide pre-trained models on these two datasets, we do not report its results in Fig.
5 but make an additional comparison on KeypointNet, which is illustrated in the next paragraph.
We use NMS to select the local peaky keypoints with a small radius (0.01 normalized distance on ModelNet40 and 0.04 meters on Redwood) for ours and baselines.
We generate 64 keypoints in each sample and show the performance under different distance thresholds ϵ , downsample rates, and Gaussian noise scales.
We set a fixed ϵ of 0.04 normalized distance and 0.2 meters on the ModelNet40 and Redwood dataset when testing under the last two cases.
As shown in Fig.
5- (b,f), SNAKE outperforms state-of-the-art at most distance thresholds.
We do not surpass USIP on Redwood in the lower thresholds.
Note that it is challenging to get higher repeatability on Redwood because the paired inputs have very small overlapping regions.
Fig.
5-(c,d,g,h) show the repeatability robustness to different downsample rates (d.r.) and Gaussian noise N (0 , σ ) levels.
SNAKE gets the highest repeatability in most cases because the shape-aware strategy helps the model reason about the underlying shapes of the objects/scenes, which makes keypoints robust to the input variations.
Fig.provides visualization of object-level and scene-level keypoints of the original and disturbed inputs.
SNAKE can generate more consistent keypoints than other methods under drastic input changes.
We have tried to train UKPGAN (official implementation) on ModelNet40 and 3DMatch datasets from scratch but observed divergence under default hyper-parameters. As such, we provide a new experiment to compare their repeatability on the KeypointNet dataset, on which UKPGAN provided a pre-trained model. We randomly perform SE(3) transformation on the test point clouds to generate the second view point clouds. Then, we select top-32 salient keypoints with NMS (radius=0.03) in each sample and report the keypoint repeatability under different distance thresholds ϵ , downsample rates, and Gaussian noise scales.
The results are summarized in Table 1, 2, which show that SNAKE achieves significant gains over UKPGAN in most cases.
More discussions can be found in the
Appendix C.1.
[Table]
[Table]
[Figure]
U [Figure] U= Datasets We follow the same protocols in [40] to train the model on KeypointNet and then directly test it on 3DMatch [43] dataset, evaluating how well two-view point clouds can be registered.
The test set consists of 8 scenes which include some partially overlapped point cloud fragments and the ground truth SE(3) transformation matrices.
Metric To evaluate geometric registration, we need both keypoint detectors and descriptors.
Thus, we combine an off-the-shelf and state-of-the-art descriptor D3Feat [1] with our and other keypoint detectors.
Following [40], we compute three metrics: Feature Matching Recall, Inlier Ratio, and
Registration Recall for a pair of point clouds.
Evaluation and Results As baselines, we choose random detection, ISS, SIFT-3D, UKPGAN, and D3Feat.
Note that D3Feat is a task-specific learning-based detector trained on the 3DMatch dataset, thus not included in this zero-shot comparison.
Ours and UKPGAN are trained on the synthetic object dataset KeypointNet only.
The results are reported under different numbers of keypoints ( i.e. , 2500, 1000, 500, 250, 100).
The NMS with a radius of 0.05m is used for D3Feat,
UKPGAN, and ours.
As shown in Table 3, SNAKE outperforms other methods consistently under three metrics.
For registration recall and inlier ratio, we achieve significant gains over UKPGAN and other traditional keypoint methods.
Notably, when the keypoints are high in numbers, SNAKE even outperforms D3Feat which has seen the target domain.
Local shape primitives like planes, corners, or curves may be shared between objects and scenes, so our shape-aware formulation allows a superior generalization from objects to scenes.
We combine the off-the-shelf descriptor D3Feat [1] and different keypoint detectors to perform two-view point cloud registration
[Table]
4.4 Ablation Study
Loss Function Table 4 reports the performance w.r.t. designs of loss functions.
(Row 1)
If the surface occupancy decoder is removed, the surface constraint cannot be performed according to Eq.
( 6), so they are removed simultaneously.
Although the model could detect significantly repeatable keypoints on ModelNet40 [39], it fails to give semantically consistent keypoints on KeypointNet [41].
Fig.
7-a shows that SNAKE is unable to output symmetric and meaningful keypoints without the shape-aware technique.
That indicates the repeatability could not be the only criterion for keypoint detection if an implicit formulation is adopted.
(Row 2-4)
Each loss function for training keypoint field is vital for keypoint detection.
Note that the model gives a trivial solution ( 0 ) for the saliency field and cannot extract distinctive points when removing the sparsity loss.
Grid Size and Volumetric Resolution
The grid size 1 /U controls the number of keypoints because
L s enforces the model to predict a single local maxima per grid of size (1 /U ) 3 .
Fig.
7-b shows
[Table] [Table] [Table] different saliency field slices obtained from the same input with various 1 /U .
When U is small,
SNAKE outputs fewer salient responses, and more for larger values of U .
We also give the relative repeatability results on ModelNet40 under distance threshold ϵ = 0 . 04 in Table [Table] indicating that
U = 6 gives the best results.
From Table [Table] we can see that higher resolution improves performance.
However, the performance drops when it reaches the resolution of 80.
The potential reason is as such: the number of queries in a single grid increases when the resolution becomes higher, as mentioned in 3.2.
In this case, finer details make the input to cosine similarity too long and contain spurious values.
Optimization Step and Learning Rate Fig.
7-c shows the importance of optimization (see Alg. 1) for refining keypoint coordinates on the ModelNet40 dataset.
It is noted that too many optimization steps will not bring more gains but increase the computational overhead.
In this paper, we set the number of update steps to 10.
The learning rate for optimization is also key to the final result.
When the learning rate is set to 0.1, 0.01, 0.001 and 0.0001, the relative repeatability (%) on ModelNet dataset with the same experimental settings as Table 6 are 0.002, 0.622, 0.854 and 0.826, respectively. In addition, the comparison of computation cost of baselines and ours can be found in the Appendix D.
Conclusion and Discussion
We propose SNAKE, a method for 3D keypoint detection based on implicit neural representations.
Extensive evaluations show our keypoints are semantically consistent, repeatable, robust to downsample, and generalizable to unseen scenarios.
Limitations.
The optimization for keypoint extraction during inference requires considerable computational cost and time, which may not be applicable for use in scenarios that require real-time keypoint detection.
Negative Social Impact.
The industry may use the method for pose estimation in autonomous robots.
Since our method is not perfect, it may lead to wrong decision making and potential human injury.
Acknowledgments
This research is jointly supported by following projects: the National Science and Technology Major Project of the Ministry of Science and Technology of China (No.2018AAA0102900); the Key
Field R&D Program of Guangdong Province (No.2021B0101410002); Sino-German Collaborative Research Project Crossmodal Learning (NSFC 61621136008/DFG SFB/TRR169); the National Natural Science Foundation of China (No.62006137); Beijing Outstanding Young Scientist Program (No.BJJWZYJH012019100020098). We would like to thank Pengfei Li for discussions about implicit field learning.
We would also like to thank the anonymous reviewers for their insightful comments. Appendix A Network Architecture Following [26], our implementation is a compilation of PointNet++ [27], 3D UNet [6], positional encoder and implicit surface occupancy decoder. The architecture of the implicit keypoint decoder is designed to be the same as the surface occupancy decoder.
The dimensions of the feature embedding
Z and Z ′ are both set to 32, i.e. , C 1 = C 2 = 32 . And each point from a query set is also encoded into a 32-dimensional feature vector.
More details can be found in the code we provide. B Implementation Details B.1 Training SNAKE is implemented in PyTorch [25] using the Adam [14] optimizer with a mini-batch size of b on 4 NVIDIA A100 GPUs for el epochs. We use a learning rate of 10 − 4 for the first ef epochs, which is dropped ten times for the remainder.
As discussed in Sec.
3.2 (repeatability loss), we perform random rigid transformation T on the input P to generate a second view input TP . Then, we use some data augmentation on TP to increase data diversity by downsampling with a random rate between 0 and 4, and Gaussian noise.
Training hyper-parameters on four datasets are provided in Table 7. In our formulation, occupied points are those on the input surface, and the others are considered all unoccupied, including the points inside the surface.
Therefore, we can only use input point clouds to learn the surface occupancy model. Specifically, we randomly sample the positives from the input point cloud. The negatives are randomly sampled in the unit 3D space. Although some of the negatives are indeed on the surface of the object, their number is so limited compared to the whole query sets that they do not affect the training.
Sem [Table]
consistency [Table] Rep.
[Table]
Repeatability evaluation, Reg.=Registration evaluation, KeypN.=KeypointNet [41], ModelN.=ModelNet40 [39] [Table] B.2 Testing For the SMPL dataset, the correspondence between the paired point clouds can be generated by
SMPL vertex index. Since the keypoint SNAKE generates may not be in the input point cloud (we enforce the keypoint scatter on the underlying surface of the input), we take the point closest to the generated keypoint in the input as the final keypoint. We use the same strategy on the 3DMatch dataset when performing geometric registration because D3feat [1] predicts descriptors for each point in the input. The testing hyper-parameters are shown in Table 7.
C Results C.1 Additional comparison with UKPGAN on keypoint repeatability
Due to the absence of pretrained model on the ModelNet40 and 3DMatch dataset, we do not report the keypoint repeatability of UKPGAN [40] on the main paper.
We have tried to train UKPGAN (official implementation) on the ModelNet40 and 3DMatch datasets from scratch but observed divergence under default hyper-parameters.
The training always reports NaN losses in early epochs.
This instability also implies limitations in implementing the idea of joint reconstruction and keypoint detection with GAN-based methods.
As such, we provide a new experiment to compare their repeatability on the KeypointNet dataset, on which the UKPGAN provided a pre-trained model. Tabke 1 and Table 2 in the main paper show that SNAKE achieves significant gains over UKPGAN in most cases. Interestingly, when the inputs are disturbed, the performance of UKPGAN increases rather than decreases. Via visualizing the results in Fig.
8, we find that when the input point clouds are disturbed, the keypoints predicted by UKPGAN are clustered in a small area, which improves the repeatability of keypoints but fails to cover the input uniformly. This illustrates that the GAN-based method adopted by UKPGAN to control the keypoint sparsity is not robust to input point cloud disturbance.
The keypoints of ours still remain meaningful under the drastic changes of inputs.
[Figure]
C.2 Quantitative Results
The specific numerical results on semantic consistency and repeatability are summarized in Table 915, which correspond to Figure 5 in the main paper.
We present the mean and standard deviation of our results over 6 models trained under different random seeds. C.3 Qualitative Visualization of Saliency Field and Keypoints We show more qualitative results on keypoint semantic consistency between intra-class instances from rigid objects plane, guitar, motorcycle, and deformable human shapes in Figure 10- 13. Owing to entangling shape reconstruction and keypoint detection, SNAKE can extract aligned representation for intra-class instances.
As shown in Figure 14- 19, we provide more visualizations of keypoints under some disturbances on object-level (ModelNet40) and scene-level (Redwood) datasets.
It can be seen that SNAKE can generate more consistent keypoints than other methods under significant variations of inputs.
We also show the detected keypoints of the same object/scene from different views to demonstrate the repeatability of keypoint in Figure 20- 22.
As shown in Figure 9, we show visualizations of the occupancy field and shape reconstruction on the
ModelNet40 dataset.
These five samples are taken from the unseen test set.
As shown by the second row, only points on the input surface have a high occupancy value, and the other points (inside or outside of the surface) have a near-zero occupancy value.
Under our definition, two surfaces can be obtained through the marching cube, and we only show the outer surface.
As shown in Table 8, we report the time taken to generate keypoints of hand-crafted detector ISS, deep-learning (DL) based methods USIP [16], UKPGAN [40] and ours.
ISS [46] is implemented by Open3d [48] and deployed on an AMD EPYC 7742 64-Core CPU. DL-based methods are deployed on an NVIDIA GeForce RTX 3090 GPU. USIP requires the lowest computational time to generate keypoints, while UKPGAN requires the highest cost since it takes much time to compute smoothed density values.
The inference time of our model is comparable to ISS when we do not refine the keypoint by optimization ( J =0), and the repeatability is still as high as around 81% when the input point number is 4096.
The time increases with the increasing number of optimization iterations J .
As discussed before, when J becomes larger (below 15), the performance of keypoint gets better. It suggests that there is a trade-off between keypoint performance and inference speed of our method. The GPU memory cost (MB) for USIP, UKPGAN, and SNAKE during a single batch inference is 3747, 10727, and 2785, which illustrates that SNAKE requires the lowest GPU memory cost to generate keypoints. The hyper-parameters of ours can be found in the Table 7.
Decimals in parentheses in italics are relative repeatability (%).
Here, the experiment setting is the same as in Sec. 4.2. [Table] E Illustrations on the Assets We Used and Released
The license of assets we used is as follows: (a) MIT License for KeypointNet dataset. (b) Software
Copyright License for non-commercial scientific research purposes on SMPL-Model.
(c) GPL-3. License for ModelNet40, 3DMatch, Redwood dataset, and USIP. (d) Microsoft research license for 3DMatch registration benchmark.
All existing datasets and codes we used in this paper are allowed for research and do not contain personally identifiable information or offensive content. Note that SMPL only has human shapes without the identity information of the person, such as the face or body texture. Our code is released under the MIT license.
[Table] on SMPL mesh.
This table corresponds to Figure 5-(e) in the main paper.
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
This table corresponds to Figure 5-(b) in the main paper. [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure] [Figure] [Figure] ShapeRecon.
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]