SNAKE: Shape-aware Neural 3D Keypoint Field
Anonymous Author(s)
AffiliationAddress email
Abstract
Detecting 3D keypoints from point clouds is important for shape reconstruction,while this work investigates the dual question: can shape reconstruction benefit3D keypoint detection?
Existing methods either seek salient features accordingto statistics of different orders or learn to predict keypoints that are invariantto transformation.
Nevertheless, the idea of incorporating shape reconstructioninto 3D keypoint detection is under-explored.
We argue that this is restrictedby former problem formulations.
To this end, a novel unsupervised paradigmnamed SNAKE is proposed, which is short for s hape-aware n eur a l 3D ke ypointfield.
Similar to recent coordinate-based radiance or distance field, our networktakes 3D coordinates as inputs and predicts implicit shape indicators and keypointsaliency simultaneously, thus naturally entangling 3D keypoint detection and shapereconstruction.
We achieve superior performance on various public benchmarks,including standalone object datasets ModelNet40, KeypointNet, SMPL meshesand scene-level datasets 3DMatch and Redwood.
Intrinsic shape awareness bringsseveral advantages as follows.
(1) SNAKE generates 3D keypoints consistentwith human semantic annotation, even without such supervision.
(2) SNAKEoutperforms counterparts in terms of repeatability, especially when the input pointclouds are down-sampled.
(3) the generated keypoints allow accurate geometricregistration, notably in a zero-shot setting.
Codes and models will be released.
Introduction
2D sparse keypoints play a vital role in both reconstruction [31] and recognition [21], with scaleinvariant feature transform (SIFT)
[18] being arguably the most important pre-Deep Learning (DL)computer vision algorithm.
Altough dense alignment using photometric or featuremetric losses is alsosuccessful in various domains [2, 35, 8], sparse keypoints are usually preferred due to compactnessin storage/computation and robustness to illumination/rotation.
Just like their 2D counterparts, 3Dkeypoints have also drawn a lot of attention from the community in both pre-DL [13, 34] and DL[15, 1, 37] literature, with various applications in reconstruction [42, 40] and recognition[25, 33].
However, detecting 3D keypoints from raw point cloud data is very challenging due to samplingsparsity.
No matter how we obtain raw point clouds (e.g., through RGB-D cameras [39], stereo[4], or LIDAR [10]), they are only a discrete representation of the underlying 3D shape.
This factdrives us to explore the question of whether jointly reconstructing underlying 3D shapes helps 3Dkeypoint detection .
To our knowledge, former methods have seldom visited this idea.
Traditional3D keypoint detection methods are built upon some forms of first-order (e.g., density in intrinsicshape signature [41]) or second-order (e.g., curvature in mesh saliency [14]) statistics, includingsophisticated reformulation like heat diffusion [32].
Modern learning-based methods rely upon theidea of consistency under geometric transformations, which can be imposed on either coordinate like
USIP [15] or saliency value like D3Feat [1].
The most related method that studies joint reconstruction
[Figure]
and 3D keypoint detection is a recent one named UKPGAN [37], yet it reconstructs input point cloudcoordinates using an auxiliary decoder instead of the underlying shape manifold.
Why is this promising idea under-explored in the literature?
We argue the reason is that formerproblem formulations are not naturally applicable for reconstructing the underlying shape surface.
Existing paradigms are conceptually illustrated in Fig.
USIP-like methods directly output keypointcoordinates while UKPGAN-like methods generate saliency values for input point clouds.
In bothcases, the representations are based upon discrete point clouds.
By contrast, we reformulate theproblem using coordinate-based networks, as inspired by the recent success of neural radiance fields[20, 16, 28] and neural distance fields [22, 30].
As shown in Fig.
1-c, our model predicts a keypointsaliency value for each continuous input query point coordinate q ( x, y, z ) .
A direct advantage of this new paradigm is the possibility of tightly entangling shape reconstructionand 3D keypoint detection.
As shown in Fig.
1-c, besides the keypoint saliency decoder, we attacha parallel shape indicator decoder that predicts whether the query point q is occupied.
The inputto decoders is feature embedding generated by trilinearly sampling representations conditioned oninput point clouds P .
Imagine a feature embedding at the wing tip of an airplane, if it can be used toreconstruct the sharp curvature of the wing tip, it can be naturally detected as a keypoint with highrepeatability.
As such, our method is named as s hape-aware n eur a l 3D ke ypoint field, or SNAKE.
Shape awareness, as the core feature of our new formulation, brings several advantages.
(1) Highrepeatability.
Repeatability is the most important metric for keypoint detection, i.e. , an algorithmshould detect the same keypoint locations in two-view point clouds.
If the feature embedding cansuccessfully reconstruct the same chair junction from two-view point clouds, they are expected togenerate similar saliency scores.
(2) Robustness to down-sampling.
When input point clouds aresparse, UKPGAN-like frameworks can only achieve reconstruction up to the density of inputs.
Incontrast, our SNAKE formulation can naturally reconstruct the underlying surface up to any resolutionbecause it exploits coordinate-based networks.
(3) Semantic consistency.
SNAKE reconstructs theshape across instances of the same category, thus naturally encouraging semantic consistency althoughno semantic annotation is used.
For example, intermediate representations need to be similar forsuccessfully reconstructing different human bodies because human shapes are intrinsically similar.
To summarize, this study has the following two contributions:
• We propose a new network for joint shape reconstruction and 3D keypoint detection basedupon implicit neural representations.
During training, we develop several self-supervisedlosses that exploit the mutual relationship between two decoders.
During testing, we designa gradient-based optimization strategy for maximizing the saliency of keypoints.• Via extensive quantitative and qualitative evaluations on standalone object datasets Model-
Net40, KeypointNet, SMPL meshes, and scene-level datasets 3DMatch and Redwood, we demonstrate that our shape-aware formulation achieves state-of-the-art performance underthree settings: (1) semantic consistency; (2) repeatability; (3) geometric registration.
Related Work
3D Keypoint Detector As discussed in the introduction, 3D keypoint detection methods can be mainlycategorized into hand-crafted and learning-based.
Popular hand-crafted approaches [41, 29, 27]employ local geometric statistics to generate keypoints.
These methods usually fail to detect consistentkeypoints due to the lack of global context, especially under real-world disturbances, such as densityvariations and noise.
USIP [15] is a pioneering learning-based 3D keypoint detector that outperformstraditional methods by a large margin.
However, the detected keypoints are not semantically salient,and the number of keypoints is fixed.
Fernandez et al.
[9] exploit the symmetry prior to generatesemantically consistent keypoints.
But this method is category-specific, limiting the generalization tounseen categories and scenes.
Recently, UKPGAN [37] makes use of reconstruction to find semanticsaware 3D keypoints.
Yet, it recovers explicit coordinates instead of implicit shape indicators.
Asshown in Fig.
1, different from these explicit keypoint detection methods, we propose a new detectionframework using implicit neural fields, which naturally incorporates shape reconstruction.
Implicit
Neural Representation Our method exploits implicit neural representations to parameterizea continuous 3D keypoint field, which is inspired by recent studies of neural radiance fields [20, 16, 28]and neural distance fields [22, 30].
Unlike explicit 3D representations such as point clouds, voxels, ormeshes, implicit neural functions can decode shapes continuously and learn complex shape topologies.
To obtain fine geometry, ConvONet [23] proposes to use volumetric embeddings to get local insteadof global features [19] of the input.
Recently, similar local geometry preserving networks show agreat success for the grasp pose generation [12] and articulated model estimation [11].
They utilizethe synergies between their main tasks and 3D reconstruction using shared local representations andimplicit functions.
Unlike [12, 11] that learn geometry as an auxiliary task, our novel losses tightlycouple occupancy and keypoint saliency estimates.
Method
This section presents SNAKE, a shape-aware implicit network for 3D keypoint detection.
SNAKEconditions two implicit decoders (for shape and keypoint saliency) on shared volumetric featureembeddings, which is shown in Fig.
2-framework.
To encourage repeatable, uniformly scattered,and sparse keypoints, we employ several self-supervised loss functions which entangle the predictedoccupancy and keypoint saliency, as depicted in the middle panel of Fig.
During inference, querypoints with high saliency are further refined by gradient-based optimization since the implicit keypointfield is continuous and differentiable, which is displayed in Fig.
2-inference.
3.1 Network Architecture
Point Cloud Encoder
As fine geometry is essential to local keypoint detection, we adopt the
ConvONets [23], which can obtain local details and scale to large scenes, as the point cloud encoderdenoted f θ en for SNAKE.
Given an input point cloud P ∈ R N × 3 , our encoder firstly processes itwith the PointNet++ [24] (or alternatives like [43]) to get a feature embedding Z ∈ R N × C 1 , where Nand C 1 are respectively the number of points and the dimension of the features.
Then, these featuresare projected and aggregated into structured volume Z ′ ∈ R C 1 × H × W × D , where H , W and D arethe number of voxels in three orthogonal axes.
The volumetric embeddings serve as input to the 3D UNet [6] to further integrate local and global information, resulting in the output G ∈ R C 2 × H × W × D ,where C 2 is the output feature dimension.
More details can be found in the supplementary.
Shape Implicit Decoder As shown in the top panel of Fig.
2, each point q ∈ R 3 from a query set
Q is enconded into a C e -dimensional feature vector q e via a multi-layer perceptron that is denotedthe positional encoder f θ pos , i.e. q e = f θ pos ( q ) .
Then, the local feature G q is retrieved from thefeature volume G according to the coordinate of q via trilinear interpolation.
The generated q e and
G q are concatenated and mapped to the occupancy probability Prob o ( q | P ) ∈
[0 , 1] by the occupancydecoder f θ o , as given in Eq.
If q is occupied, the Prob o ( q | P ) would be 1, otherwise be 0.
[Equation]
OccupancyDecoder
KeypointDecoder
Shape Recon.
[Figure]
Volume
Point
[Figure]
Encoder
Trilinear
Framework
[Figure]
Keypoint Detection 𝑷𝒓𝒐𝒃 𝒐 (𝑸|𝑷)
𝑷𝒓𝒐𝒃 𝒔 (𝑸|𝑷)
(a) Repeatability
Similarity 𝑷𝒓𝒐𝒃 𝒔 (𝑸|𝑷)
𝑷𝒓𝒐𝒃 𝒔 (𝑻𝑸|𝑻𝑷)
NMS
MarchingCubes
Positional Encoding
Sampling
Local Feature
Encoded Coord.
𝐓∈ SE(3) (b)
SurfaceConstraint 𝑷𝒓𝒐𝒃 𝒔 (𝑸 𝟎 |𝑷) 𝑸 {+
+}
={𝒒|𝑷𝒓𝒐𝒃 𝒐 (𝒒|𝑷)→𝟎} 0close to𝒒 … 𝑸 𝒆 = {𝒒 𝒆 } 𝑮 𝑸 = {𝑮 𝒒 } … …
𝑮
SNAKENetwork
Occupancy
[Figure]
𝑷𝒓𝒐𝒃 𝒔 (𝑸 𝟏 |𝑷) 𝑸
[Figure]
𝒒|
[Figure]
𝒐 (𝒒|𝑷)→
[Figure]
[Figure]
Keypoint Implicit Decoder Most of the process here is the same as in shape implicit decoder, exceptfor the last mapping function.
The goal of keypoint implicit decoder f θ s is to estimate the saliencyof the query point q conditioned on input points P , which is denoted as Prob s ( q | P ) ∈ [0 , 1] andformulated by:
[Equation]
Here, saliency of the query point q is the likelihood that it is a keypoint.
3.2 Implicit Field Training
The implicit field is jointly optimized for occupancy and saliency estimation by several self-supervisedlosses.
In contrast to former arts [12, 11] with a similar architecture that learn multiple tasks separately,we leverage the geometry knowledge from shape field to enhance the performance of keypoint field,as shown in the green arrows of Fig. 2.
Specifically, the total loss is given by:
[Equation]
where L o encourages the model to learn the shape from the sparse input, L r , L m and L s respectivelyhelp the predicted keypoint to be repeatable, located on the underlying surface and sparse.
Occupancy Loss
The binary cross-entropy loss l BCE between the predicted occupancy Prob o ( q | P )and the ground-truth label Prob gto is used for shape recovery.
The queries Q are randomly sampledfrom the whole volume size H × W × D .
The average over all queries is as follows:
[Equation]
[Equation]
Algorithm
Require: P, Q infer , f θ en , f θ pos , f θ o , f θ s .
Hyper-parameters: λ , J , thr o , thr s .
Get initial Prob o ( Q infer | P ) according to Eq.
Filter to get new query set Q infer ′
= { q | q ∈ Q infer , Prob o ( q | P ) > 1 − thr o } .
for to J do
Evaluate energy function E ( Q infer ′ , P ) .
Update coordinates with gradient descent: Q infer ′ =
Q infer ′ − λ ∇ Q infer ′
E ( Q infer ′ , P ) .
end for keypoints Q k = { q | q ∈ Q infer ′ , Prob s q | P ) > thr s } .
where | Q | is the number of queries Q .
Repeatability Loss Detecting keypoints with high repeatability is essential for downstream tasks likeregistration between two-view point clouds.
That indicates the positions of keypoint are covariant tothe rigid transformation of the input.
To achieve a similar goal, 2D keypoint detection methods [26, 7]enforce the similarity of corresponding local salient patches from multiple views.
Inspired by them,we enforce the similarity of local overlapped saliency fields from two-view point clouds.
Since theimplicit field is continuous, we uniformly sample some values from a local field to represent the localsaliency distribution.
Specifically, as shown in the top and the middle part of Fig. 2, we build severallocal 3D Cartesian grids { Q i } ni =1 with resolution of H l × W l × D l and size of 1 /U .
We empiricallyset the resolution of Q i to be almost the same as the feature volume G .
As non-occupied regions areuninformative, the center of Q i is randomly sampled from the input.
Then, we perform random rigidtransformation T on the P and Q i to generate TP and TQ i .
Similar to [26], the cosine similarity,denoted as cosim , is exploited for the corresponding saliency grids of Q i and TQ i :
[Equation]
Surface Constraint Loss As discussed in [15], 3D keypoints are encouraged to close to the input.
They propose a loss to constrain the distance between the keypoint and its nearest neighbor from theinput.
Yet, the generated keypoints are inconsistent when given the same input but with a differentdensity.
Thanks to the shape decoder, SNAKE can reconstruct the underlying surface of the input,which is robust to the resolution change.
Hence, we use the occupancy probability to represent theinverse distance between the query and the input.
As can be seen in Fig.
2-(surface constraint), weenforce the saliency of the query that is far from input P close to 0, which is defined as
[Equation]
Sparsity Loss Similar to 2D keypoint detection methods [26], we design a sparsity loss to avoid thetrivial solution ( Prob s ( Q | P ) =0) in Eq.
As can be seen in Fig.
2, the goal is to maximizethe local peakiness of the local saliency grids.
As the sailency values of non-occupied points areenforced to 0 by L m , we only impose the sparsity loss on the points with high occupancy probability.
Hence, we derive the sparsity loss with the help of decoded geometry by
[Equation]
[Equation]
where Q 1 i = { q | q ∈ Q
i , Prob o ( q | P ) > 1 − thr o } , thr o ∈ (0 , 0 . 5] is a constant, and n is the numberof grids.
It is noted that the spatial frequency of local peakiness is dependent on the grid size 1 /U ,see 4.4.
Since the network is not only required to find sparse keypoints, but also expected to recoverthe object shape, it would generate high saliency at the critical parts of the input, like joint points of adesk and corners of a house, as shown in the Fig.
2-result.
3.3 Explicit Keypoint Extraction
The query point q whose saliency is above a predefined threshold thr s ∈ (0 , 1) would be selected asa keypoint at the inference stage.
Although SNAKE can obtain the saliency of any query point, ahigher resolution query set results in a high computational cost.
Hence, as shown in Fig.
2-inference, we build a relatively low-resolution query sets Q infer which are evenly distributed in the input spaceand further refine the coordinates of Q infer by gradient-based optimization on this energy function:
[Equation]
Specifically, details of the explicit keypoint extraction algorithm are summarized in Alg. 1.
Experiment
In this section, we evaluate SNAKE under three settings.
First, we compare keypoint semanticconsistency across different instances of the same category, using both rigid and deformable objects.
Next, keypoint repeatability of the same instance under disturbances such as SE(3) transformation,noise and downsample is evaluated.
Finally, we inspect the point cloud registration task on the3DMatch benchmark, notably in a zero-shot generalization setting.
Besides, an ablation study is doneto verify the effect of each design choice in SNAKE.
The implementation details and hyper-parametersfor SNAKE in three settings can be found in the supplementary.
4.1 Semantic Consistency
Datasets The KeypointNet [38] dataset and meshes generated with the SMPL model [17] are utilized.
KeypointNet has numerous human-annotated 3D keypoints for 16 object categories from ShapeNet [3].
The training set covers all categories that contain 5500 instances.
Following [37], we evaluate 630unseen instances from airplanes, chairs, and tables.
SMPL is a skinned vertex-based deformablemodel that accurately captures body shape variations in natural human poses.
We use the samestrategy in [37] to generate both training and testing data.
Metric Mean Intersection over Union (mIoU) is adopted to show whether the keypoints acrossintra-class instances have the same semantics or not.
For KeypointNet, a predicted keypoint isconsidered the same as a human-annotated semantic point if the geodesic distance between themis under some threshold.
Due to the lack of human-labeled keypoints on SMPL, we compare thekeypoint consistency in a pair of human models.
A keypoint in the first model is regarded semanticallyconsistent if the distance between its corresponding point and the nearest keypoint in the secondmodel is below some threshold.
[Figure]
Evaluation and Results We compare SNAKEwith random detection, hand-crafted detectors:
ISS [41], Harris-3D [29] and SIFT-3D [27], and
DL-based unsupervised detectors: USIP [15] and
UKPGAN [37].
As USIP has not performed semantic consistency evaluations, we train the modelwith the code they provided.
We follow the sameprotocols in [37] to filter the keypoints via NMSwith a Euclidean radius of 0.1.
Quantitative results are provided in Fig.
5-(a,e).
SNAKE obtainshigher mIoU than other methods under most thresholds on KeypointNet and SMPL.
Qualitativeresults in Fig.
3 show our keypoints make good alignment with human annotations.
Fig.
4 providesqualitative comparisons of semantically consistent keypoints on rigid and deformable objects.
Owingto entangling shape reconstruction and keypoint detection, SNAKE can extract aligned representationfor intra-class instances.
Thus, our keypoints better outline the object shapes and are more semantically consistent under large shape variations.
As shown in the saliency field projected slices, we canget symmetrical keypoints, although without any explicit constraint like the one used in [37].
4.2 Repeatability
Datasets ModelNet40 [36] is a synthetic object-level dataset that contains 12,311 pre-aligned shapesfrom 40 categories, such as plane, guitar, and table.
We adopt the official dataset split strategy.3DMatch [40] and Redwood [5] are RGB-D reconstruction datasets for indoor scenes.
Following [15],we train the model on 3DMatch and test it on Redwood to show the generalization performance.
Thetraining set contains around 19k samples and the test set consists of 207 point clouds.
[Figure]
[Figure]
Metric We adopt the relative repeatability proposed in USIP [15] as the evaluation metric.
Given twopoint clouds captured from different viewpoints, a keypoint in the first point cloud is repeatable if itsdistance to the nearest keypoint in the other point cloud is below a threshold ϵ .
Relative repeatabilitymeans the number of repeatable points divided by the total number of detected keypoints.
Evaluation and Results Random detection, traditional methods and USIP are chosen as our baselines.
Since UKPGAN does not provide pre-trained models on these two datasets, we do not report itsresults in Fig.
5 but make an additional comparison on KeypointNet, which is summarized inthe supplymentary.
We use NMS to select the local peaky keypoints with a small radius (0.01normalized distance on ModelNet40 and 0.04 meters on Redwood) for ours and baselines.
Wegenerate 64 keypoints in each sample and show the performance under different distance thresholds ϵ ,downsample rates, and Gaussian noise scales.
We set a fixed ϵ of 0.04 normalized distance and 0.2meters on the ModelNet40 and Redwood dataset when testing under the last two cases.
As shown in [Figure] Fig.
5-(b,f), SNAKE outperforms state-of-the-art at most distance thresholds.
We do not surpass USIPon Redwood in the lower thresholds.
Note that it is challenging to get higher repeatability on Redwoodbecause the paired inputs have very small overlapping regions.
Fig.
5-(c,d,g,h) show the repeatabilityrobustness to different downsample rates and noise levels.
SNAKE gets the highest repeatability inmost cases because the shape-aware strategy helps the model reason about the underlying shapes ofthe objects/scenes, which makes keypoints robust to the input variations.
Fig.
6 provides visualizationof object-level and scene-level keypoints of the original and disturbed inputs.
SNAKE can generatemore consistent keypoints than other methods under drastic change of inputs.
4.3 Zero-shot Point Cloud Registration
Datasets We follow the same protocols in [37] to train the model on KeypointNet and then directlytest it on 3DMatch [40] dataset, evaluating how well two-view point clouds can be registered.
Thetest set consists of 8 scenes which include some partially overlapped point cloud fragments and theground truth SE(3) transformation matrices.
Metric To evaluate geometric registration, we need both keypoint detectors and descriptors.
Thus,we combine an off-the-shelf and state-of-the-art descriptor D3Feat [1] with our and other keypointdetectors.
Following [37], we compute three metrics: Feature Matching Recall, Inlier Ratio, and
Registration Recall for a pair of point clouds.
Evaluation and Results As baselines, we choose random detection, ISS, SIFT-3D, UKPGAN,and D3Feat.
Note that D3Feat is a task-specific learning-based detector trained on the 3DMatchdataset, thus not included in this zero-shot comparison.
Ours and UKPGAN are trained on thesynthetic object dataset KeypointNet only.
The results are reported under different numbers ofkeypoints ( i.e. , 2500, 1000, 500, 250, 100).
The NMS with a radius of 0.05m is used for D3Feat,
UKPGAN, and ours.
As shown in Table 1, SNAKE outperforms other methods consistently underthree metrics.
For registration recall and inlier ratio, we achieve significant gains over UKPGAN andother traditional keypoint methods.
Notably, when the keypoints are high in numbers, SNAKE evenoutperforms D3Feat which has seen the target domain.
Local shape primitives like planes, corners, orcurves may be shared between objects and scenes, so our shape-aware formulation allows a superiorgeneralization from objects to scenes.
[Table]
[Figure]
U
[Figure]
4.4 Ablation Study
[Figure]
Loss Function Table 2 reports the performance w.r
[Table]
t. designs of loss functions.
(Row 1) If theoccupancy decoder is removed, the surface constraint cannot be performed according to Eq.
( 6), sothey are removed simultaneously.
Although the model could detect significantly repeatable keypointson
ModelNet40 [36], it fails to give semantically consistent keypoints on KeypointNet [38].
Fig.
7-ashows that SNAKE is unable to output symmetric and meaningful keypoints without the shape-awaretechnique.
That indicates the repeatability could not be the only criterion for keypoint detection if animplicit formulation is adopted.
(Row 2-4)
Each loss function for training keypoint field is vital forkeypoint detection.
Note that the model gives a trivial solution ( 0 ) for the saliency field and cannotextract distinctive points when removing the sparsity loss.
[Table] [Table]
[Table] [Table]
Volumetric Resolution The grid size 1 /U controls the number of keypoints because
L s enforces the model to predict a single local maxima per grid of size (1 /U ) 3 .
Fig.
7-b showsdifferent saliency field slices obtained from the same input with various 1 /U .
When U is small,
SNAKE outputs fewer salient responses, and more for larger values of U .
We also give the relativerepeatability results on ModelNet40 under distance threshold ϵ = 0 . 04 in Table 3, indicating that
U = 6 gives the best results.
From Table 4, we can see that higher resolution improves performance.
However, the performance drops when it reaches the resolution of 80.
The potential reason is as such:the number of queries in a single grid increases when the resolution becomes higher, as mentioned in3.2.
In this case, finer details make the input to cosine similarity too long and contain spurious values.
Step and Learning Rate Fig.
7-c shows the importance of optimization (see Alg.
1)for refining keypoint coordinates on the ModelNet40 dataset.
It is noted that too many optimizationsteps will not bring more gains but increase the computational overhead.
In this paper, we set thenumber of update steps to 10.
The learning rate for optimization is also key to the final result.
Whenthe learning rate is set to 0.1, 0.01, 0.001 and 0.0001, the relative repeatability (%) on ModelNet40dataset with the same experimental settings as Table 4 are 0.002, 0.622, 0.854 and 0.826, respectively.
Conclusion and Discussion
We propose SNAKE, a method for 3D keypoint detection based on implicit neural representations.
Extensive evaluations show our keypoints are semantically consistent, repeatable, robust to downsample, and generalizable to unseen scenarios.
Limitations.
The optimization for keypoint extractionduring inference requires considerable computational cost and time, which may not be applicablefor use in scenarios that require real-time keypoint detection (see supplementary).
Negative Social
Impact.
The industry may use the method for pose estimation in autonomous robots.
Since ourmethod is not perfect, it may lead to wrong decision making and potential human injury.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b
) Did you describe the limitations of your work
Yes ] See Section 5.
c) Did you discuss any potential negative societal impacts of
your
work?
[Yes] See Section (d Have
you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See supplemental material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes] See supplemental material.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See supplemental material.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] See supplemental material.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]See supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[Yes] See supplemental material.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] See supplemental material.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]