NCP: Neural Correspondence Prior for Effective Unsupervised Shape Matching
Abstract
We present Neural Correspondence
Prior (NCP), a new paradigm for computing correspondences between 3D shapes.
Our approach is fully unsupervised and can lead to high-quality correspondences even in challenging cases such as sparse point clouds or non-isometric meshes, where current methods fail.
Our first key observation is that, in line with neural priors observed in other domains, recent network architectures on 3D data, even without training , tend to produce pointwise features that induce plausible maps between rigid or non-rigid shapes.
Secondly, we show that given a noisy map as input, training a feature extraction network with the input map as supervision tends to remove artifacts from the input and can act as a powerful correspondence denoising mechanism, both between individual pairs and within a collection.
With these observations in hand, we propose a two-stage unsupervised paradigm for shape matching, by (i) performing unsupervised training by adapting an existing approach to obtain an initial set of noisy matches, and (ii) using these matches to train a network in a supervised manner.
We demonstrate that this approach significantly improves the accuracy of the maps, especially when trained within a collection.
We show that NCP is data-efficient, fast, and achieves state-of-the-art results on many tasks.
Our code can be found online: https://github.com/pvnieo/NCP .
Introduction
Establishing dense correspondences between 3D shapes is a fundamental problem in computer vision and computer graphics, as it enables many downstream applications such as statistical shape analysis [1, 2], texture [3] and deformation transfer [4, 5], and registration [6], to name a few.
A particularly challenging setting for this task is the computation of 3D correspondences between shapes that undergo non-rigid, non-isometric deformations and that may exhibit some partiality, such as missing semantic parts and can be represented as sparse point clouds.
The standard approach is to formulate shape correspondence as a supervised learning problem, by relying on ground truth maps within large datasets of shape pairs.
Given such ground truth, it is possible to train neural networks to either produce deformation fields [7], segmentation maps [8, 9, 10] or functional maps [11, 12, 13] between the input shapes.
However, all these methods rely on the presence of labeled point-to-point correspondences, which are expensive to obtain, and are only available in a handful of cases.
At the same time, unsupervised techniques try to solve the matching problem by imposing structural properties on the maps, either in the intrinsic [14, 15, 16] or extrinsic domains [17, 18].
However, the priors used by these methods tend to be purely geometric (e.g., near isometry, divergence-free deformation fields), and, as we demonstrate below, are not always applicable, especially in challenging non-isometric settings.
In this work, we demonstrate that the neural network itself can act as a powerful prior for shape correspondence problems.
Specifically, we first observe that even without training, recent architectures for 3D shape analysis [19, 20, 21] produce pointwise features that capture local geometry and lead to well-structured maps between shapes.
Secondly and perhaps more importantly, we demonstrate an effect that we call neural correspondence prior .
Specifically, given noisy input maps, we formulate a supervised learning problem, where we aim to learn pointwise features that, when compared, would recover the input maps.
Remarkably, we show that the correspondences computed by the trained networks are typically of higher quality than the input.
Our results are consistent with the notion of noise impedance observed in other works on neural priors, pioneered in [22], and neural bias [23], which states that neural networks tend to optimize lower frequency signals before higher-frequency ones.
We show that in the context of shape matching, these effects imply that neural networks can learn powerful features by training to overfit to given noisy or incoherent correspondences.
Based on these observations, we develop a two-stage algorithm for unsupervised shape matching.
In the first stage, we adapt an existing unsupervised neural network, using variants of standard methods from the literature.
We then train a new network from scratch in a supervised manner using the noisy maps as ground truth.
We demonstrate that this second stage not only helps to denoise the maps but also improves the matching result overall and achieves state-of-the-art results on multiple tasks.
We show that neural correspondence prior is applicable within both collections and on individual shape pairs, to provide test time refinement of dense input maps.
Since this method makes no assumptions about the topology of the shapes, it can be used for complex cases such as non-isometric matching.
Overall, our contributions can be summarized as follows:
• We demonstrate that untrained neural networks for 3D shapes can produce pointwise features that are on par with complex axiomatic local descriptors.
• We show that when trained using artifact-laden maps for supervision, the features learned by neural networks lead to correspondences that are more coherent and of higher quality than the input.
We call this effect Neural Correspondence Prior .
• Based on these insights, we develop a two-stage algorithm for unsupervised 3D non-rigid shape matching that achieves SOTA results for difficult matching scenarios such as nonisometric and point cloud matching.
• Using our unsupervised point-to-point correspondence method, we propose an approach for few-shot keypoint detection.
Related Work
Shape matching is a well-studied area in computer vision and graphics, and a full overview is beyond the scope of this paper.
We refer the interested reader to recent surveys [24, 25, 26, 27] for a more in-depth treatment of this field.
Below, we review methods that are most related to our work, with a focus on learning-based techniques, both supervised and unsupervised.
3D Non-Rigid Shape Correspondence There is a vast literature on learning non-rigid shape correspondences.
In terms of supervised methods, an important direction is to consider the matching problem as a dense semantic segmentation problem where the labels are vertices on some template shape [8, 9, 10], or by mapping via template deformation [7].
However, these techniques are known to require a considerable amount of data, and may fail when discretization changes [19, 11].
Another approach is to use the functional map framework that was introduced in [28] and extended in many follow-up works [29, 30, 31, 32, 33, 34, 35, 36, 37] to name a few (see [38] for an overview).
The functional map (fmap) framework is based on encoding and optimizing maps as small matrices in a reduced basis.
The fmap formulation has been successfully used in the supervised setting to establish correspondences between complete and partial non-rigid shapes [11, 12, 13].
However, it is typically restricted to near-isometric matching between relatively clean discretizations.
The functional map framework has also been adapted to the unsupervised setting, either by imposing structural properties on the functional maps such as bijectivity, orthogonality, and commutativity with the Laplacian [15, 39], or by enforcing cycle consistency [40], geodesic distortion preservation [14], or by combining intrinsic and extrinsic properties [17, 18].
However, these methods all assume that the shapes are nearly isometric, not partial, and thus fail in these contexts.
Another line of work proposes to do unsupervised shape matching by reconstructing the input shapes in canonical order.
These approaches are based on the encoder-decoder architecture, and find matches either by nearest neighbor in the latent space [41], or by nearest neighbor on the reconstruction of the two input shapes [42, 43].
However, these methods require a lot of shapes and learning time, and have only successfully been applied to man-made shapes that do not undergo significant non-rigid deformations.
Neural Prior
The neural prior was first introduced in the seminal work of Ulyanov et al.
[22], where the authors introduced the Deep Image Prior (DIP), and showed that a randomly initialized convolutional neural network can serve as a good prior for many 2D inverse problems, such as inpainting and denoising.
Since then, several modifications and improvements have been made to
DIP [44, 45, 46, 47, 48], in order to improve its performance or adapt it to new tasks.
Another line of work [49, 50] has theoretically studied DIP either through the lens of regularization theory or through its connection to Gaussian processes.
Despite its success in 2D computer vision, neural prior has been little exploited in the 3D domain, the only application being shape reconstruction.
Indeed, [51, 52] attempt to reconstruct a water-tight mesh from a point cloud, either by overfitting several local patches to the point cloud or by shrinking an input mesh by a neural network.
The objective of our work is to fill this gap and to propose a neural prior technique for 3D matching.
Unsupervised Keypoint Detection Unlike the 2D case, [53, 54, 55, 56], unsupervised 3D keypoint detection is relatively under-explored in the literature.
The recently introduced KeyPointNet [57] benchmark provides a good testing bed.
Prominent unsupervised keypoint detection methods are based on a self-supervised paradigm using an encoder-decoder network, with a reconstruction error.
The encoder of Skeleton Merger [58] predicts a set of salient keypoints, that the decoder uses to produce a skeleton of the shape.
UKPGAN [59] harnesses the power of GANs to detect keypoints, by forcing the decoder to reconstruct the input shape using solely the set of keypoints produced by the encoder.
Finally, KeypointDeformer [60] predicts a set of keypoints that allow for efficient shape manipulation.
In particular, given a source and a target shape, the keypoints predicted on the source shape are used to deform it to the target shape using a cage skinning deformer [61].
Motivation & Method Overview
Our main goal is to develop an unsupervised learning-based shape correspondence method that would take as input a pair of shapes M and N and produce a dense correspondence, also called a map,
T : M → N .
For this, we explore the direction of neural priors and show how they can be used for 3D shape matching.
The motivation behind this direction is twofold.
First, as in the 2D case [62, 46], we hope that the structure of an untrained 3D neural network captures the low-level statistics of a single 3D shape, and thus can enable fine-tunning using a single pair.
Second, as demonstrated in
DIP [22], deep neural networks have high noise impedance, which allows them to learn good features from noisy inputs.
This property can be used to guide the network to good local minima while trying to use, in our case, noisy point-to-point (p2p) matches for supervision.
Feature embeddings.
In this work, we formulate the shape-matching problem for both man-made and organic shapes, by learning pointwise features.
Such features can be used to compute correspondences either through nearest neighbor search or with minimal post-processing.
We focus on features that are learned by a neural network.
Specifically, the network N takes as input a 3D shape M and produces, as output, a feature vector N ( M ) x ∈ R d , for every point of the shape x ∈ M .
When considering the entire object as a whole, we call the set N ( M ) x for all x ∈ M a feature embedding of shape M .
Our method is based on two main observations: neural bias for untrained networks, and an effect that we call neural correspondence prior .
Below we describe each of these effects and then describe our unsupervised shape-matching method in Sec. 4.1.
3.1 Neural bias for pointwise features
Our first observation is that feature embeddings produced by neural networks have a particular structure, which can be exploited in the context of computing features for shape correspondence.
To highlight this effect we performed the following experiment.
We take the test set of the FAUSTRemeshed ( FR ) and SCAPE-Remeshed ( SR ) datasets introduced in [33], and used in many recent works [11, 17, 39], and compute the correspondences using the pointwise features extracted by a
DiffusionNet network [19] with randomly set weights.
We use the default variant of DiffusionNet, which takes as input the XYZ coordinates of the shapes and produces a 128-dimensional feature
[Figure]
vector for every point.
We emphasize that the weights of the network are set randomly and without any training .
We then used the extracted features to produce p2p maps either via a simple nearest neighbor (NN) search in the feature space or with the functional map framework [63] (FMAP).
We compare the results to maps produced using the same procedure with classical axiomatic features such as SHOT [64], or using other axiomatic and training-based non-rigid correspondence methods.
As shown in Tab. 1, remarkably, the features computed with an untrained DiffusionNet network perform on par or better than axiomatic features and even outperform the supervised learning method, FMNet [65], based on training MLPs to refine pre-defined axiomatic features, as well as recent unsupervised extensions of that approach.
We attribute the fact that a randomly-initialized
DiffusionNet performs better than trained point-wise MLPs to the spatially-aware nature of the architecture of the network, which uses diffusion to simulate intrinsic convolution, thereby providing a strong neural prior [66, 67, 52].
Notice that using untrained features for matching was recently used in the context of 2D keypoint matching in [68].
3.2 NCP: Neural Correspondence Prior
While the previous effect relates to properties of untrained networks, in this work, we also observe another, complementary phenomenon that we call Neural Correspondence Prior , which we exploit in our approach below to train powerful feature extractors.
This effect can be formulated as follows.
Suppose we are given a fixed map T MN , between some shapes M and N .
We can formulate an optimization problem, where we aim to learn the feature embedding of M and N so that the induced map, e.g., computed via nearest neighbors in the feature space: i.e. x → arg min y ∥
N ( M ) x −
N ( N ) y ∥ , is as close as possible to T MN .
Note that unlike the neural bias mentioned in the previous section, here we formulate an optimization problem, where the parameters of the network are optimized to fit the given input map.
Our key observation is that when the target map T is noisy or approximate, networks tend to produce well-structured feature embeddings until the very late stages of the optimization.
I.e., throughout the early stages of optimization, the maps computed by the optimized features tend to be of higher accuracy than the input noisy map used for supervision.
To demonstrate the effect of NCP quantitatively, we performed an experiment in which we corrupted the ground truth map between a pair of shapes from the FR dataset, and then trained the DiffusionNet network, with a gradient descent optimizer using the standard NCE loss [72] to overfit to those corrupted maps.
Specifically, to supervise this training, we use: ( a ) the ground truth (GT) map between the shapes, ( b ) the GT map where 25% of its entries have been assigned to random vertices, ( c ) the GT map with 50% noise, ( d ) the GT map with 75% noise.
The results of this experiment are shown in Fig.
We plot both the NCE loss being optimized as well as the geodesic error of the intermediate maps with respect to the ground truth.
As shown in Fig.
1, while the network can easily adapt to the correct map, it has difficulty minimizing the loss in the case of noisy maps.
Moreover, and perhaps more remarkably, observe that the geodesic error starts to decrease towards lower values than the input map , in the first iterations of optimization and diverges only in the later stages, when the network starts to overfit to the noise in the map.
This highlights that the intermediate maps tend to be of higher quality than the supervisory signal, and implies that such optimization with a neural network and early stopping can be used as an effective map denoising mechanism (Sec. 4.2).
It should be mentioned that this effect does not depend on the shape pair, and other examples of different pairs are provided in Sec.
5.1.2 and the supplementary.
We observe this effect to hold very broadly in shape correspondence problems.
For a single shape pair with a noisy input map, eventually, any map can be learned by the network if the parametrization and the number of training iterations are sufficient.
However, the network architecture strongly regularizes the map search space, providing low impedance to ‘signal’ and high impedance to ‘noise’, resulting in an optimization trajectory that either converges to a good local minimum or passes near one [22].
Furthermore, in the presence of shape collections , this effect is even stronger and, as we demonstrate below, significantly helps to regularize computed features so that no early stopping or post-processing of the results becomes necessary (see our Algorithm 1 that we explain in detail in Sec. 4.1).
In addition to the network architecture itself, we attribute the Neural Correspondence Prior to the spectral bias principle [23, 73], which states that neural networks tend to learn low frequencies in the early stages of training and that low frequencies are more robust to random perturbations of network parameters.
Since we formulate shape matching as the problem of computing optimal features, spectral bias, in our context, implies that even given a noisy or artifact-laden map as input, the feature embeddings produced by the optimized neural network tend to be smooth , especially in the early stages of the optimization.
In other words, it is significantly harder for the network to produce a noisy, high-frequency feature embedding than a smooth one.
Furthermore, a smooth feature embedding will tend to produce smooth correspondences between shapes, as suggested by the following theorem, which we state here and prove in the supplementary.
Theorem 1.
Let M and N be two compact smooth surfaces (smooth manifolds of dimension 2), and let M : M ψ −→ R d (resp.
N : N ψ −→ R d ) be their respective feature embeddings produced by a neural network ψ .
If the feature embeddings produced by ψ are injective and smooth, i.e., the mapping ψ : M → R d , x → ψ ( x ) , ∀ x ∈ M is smooth and invertible on its image (and similarly for N ), then up to an arbitrarily small perturbation of M and N , the map T nn : M → N given by
T nn ( x ) =
arg min y ∈N
∥ ψ ( x ) − ψ ( y ) ∥ is smooth up to sets of measure 0.
This result highlights the fact that smooth feature embeddings, generically, translate into smooth maps, which is a desirable property in most typical correspondence scenarios.
Method Description
Based on the observations above, in this section, we introduce contexts in which the NCP can be exploited, resulting in two algorithms: one for unsupervised shape matching, and the second for p2p map denoising.
4.1 NCP within a shape collection
Given a collection of shapes {N i } , we propose to exploit the NCP in two ways.
The first configuration assumes that we are also given a collection of artifact-laden maps { T j } between some shape pairs in {N i } as input.
In that case, we propose to train a randomly-initialized neural network to produce feature embedding for each shape, s.t., the induced maps are as close as possible to the target maps { T j } .
We provide the exact choice of the loss function depending on the nature of the underlying shapes in Sec.
5 below.
In the second configuration, no maps are given as input.
In this case, we propose a two-stage pipeline where in the first stage, we adapt an existing unsupervised correspondence method to obtain possibly approximate very noisy initial correspondences.
We then formulate a supervised learning problem, where we learn feature embeddings that would induce these computed correspondences as a target.
Once the network is trained, we establish final correspondences via a simple nearest neighbor search in the feature space.
Note that the trained network can be used to establish correspondences both between shapes within the given shape collection, but also across new, never seen shape pairs.
In our evaluation below, and unless otherwise noted, we always evaluate on a test set, never seen during training.
We dubbed the resulting algorithm NCP-UN , and we summarize it in Algorithm 1.
This is the algorithm that is used throughout most of our experiments.
We argue that it is advantageous to perform the second step for the entire shape collection, instead of doing it for each pair individually for two main reasons: first, it is faster and avoids the need for test-time optimization (see Sec. 4.2), and second, the collection provides an additional layer of regularization.
In fact, since we optimize for features on each shape, it is difficult to compute features that would overfit to all O ( N 2 ) noisy maps in a collection of N shapes, as the artifacts across pairs are typically not consistent.
We observe that the noise impedance property is even stronger in the case of a collection, and the produced maps are better than if the refinement is done per pair.
Intuition behind stage 2 of the algorithm
Our intuition for Stage 2 is threefold.
First, and motivated by the results of Sec.
3.2 (the NCP effect), we showed that neural networks tend to have difficulty overfitting to corrupted inputs, and tend to favor smooth outputs.
Thus, in the process of over-fitting to approximate maps by a neural network, the output maps tend to be of better quality than the input.
This motivates the use of artifact-laden maps as supervision for a network thus capitalizing on the NCP effect to recover better correspondences.
Second, as we mentioned above, given a collection of approximate maps, their errors tend to be inconsistent, and training a network using such maps for supervision adds an extra layer of regularization, as it is difficult for the network to adapt to all inconsistent errors on all pairs at once.
Finally, in our formulation of Stage 2, we learn one feature embedding per shape and compute correspondences via nearest neighbor search between feature embeddings.
This also adds strong regularization, since it makes it difficult for the network to learn a feature embedding, which can reproduce errors in the maps between all shape pairs.
For our applications (see Sec. 5), we used the following design choices.
For the UM method, we used the unsupervised functional map method [15, 39], with a DiffusionNet [19] backbone for 3D meshes, and Point-MLP [74] backbone for point clouds.
The same backbone was used as RN for step 3.
in our Algorithm 1.
Concerning the p2p loss in step 3, we used the NCE loss [72] for 3D meshes and the LIE loss [75] for point clouds.
We also used a variant of our approach, dubbed
NCP-UN fmap , that uses the FMAP framework and loss from [11] in steps 3 and 5 of Algorithm 1, especially on organic shapes, where the Laplacian basis tends to be of high quality.
We want to emphasize, first, that in all of our applications below, we followed the best practices depending on the nature of shapes (e.g., organic shapes vs. point clouds), and second, our observations regarding NCP are independent of the choice of loss and network architecture.
A demonstration of the general nature of NCP, as well as more details about the implementation, are provided in the supplementary.
Across all cases, we observe a significant improvement in results when using the second stage of our NCP-UN pipeline (supervised learning) compared to the initial input maps.
Finally, we want to emphasize that it is possible to repeat Stage 2 several times, and we observed that there is a slight improvement in some cases, but most often it stagnates after the first iteration.
We experimented with this and chose to keep the method simple and use only one iteration thus, avoiding an additional tunable hyperparameter.
4.2 Test Time Denoising
In addition to the previous setting, we also observe that NCP can be used to denoise a correspondence when given a single shape pair.
Specifically, given a noisy initial map, we train a neural network to produce feature embeddings that would induce the given input map, used as supervision.
Again, we tailor the exact choice of loss to the nature of the shapes and describe one option in Sec.
5.1.2 below.
Test time denoising can be advantageous in the absence of a shape collection.
At the same time, as mentioned in the original DIP work [22], in the case of a single shape pair, early stopping must be used in order to stop training at the best local minimum.
For the shape-matching task, we propose to use a cycle loss based on bijectivity as the criterion for stopping.
Given a pair of shapes M , N , and two maps between them Π MN and Π NM computed via nearest neighbors between optimized features, and represented as soft binary matrices (see Eq.
(4) in the supplementary materials for computation details), we compute L cycle =
∥
X N − Π NM Π MN X N ∥ 2 F , where X N is the matrix of
XYZ coordinates of shape N .
We overfit the network for a number of iterations and stop when L cycle stops improving after a predetermined patience period.
The weights that produce the minimum value of L cycle are used to compute the pointwise features that we then use for extracting the p2p map, using the nearest neighbors in the feature space.
4.3 Correspondences as a Tool for Downstream Tasks
Given computed correspondences, we also use them to solve multiple downstream tasks.
Indeed, since our proposed algorithm is unsupervised, we can use its output, i.e., p2p maps, to transfer multiple quantities between shapes, such as keypoints.
For this, we only need a few labeled examples with the quantities we want to transfer, resulting in a few-shot algorithm.
In the following, we present a method for few-shot keypoint detection, dubbed FSKD hereafter, and based on the p2p maps obtained by our unsupervised algorithm NCP-UN .
A particular challenge in this setup is that some keypoints and parts appear exclusively on some shapes and not in others, and so to account for the keypoint appearing there, multiple labeled source shapes must be used.
We begin our FSKD by randomly selecting N shapes that cover the largest set of keypoints.
These
N shapes will be labeled and we transfer their keypoints to the shapes in the test set, in order to “discover” their keypoints.
Given a target shape, FSKD is composed of three steps: 1.
Detection of potential keypoints by transferring keypoints from labeled shapes, 2.
Filtering to remove keypoints that are likely not to exist on the target shape, 3.
Combination: merge transferred keypoints if multiple points on the target shape are assigned to the same keypoint ID.
We provide the implementation details for these three steps in the supplementary.
Results & Applications
In this section, we provide results in a wide range of challenging tasks, showing the efficiency and robustness of our approach to different types of data and tasks.
In particular, we consider the tasks of shape correspondence between man-made shapes in Sec.
5.1.1, non-rigid shape correspondence in
Sec. 5.1.2, part segmentation in Sec. 5.2, and few-shot keypoint detection in Sec.
The fact that our model does not rely on a geometric prior allows it to provide good results on different types of data, including isometric and non-isometric, organic, and man-made data.
Datasets We conduct our experiments on four different datasets, both on point clouds and 3D meshes.
To the best of our knowledge, there is no point cloud dataset that provides dense pointto-point ground-truth correspondences.
For this purpose, we follow the setup of [41], and use
KeyPointNet dataset [57] for the man-made shape correspondence experiment in Sec.
We also use this dataset for our few-shot keypoint detection experiment in Sec.
KeypointNet is composed of over 8K models and 100K semantic keypoints labeled by professional humans.
We use the same train/validation/test splits as the original paper [57].
For the part segmentation experiment in Sec.
5.2, we use the PartNet dataset [76], which is composed of 16881 shapes of 16 categories, where each category has a different number of parts ranging from 2 to 6.
For the 3D meshes, we use the organic, non-isometric non-rigid SMAL-based [77, 36] dataset and the
SHREC’20 Non-Isometric benchmark [78] for our non-rigid correspondence experiment in Sec. 5.1.2, as these are challenging non-isometric datasets.
The former dataset is composed of 50 animal shapes represented as 3D meshes.
25 shapes are used for training, and the remaining 25 are used for testing.
It should be noted that the set of animals and poses seen in training is different from the one used in testing.
The SHREC’20 benchmark consists of 14 scans of different animals, some of which contain holes and topological deformations.
This, in addition to the fact that the ground truth maps are given as a set of sparsely annotated keypoints, means that most supervised methods cannot be applied.
5.1 3D Shape Correspondences 5.1.1 Correspondences on Point Clouds
We test the quality of the matches produced by NCP-UN on sparse point clouds using the KeyPointNet dataset.
We train the first stage of our method using the unsupervised functional map method [15, 39], with a PointMLP backbone [79] and XYZ coordinates as input, and impose the bijectivity loss on the functional map.
The second stage is trained using LIE loss, and maps are extracted using the nearest neighbor in the feature space.
For testing, we follow the same setup as [41], by generating pairs of shapes between all point clouds in a given category and removing the pairs that do not share the same set of semantic keypoints.
We compute the L2 distance between the transferred keypoint and the ground truth one.
Our method is evaluated against state-of-the-art learning-based 3D dense correspondence prediction approaches, including AtlasNetV2 [80], FoldingNet [81], IDC [43], CPAE [41], in addition to the nearest neighbor baseline between the XYZ coordinates of the points, which turns out to be a competitive baseline, not considered by previous methods.
In Fig.
2, we report the percentage of testing pairs where the distances between predicted and ground truth maps are below a given threshold for 8 categories and report the remaining categories in the supplementary.
It can be seen that our method outperforms all methods in the 8 categories, and achieves state-of-the-art results.
Note that our method obtains SOTA results although the method used in stage 1 is not fully designed to work on point clouds (the quality of the Laplacian is not good, and the used bijectivity loss is weak).
We believe that using a more adapted method for the first stage will further improve the results.
5.1.2 Correspondences Between Non-Rigid Meshes
Evaluation Protocol We follow the Princeton protocol [82] for evaluating all correspondences, as used in all recent works.
Specifically, we compute the pointwise geodesic error between the predicted maps and the ground truth, normalized by the square root of the surface area of the target shape.
For the non-rigid non-isometric application, we trained the unsupervised functional map network from [19] and used NCE loss for the second stage (step 3.
in Algorithm 1).
P2P maps were obtained using nearest neighbors between feature embeddings.
Since the SHREC’20 dataset is small (onlyshapes), we follow previous work [83] by training and testing on the same set, whereas for SMAL, the training and testing sets are different.
For a fair evaluation, we compared our method to many state-of-the-art supervised and unsupervised methods.
For the supervised methods, we compared to
FMNet [13], GeomFMaps [11], GeomFMaps + DiffusionNet [19] and LIE [75].
For the unsupervised methods, we compared our method to WSupFMNet [39], Smooth Shells [18], Deep Shells [17], NeuroMorph [83] and WSupFMNet + DiffusionNet [19].
As shown in Tab. 2 and Fig.
3, our method achieves state-of-the-art results among unsupervised methods and supervised methods.
The majority of unsupervised methods and supervised methods fail because of the near-isometry assumption they make.
Since our method makes no assumptions about the topology of the shape, it can perform very well in difficult scenarios such as non-isometric animals.
It should be noted that these results would not be possible without the second stage of our algorithm, which is enabled by NCP.
For example, for the SMAL dataset, the result of the first stage
[Figure] [Figure]
is 7.8 , and the second stage improves this result to 5.8 , which is a 25% improvement, achieving state-of-the-art results.
We provide in Fig.
4 some qualitative results showing the performance of our algorithm against the state-of-the-art NeuroMorph method on the SMAL dataset.
It can be seen that our method produces visually plausible maps, especially after applying the second stage.
Test Time Optimization To show the utility of test time optimization, as well as the early stopping criteria we introduced, the performed the following experiment.
After training the first stage of our algorithm, the trained model was used to predict p2p on the test set of SMAL dataset.
Pairs of shapes are then created, and we use our test time optimization method (see Sec. 4.2) to improve upon the maps produced by the first stage.
Results are reported in Tab. 2, see row “Ours (Test-Time
Optimization)”.
It can be seen that similarly to the previous case, training a random network on top of artifact-laden maps helps to denoise them.
We also show in Fig.
4 some qualitative results that illustrate the effect of test time optimization and compare it to NCP-UN.
The results of this experiment confirm our hypothesis that doing the second stage on a collection, instead of individual pairs, provides a second layer of regularization, as the results are better, in addition to improving computational complexity.
In fact, the second stage using the whole collection takes around 15 minutes , meanwhile, doing it on each pair individually takes around 160 minutes , which represents more than 10 × improvement in running time.
5.2 Part Segmentation Transfer
We further validate our approach on the part label transfer task on the PartNet dataset following the setup of [41].
Result are summarized in Tab. 3, where the average intersection-over-union IOU is reported.
Our method outperforms the baselines on 14 out of 16 categories, achieving state-of-the-art result by more than 3% IOU on average.
In classes with large intra-class variations such as Lamps, our method outperforms the baselines by up to 8 % average IOU points.
In addition to being performant,
[Table]
[Figure]
our method is extremely fast.
In fact, training both stages of our method takes on average 1 hour (see the the computational specifications in Appendix B), which is more efficient in comparison to the baselines, where CPAE for example takes around 24 hours to be trained for one category.
5.3 Few-Shot 3D KeyPoint Detection
We evaluated our few-shot keypoint detection algorithm FSKD on the KeyPointNet dataset, following the setup of [59]
[Table]
and using only 3 labeled source shapes (the resulting algorithm is thus a threeshot keypoint detection algorithm).
The goal of this task is to predict keypoints that correlate with human annotation.
The performance is evaluated by the mean Intersection over Union (mIoU).
The intersection is counted if the geodesic distance between a predicted keypoint and its nearest ground truth keypoint is less than some geodesic threshold.
The union is simply the union of the detected keypoints and ground truth keypoints.
We compare FSKD against multiple baselines, including SIFT-3D [84], HARRIS-3D [85], ISS [86],
D3Feat [87], USIP [88] and UKPGAN [59].
Results are summarized in Fig.
5 - left.
It can be seen that our simple, correspondence-based method outperforms the baselines tailored specifically for this task.
In particular, our method achieves state-of-the-art results for a threshold superior to 0.035, using only three labeled shapes.
We provide some qualitative results in Fig.
5 - right.
Conclusion & Limitations
In this work, we showed that using noisy maps as a supervisory neural network training signal for the 3D shape-matching task, can lead to significantly higher quality correspondences.
We named this effect Neural Correspondence Prior (NCP).
Through extensive experiments, we shed light on the properties of NCP and developed a two-stage algorithm for unsupervised 3D shape matching.
We demonstrate the effectiveness and generality of our algorithm on a range of shape-matching problems, including unsupervised shape matching on both man-made and organic non-rigid shapes, achieving state-of-the-art results on a wide range of benchmarks.
One limitation of our approach is that it assumes that the p2p maps used to train the second stage of the algorithm still have some structure.
In fact, if the unsupervised method used in the first stage fails completely and provides random maps, our method will not converge.
This problem can be mitigated by using a good application-specific unsupervised method for the first stage, especially with the recent interest and increase in unsupervised 3D learning methods.
Acknowledgements We gratefully acknowledge the anonymous reviewers for their valuable suggestions.
Parts of this work were supported by the ERC Starting Grant No. 758800 (EXPROTEA) and the ANR
A Background On Functional Maps & Notation
Our work uses the functional map framework as a first estimator for p2p maps, and multiple losses on point-to-point (p2p) maps to learn robust features that allow extracting good correspondences using the nearest neighbor in feature space.
We provide a brief overview in the next section.
Functional Maps The functional map (fmap) framework was used for the first stage of our
NCP-UN algorithm.
For this, we follow the general strategy of recent fmap-based techniques [11, 12, 39, 15, 19], as follows: given source and target shapes M and N , represented as either triangular meshes or point clouds, having m and n vertices respectively, we pre-compute their
Laplace-Beltrami operator [89], and store their first k eigenfunctions in the matrices
Φ M ∈ R m × k and Φ N ∈ R n × k respectively.
Using a Siamese network F θ , we compute for each shape a ddimensional descriptor F θ ( M ) =
F ∈ R m × d and F θ ( N ) =
G ∈ R n × d respectively.
These descriptors are then projected to the spectral domain to form the spectral features A =
Φ †M F and
B = Φ †N G ( • † is the Moore pseudo-inverse).
A functional map is then computed by solving the following linear system:
[Equation]
where ∆ M , ∆ N are diagonal matrices of Laplace-Beltrami eigenvalues of the corresponding shapes and λ is a scalar hyper-parameter.
Following the unsupervised literature [15, 39, 19], the Siamese network F θ is trained by imposing structural properties on the fmap C such as bijectivity and orthogonality on the shape pairs in the training set.
In fact, given the fmap C MN from M to N , and C NM from N to M , the bijectivity loss is formulated as ∥ C MN C NM − I k ∥ 22 , and the orthogonality loss is ∥ C ⊤MN C MN − I k ∥ 22 .
I k denotes the identity matrix of size k and • ⊤ is the matrix transpose operator.
Feature Learning The goal of feature learning is to learn robust descriptors that can allow direct nearest-neighbor matching in the descriptor space.
In this work, we use two losses: PointInfoNCE and the LIE loss.
PointInfoNCE [72] is a contrastive loss such that, given a set of matched points P , and two features of dimension s , it is formulated as follows:
[Equation]
where τ is a temperature parameter, and d ( · , · ) is the Euclidean distance between the two features.
In all our experiments, we took τ = 0 .
The purpose of this loss is to force the distance between the features of the matched points to be minimized, while this distance must be maximized between the unmatched points.
The NCE loss is applied to each point individually and thus cannot penalize the overall consistency of the matches.
To remedy this, especially when the number of vertices of the shapes is moderate, as is the case for sparse point clouds, we use the LIE loss introduced in [75].
Given the extracted features F and G , and the coordinate xyz of the shape N represented by the matrix N ∈ R n × 3 , we first compute the soft correspondences matrix S MN , and then formulate the LIE loss as follows:
[Equation]
where Π gt MN is the ground truth p2p correspondence matrix.
This loss forces the soft correspondences matrix to be as close as possible to the ground truth map, by forcing their action (pull-back) on the shape coordinates, thus taking into account the geometry of the shape.
Indeed, erroneous predictions that are geometrically close to the ground truth are penalized less than those that are far from it in terms of L2 distance.
B Implementation details
In Section 3.1 in the main text, we used a randomly initialized DiffusionNet [19] network to predict features on the test set of FAUST-Remeshed and SCAPE-Remeshed datasets [33].
For that, we used the publicly available implementation of DiffusionNet released by the authors 1 .
Unless specified otherwise, all our experiments on 3d-triangular meshes use 4 DiffusionNet blocks of width 128, the input to the network is the XYZ coordinates of the shape.
For the competing features, both the Heat
Kernel Signature (HKS) [69] and the Wave Kernel Signature (WKS) [70] were sampled at 100 values of energy t, logarithmically spaced in the range proposed in their respective original papers.
SHOT descriptors are 352-dimensional, and we used the implementation provided by the PCL library [90].
For the Laplace-Beltrami computation, we used the discretization introduced in [89] for both 3D meshes and point clouds.
In Section 3.2 of the main text, we train a DiffusionNet to produce feature embeddings that will induce the maps used for supervision, using the NCE loss.
For this experiment and all the following learning experiments, ADAM optimizer [91] was used with a learning rate of 0.001.
In Section 5.1.1 and 5.2 of the main text, we applied our NCP-UN algorithm on the KeyPointNet [57] and the PartNet [76] datasets.
For the first stage, we used the unsupervised geometric functional map from [19] but using PointMLP [74] as a feature extractor, instead of DiffusionNet, as it is better suited to the point cloud context.
We used the default segmentation configuration provided by the authors 2 .
In Eq.
(1), we take λ = 0 .
The network is trained using only the bijectivity loss presented in Appendix A. For the second stage, we used the LIE loss, and we trained a randomly initialized
PointMLP, which has the same architecture as the first stage, to produce feature embeddings that induce the input maps.
In Section 5.1.2 of the main text, NCP-UN was applied on the SMAL [77, 36] and SHREC’20 [78] datasets.
The first stage was performed using the unsupervised geometric functional map [19] with the DiffusionNet backbone, while applying the bijectivity and orthogonality losses, as described above in Appendix A.
The second stage was performed using the NCE loss and the same backbone as the first stage.
P2P maps were extracted using either the nearest neighbor in the space of features or using the functional map pipeline.
In fact, given two feature embedding for two shapes M and N , we compute the functional map C MN from M to N using Eq.
(1), and then convert it to a p2p map
T fmap : N → M using (borrowing the same notation from Appendix A):
[Equation]
For the test time optimization experiment in Section 5.1.2 of the main text, after extracting the maps predicted by the first stage, we construct pairs between all the shapes of the test set, and for each pair, we train a randomly-initialized DiffusionNet network using the NCE loss and Adam optimizer, to produce feature embedding that induces the map from stage 1.
We stop the training when the cyclic loss (see Section 4.2 of the main text) stops improving giving a patience period of 100 optimization iterations.
In all the experiments of Section 5 of the main text, except for test time optimization, data augmentation was used.
In particular, we augment the training data on the fly by randomly rotating the input shapes, applying random scaling in the range [0.9, 1.1], and jittering the position of each point by
Gaussian noise with zero mean and 0.01 standard deviation.
Computational specifications
All our experiments are executed using Pytorch [92], on a 64-bit machine, equipped with an Intel(R) Xeon(R) CPU E5-2630 v4 @
2.20GHz and a RTX
Ti
Graphics Card.
For all competing methods, we use the original code released by the authors and apply the best parameters reported in the respective papers.
As mentioned in the main paper, we will release our complete implementation to ensure the full reproducibility of all of our results.
C Implementation Details on FSKD
Below we provide the implementation details on our Few-Shot Keypoint Detection method (FSKD), described in Section 4.3 of the main manuscript.
As mentioned in that section, FSKD is composed of three steps: 1.
Detection of potential keypoints by transferring keypoints from labeled shapes, 2.
Filtering to remove keypoints that are likely not to exist on the target shape, 3.
Combination: merge transferred keypoints if multiple points on the target shape are assigned to the same keypoint ID.
Step 1. is done using our established maps predicted by NCP-UN .
For step 2. , we compute the cycle consistency loss of transferred keypoints and only keep the ones that are below a predetermined threshold.
I.e., given a pair of shapes M , N , and two maps between them Π MN and Π NM computed via nearest neighbor matching between their feature embeddings, and represented as binary matrices, the cycle consistency loss of keypoints i is computed as l i = ∥
( X N ) i − (Π NM Π MN X N ) i ∥ 2 F , where X N is the matrix of XYZ coordinates of the source shape N .
If l i is bigger than a predefined threshold ν , the keypoint i is considered not to exist on the target shape.
In our experiments, we take ν = 0 .
Concerning step 3. , we perform a spatially weighted average of the different filtered keypoints if there are many.
We associate for each keypoint i the weight w i =
D i (cid:80) j D j , where D i = exp( − l i σ ) .
We use σ = 0 .
D Proof of smoothness of maps produced by smooth networks
In Section 3.2 of the main text, we briefly mentioned a theorem that states that maps based on the nearest neighbor between smooth features are smooth.
In this section, we formally restate it and provide proof.
Theorem 1.
Let M and N be two compact smooth surfaces (smooth manifolds of dimension 2).
Let M and N be their embeddings in R d , given by some functions: ψ : M → R d and ϕ : N → R d , so that M = ψ ( M ) and N = ϕ ( N ) .
Suppose that ψ and ϕ that are both smooth and injective.
Then up to arbitrarily small perturbations of ϕ, ψ , the map T nn : M → N given by
T nn ( x ) =
arg min y ∈N
∥ ψ ( x ) − ϕ ( y ) ∥ must be smooth up to sets of measure 0 on M .
Proof.
First, note that the distance function to an embedded manifold is smooth almost everywhere.
Indeed, it is well-known that if N is a C k -continuous manifold embedded in R d , then the distance function to N must be at least C k continuous on the complement of the medial axis of N .
I.e., let d N : R d → R d be given by d N ( x ) =
arg min y ∈ N ∥ x − y ∥ .
Let cut( N ) denote the medial axis of N , which is defined as the set of points in R d with more than one nearest neighbor to N .
I.e., cut( N ) = { x ∈ R d | ∃ y 1 , y 2 ∈ N , y 1 ̸
=
y 2 , s . t .
∥ x − y 1 ∥ =
∥ x − y 2 ∥ = min y ∈ N ∥ x − y ∥} .
Then, d N is at least C k continuous on R d \ cut( N ) (see Lemma 2.5 in [93], and [94, 95] for this and related results).
It remains to prove that up to arbitrarily small perturbations of ϕ, and ψ the intersection between
M and cut( N ) has measure zero on M .
For this, we first use the fact that the medial axis of compact subanalytic manifolds is also subanalytic [96].
This means that the medial axis can be stratified (decomposed into a finite union of submanifolds of dimension d − 1 in R d ).
Since (by
Stone-Weierstrass’s theorem) subanalytic manifolds are dense in the space of smooth manifolds, up to an arbitrarily small perturbation of ϕ , N is subanalytic.
Finally, the intersection between M (which is an embedded manifold of dimension 2) and any manifold of dimension d − 1 by Thom’s transversality theorem [97, 98], must generically be measure zero on M and thus on M .
E Verification of assumptions of Theorem
The main motivation behind Theorem 1 is to highlight the fact that, given smooth feature embeddings, if we add the injectivity condition, the maps extracted using the nearest neighbors in the feature space tend to be smooth.
As smoothness is a generally desirable property, and the frequency bias shown in prior works, such as [23] suggests that neural networks are biased towards low frequency (and thus smooth) functions, we provide this result as a partial explanation for the Neural Correspondence
Prior, which we have observed, for the first time, in our work.
Regarding injectivity, although such a property is not trivial, and might need to be specifically enforced (using, for example, invertible networks [99]).
However, we observe that when training a network using contrastive learning, such as NCE loss, the latter forces the networks to produce embeddings that are unique for each point, in order to minimize the NCE loss, which can be considered a form of injectivity.
We include in this section the results of an experiment checking the smoothness and injectivity of a randomly initialized network.
For this, we performed the following experiment.
We first computed the embeddings produced by a randomly initialized DiffusionNet network for the 20 test shapes in the FAUST-Remeshed [33] dataset.
To evaluate the smoothness of the embedding, we computed the standard Dirichlet energy of both the embedding produced by the network and the original embedding of the shape in (the 3D coordinates of the shape’s vertices), using the following formula:
[Equation]
where G is the considered embedding of dimension n , while W and A are, respectively, the standard stiffness and mass matrices, computed using the classical cotangent discretization scheme of the
Laplace-Beltrami operator [100].
We computed the Dirichlet energy over the whole test set and found
[Figure]
it equal to 47.2 (the average) for coordinate functions (original shape embeddings in ), while it is equal to 13.1 for embeddings produced by the network.
This shows that the latter is smoother than embeddings in the original space.
For injectivity, we computed for each point of the embedding, the distance to its nearest neighbor, and took the minimum across all points of a shape.
To make the comparison between the original embedding and the embedding produced by the network fair, we normalized both embeddings to the unit sphere.
We find that on average, the minimum distance between points in the original 3D space is 0.0004 , while for the feature embedding, given by the network it is equal to 0.0015 .
This shows that the network is injective and that distances between points are larger than in the original domain.
F Neural Correspondence Prior
In section 3 of the main text, we demonstrated the effect of Neural Correspondence Prior (NCP), on a pair of shapes, using the DiffusionNet network.
The objective of this section is to show that the NCP is independent of the choice of the shape pair, the choice of the p2p loss, the choice of the architecture, and finally the choice of the first stage.
F.1 Independence from shape pairs
In order to show that the NCP demonstrated in Figure 1 of the main text does not depend on the chosen shape, we redid the same experiment using new random pairs from FAUST-Remeshed and SCAPERemeshed datasets [33].
Following the same setup of Section 3.2 of the main text, given a pair of shapes, we corrupt the ground truth map between them with 50% noise, and then train a randomly initialized DiffusionNet to produce feature embeddings that overfit the noisy map.
Examples of four different pairs are shown in Fig.
We see that the same effect is always present, i.e., the network resists overfitting the noisy maps (high noise impedance), and the intermediate maps during optimization are of high quality, compared to the noisy maps used as supervision.
F.2 Independence from the Loss Function and Network Architecture
In order to check the dependence of NCP on both the network architecture and on the p2p loss used in stage 2 of NCP-UN , the following experiment was performed on the Chair subset of the
KeyPointNet [57] dataset.
Since the latter doesn’t provide dense ground truth p2p maps between the network
It appears that multiple network architectures losses improve the maps by
, proving that NCP is not tied to a single method choice
[Table] [Table]
shapes, we took the p2p maps produced by the first stage of our algorithm (see Section 5.1.1 of the main text), and perturbed them with noise.
These noisy maps are used as inputs to the second stage of the NCP-UN algorithm.
Differently from the experiment performed in Section 5.1.1 of the main text, here, we choose different network architectures and losses for training the second stage.
In particular, we consider the NCE loss, the LIE loss, and the supervised FMAP loss from [11].
Concerning the network architectures, we considered the ResidualMLP network introduced in [13], PointNet++ [21], and DGCNN [20].
For the network architectures, we used the official implementation provided by the authors.
In addition to measuring the geodesic error produced by the maps predicted by the second stage, we also measure their smoothness.
Given two shapes M , N , and a map between them T MN represented as a binary matrix Π MN , we compute the smoothness of the latter based on the Dirichlet energy using the following (see [100] for more details):
[Equation]
where X N is the matrix of XYZ coordinates of shape N , E M is the set of all the edges of the triangular mesh, and w uv are the stiffness weights of the cotangent Laplacian [100] for shape M .
Results of this experiment are summarised in Tab. 4.
It can be seen that all the losses and network architectures perform well and manage to improve the noisy input maps.
One can also notice the smoothing effect of NCP.
In fact, the second stage of the network produces feature embeddings that result in p2p maps that are not only geometrically correct but also remove noise and outliers from the input maps.
It is worth noting that although the ResidualMLP architecture is weak, since it is applied on each vertex individually, and has no global shape awareness, we note that even if the geodesic error is bad, the network tends to produce smooth embeddings.
F.3 Independence from the first stage
As we stated in the main paper, our NCP-UN algorithm is independent from the first stage, i.e the method used to extract the artifact-laden p2p is not crucial, and the NCP effect still applies.
To demonstrate this, we followed the same setup of section 5.1.2 of the main text, and we perform the same experiment on the SMAL dataset but using different methods for stage 1.
In particular, we consider as stage 1 the Unsupervised geometric functional map (Unsup GeomFMaps)
[19], Smooth
Shells [18], Deep Shells [17] and NeuroMorph [83].
Results are summarized in Tab. 5.
As can be seen, despite the method used in the first step, the NCP still applies, and a significant improvement in results is observed, up to 68% improvement.
It can be seen that the second stage of our algorithm always improves upon the first stage due to the NCP effect.
G Shape Matching On Man-Made Data
In Section 5.1.1 of the main text, we applied our NCP-UN algorithm on the KeyPointNet dataset, and only provided quantitative results for 8 classes due to the page limit.
We provide in Fig.
7 quantitative results for all the 16 classes, in addition to the results of the first stage of our algorithm, i.e the unsupervised method described above in Appendix B.
For the bottle, cap, knife, and skate categories, we only compared them to the best-performing baseline.
It can be seen that our method achieve state-of-the-art results in 13 out of 16 classes on this benchmark, with an impressive improvement in some classes such as skate and table.
We also see that the second stage of NCP-UN always improves the result provided by the first stage, which again demonstrates the role of the NCP effect, without which the SOTA result would not be achieved, see for example the chair category.
Additionally, we provide in Fig.
8 some qualitative results, showing the p2p maps produced by both stages of NCP-UN , visualized using texture transfer, as well as the usage of these maps to transfer keypoints between shapes.
It can be seen that the keypoint transferred by the maps of stage 2 are more accurate.
H Shape Matching On Non-Rigid Near-Isometric Data
In our work, we focus on difficult non-rigid non-isometric datasets, where existing methods tend to fail.
This is because on near-isometric datasets such as FAUST Remeshed or SCAPE Remeshed [33], current unsupervised methods can exploit the assumption of near-isometry and achieve good results.
Nevertheless, we include a comparison of our method on the FAUST-Remeshed and SCAPERemeshed datasets, using the same train/test split used in all previous works (e.g. [11]).
The notation X on Y means the method is trained on X and tested on Y.
[Figure]
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
As input for our Stage 2, we use WSupFMNet + DiffusionNet [19] (referred to as stage 1 in the table), which is the same method used in the main manuscript for all our experiments.
For our method (Stage 2), we use the same implementation as the one used for the SMAL dataset, described in Section 5.1.
of the main manuscript.
As shown in Tab. 6, even on a near-isometric dataset, our Stage 2 improves upon the initial maps in all categories, by 18.4% on average.
Nevertheless, we remark that in such settings it is more advantageous to use specialized methods, such as ZoomOut [101], that directly exploit the nearisometry assumption.
I Societal Impact
Efficient methods for shape matching and analysis have immediate impact in many areas of science and engineering from medical imaging (for instance for detecting anomalies, and performing follow-up analysis) to shape recognition and classification in areas such as computational biology, archaeology and paleontology to name a few.
Our approach can immediately be adapted in such diverse scenarios, due to its strong generalization power, and to its generic unsupervised nature, especially in domains where acquiring data is easy, but labeling it is very expensive, e.g in structural or molecular biology.
Our work also opens up important avenues for future research, as it can enable geometric deep learning methods without the need to label large-scale datasets, thus potentially allowing small labs to conduct research in this area without the need to hire many annotators, which is an expensive task.
Finally, our method paves the way for more accurate results, helping to improve our understanding in many fields, such as biology where shape matching techniques are used to analyze gene expression patterns to understand the cause of many human syndromes [102].
Since our method attempts to solve a fundamental problem in computer graphics and computer vision, we do not expect negative results.
However, one should note that highly accurate shape correspondence methods might have possibly problematic uses, e.g., in surveillance applications, although we advocate against such uses of our technique.