Paper: GenCorres: Consistent Shape Matching via Coupled Implicit-Explicit Shape Generative Models (ICLR-2024)

Reviewer comment (Writing):
The central part of the method is not easy to grasp. I find the overall principle clear, and it is probably possible to replicate the work in principle. Still, the technique is not well explained in detail, and the notation is often confusing. For example, the letter "g" is used both for the shape generator and for the mesh generator, and requires some back-and-forth to get used to it. I suggest revisiting the method explanation and clarifying the methodological details.

Author response:
> **Q: Confusing notation.**

In the original submission, the implicit generator uses lowercase letters $g^{\phi}$. The explicit generator uses bold letters $\mathbf{g}^{\theta}$. In the revised manuscript, we have made changes to the notations to make them clearer:

- $f^{\phi}$ denotes the implicit generator, where $\phi$ are the network parameters.

- $f^{\phi}(\mathbf{x}, \mathbf{z})=0$ denotes the implicit surface (i.e. the iso-surface where the signed distance value is 0). 

- $\mathbf{g}^{\phi}(\mathbf{z})$ denotes the discretized mesh of the implicit surface $f^{\phi}(\mathbf{x}, \mathbf{z})=0$. 

- $h^{\psi}$ denotes the encoder in the VAE.

- $\mathbf{m}^{\theta}$ denotes the mesh generator in the third stage.

We have also provided the explanations in the revised main paper.

Reviewer comment (Experiments):
By my understanding, the method highly relies on the target shapes belonging to a given distribution that should not only be bounded in terms of structure and class but, in particular, should be possible to express by the registered template. Many methods rely on template registration, and it is not a weakness per se, but the proposed approach makes use of techniques that aim to be general and flexible (e.g., implicit representation, unsupervised learning), yet the paper does not show any out-of-distribution results (and also, results only on shapes for which data are available and can be even generated synthetically). I also believe that the topological constraint given by the template limits the generality of the method. The limitations barely mention this, and it should be emphasized more.

Author response:
> **Q: The topological constraint given by the template limits the generality of the method.**

The template is introduced for the explicit mesh generator. In fact, any mesh generator in the literature requires a template. 
The template is not used in the implicit generator, which is a key contribution of this paper. The implicit generator can already provide pair-wise correspondences between pairs of implicit surfaces, which are not constrained by topology. Specifically, given any two shapes, we can perform latent space interpolation between them and propagate the correspondences along the interpolation path using equation (7) in the main paper. This framework is applicable to any shape collection, regardless of its topology. These correspondences can be evaluated through the interpolation results shown in the paper; smoother and more rigid interpolations indicate better correspondences. In this paper, we utilize the template solely for the task of joint shape matching of deformable shapes, as they typically share similar topology and structure.

Reviewer comment (Experiments):
Experiments are performed on a relatively small set of data. The datasets are outdated and do not deal with the literature's more recent and real challenges (e.g., partiality, noise, clutter, ...). In this sense, I suggest stressing the method further; I believe that it would be interesting to investigate the latent interpolation when the learned space represents more diverse shapes (i.e., I wonder if relying on implicit representation may help in the context of limited topological variation during the interpolation, or if relying on a template does not let the method generalize to these cases).

Author response:
> **Q: Data is generated synthetically and does not deal with the real challenges (e.g., partiality, noise, clutter, …)**

We have included additional experiments in the supplementary material, submitted during the rebuttal period, to demonstrate that our method is also applicable to real scans. We trained the implicit network on a real human dataset featuring various clothes and hairstyles. The experiments indicate that our method can still provide smooth interpolation even when the learned space represents a more diverse range of shapes. Please refer to the newly submitted supplementary material for more details. Additionally, we want to emphasize that the Human dataset we evaluated, which comes from SMPL, is sufficiently challenge for inter-shape matching.

Reviewer comment (Theory):
How would linear interpolation behave in the presence of significant diverse geometrical topology data? Is the template triangulation the main limitation in this case?

Author response:
> **Q: How would linear interpolation behave in the presence of significant diverse geometrical topology data? Is template triangulation the main limitation in this case?**

In this paper, we focus on deformable shapes (humans and animals), which usually do not have significant topological changes. For other types of shape collections with significantly diverse geometrical topologies, we leave this for future research. Note that we perform linear interpolation for the implicit surfaces, where template triangulation is not involved.

Reviewer comment (Evaluation):
Could you provide some measure of the computational cost of the method? I think it would be important to understand the scalability of the method, and I assume that the method would require a significant computational effort. How does it compare with NeuroMorph?

Author response:
> **Q: Computational cost of the method compared with NeuroMorph.**

Similar to SALD, we train our model with 8 NVIDIA TITAN V GPUs (12G). For the Human collections (about 1000 shapes), training the implicit generator takes about 3 days, while the explicit generator requires approximately 12 hours. For the FAUST dataset (80 shapes), NeuroMorph is trained with 1 GPU for roughly 3 days. Note that NeuroMorph requires a pair of shapes as input, so the training complexity is $O(n^2)$ for $n$ training shapes. In contrast, our method directly learns the shape space, meaning the complexity scales linearly with the number of training shapes.

Reviewer comment (Experiments):
Would it be possible to test the method on a class for which a dense correspondence is not provided; for example, some classes of ShapeNet (e.g., cars, airplanes)?

Author response:
> **Q: Would it be possible to test the method on a class for which a dense correspondence is not provided; for example, some classes of ShapeNet (e.g., cars, airplanes)?**

We propose a general framework for computing correspondences between implicit surfaces. We are encouraged by your recognition of our framework's potential applicability to a wide range of shape collections, including man-made shapes. For man-made shapes, we can adopt alternative energies, such as as-affine-as-possible (AAAP) energy, which enforces local region transformations to be affine rather than rigid, in place of ACAP energy. However, we want to underscore that deformations among man-made shapes are more complex than deformable shape collections. Also there are structural variations, where AAAP should be enforced among shapes or parts that are structurally similar. Those issues should be addressed by other papers (we are currently looking into this).

Reviewer comment (Experiments):
The human shape collection seems to include a multitude of body types and is clearly able to correspond/interpolate between them. The animal collections seem to only have shapes of the same animal. The paper would be stronger if it showed an interpolation between, for example, a lion and a horse. It is not clear whether this method would blend well between very different quadruped shapes.

Author response:
> **Q: Interpolation between a lion and a horse.**  

We have included more qualitative results in the supplementary material submitted during the rebuttal period. We trained the implicit network on a new quadruped shape dataset. Visualization of the interpolation results show that our method transitions smoothly between a lion and a horse. Please refer to the newly submitted supplementary material for more details.

Reviewer comment (Novelty):
A more suitable ARAP reference should be: As-rigid-as-possible shape interpolation by Alexa et al. 2000.

Author response:
> **Q: Citation, acronyms and typos.**  

We have added the new ARAP reference and corrected the acronyms and typos in the title.

Reviewer comment (Writing):
The different fonts for g should be explained (it looks like implicit expressions use lowercase letters, while explicit use bold letters).

Author response:
> **Q: Math notations and what are the different fonts for g?**  

In the original submission, the implicit generator uses lowercase letters because the signed distance is a scalar. The explicit generator uses bold letters because the output vertices are vectors. They can also be distinguished by the network parameters: $\phi$ for the implicit generator and $\theta$ for the explicit generator. We used g because both of them are generative models. In the revised manuscript, we have made changes to the notations to make them clearer:  

- $f^{\phi}$ denotes the implicit generator, where $\phi$ are the network parameters. Yes, the reason for $\mathbb{R}^3 \times \mathcal{Z}$ is that it computes the signed distance value at the specific 3D coordinate for the shape defined by a latent code.  

- $f^{\phi}(\mathbf{x}, \mathbf{z})=0$ denotes the implicit surface (i.e. the iso-surface where the signed distance value is 0). $\mathbf{x}$ is the 3D query location and $\mathbf{z}$ is the latent code.  

- $\mathbf{g}^{\phi}(\mathbf{z})$ denotes the discretized mesh of the implicit surface $f^{\phi}(\mathbf{x}, \mathbf{z})=0$.  

- $h^{\psi}$ denotes the encoder in the VAE, where $\psi$ are the network parameters.  

- $\mathbf{m}^{\theta}$ denotes the mesh generator in the third stage, where $\theta$ are the network parameters.  

We have also provided these explanations in the revised main paper.

Reviewer comment (Writing):
The meaning of Theta in g^Theta in the "Problem statement" paragraph (page 3) should be clarified.

Author response:
> **Q: Math notations and what are the different fonts for g?**  

In the original submission, the implicit generator uses lowercase letters because the signed distance is a scalar. The explicit generator uses bold letters because the output vertices are vectors. They can also be distinguished by the network parameters: $\phi$ for the implicit generator and $\theta$ for the explicit generator. We used g because both of them are generative models. In the revised manuscript, we have made changes to the notations to make them clearer:  

- $f^{\phi}$ denotes the implicit generator, where $\phi$ are the network parameters. Yes, the reason for $\mathbb{R}^3 \times \mathcal{Z}$ is that it computes the signed distance value at the specific 3D coordinate for the shape defined by a latent code.  

- $f^{\phi}(\mathbf{x}, \mathbf{z})=0$ denotes the implicit surface (i.e. the iso-surface where the signed distance value is 0). $\mathbf{x}$ is the 3D query location and $\mathbf{z}$ is the latent code.  

- $\mathbf{g}^{\phi}(\mathbf{z})$ denotes the discretized mesh of the implicit surface $f^{\phi}(\mathbf{x}, \mathbf{z})=0$.  

- $h^{\psi}$ denotes the encoder in the VAE, where $\psi$ are the network parameters.  

- $\mathbf{m}^{\theta}$ denotes the mesh generator in the third stage, where $\theta$ are the network parameters.  

We have also provided these explanations in the revised main paper.

Reviewer comment (Writing):
The meaning of Phi in "Approach overview" (page 3) should be explained, including what it maps from and to, and why it is R^3 x Z. It should also be clarified if it computes the distance field value at the specific 3D coordinate for the shape defined by a latent code.

Author response:
> **Q: Math notations and what are the different fonts for g?**  

In the original submission, the implicit generator uses lowercase letters because the signed distance is a scalar. The explicit generator uses bold letters because the output vertices are vectors. They can also be distinguished by the network parameters: $\phi$ for the implicit generator and $\theta$ for the explicit generator. We used g because both of them are generative models. In the revised manuscript, we have made changes to the notations to make them clearer:  

- $f^{\phi}$ denotes the implicit generator, where $\phi$ are the network parameters. Yes, the reason for $\mathbb{R}^3 \times \mathcal{Z}$ is that it computes the signed distance value at the specific 3D coordinate for the shape defined by a latent code.  

- $f^{\phi}(\mathbf{x}, \mathbf{z})=0$ denotes the implicit surface (i.e. the iso-surface where the signed distance value is 0). $\mathbf{x}$ is the 3D query location and $\mathbf{z}$ is the latent code.  

- $\mathbf{g}^{\phi}(\mathbf{z})$ denotes the discretized mesh of the implicit surface $f^{\phi}(\mathbf{x}, \mathbf{z})=0$.  

- $h^{\psi}$ denotes the encoder in the VAE, where $\psi$ are the network parameters.  

- $\mathbf{m}^{\theta}$ denotes the mesh generator in the third stage, where $\theta$ are the network parameters.  

We have also provided these explanations in the revised main paper.

Reviewer comment (Writing):
The meaning of Psi in h^Psi in equation (1) should be clarified. Also, lambdas for equations (1) and (2) should be mentioned.

Author response:
> **Q: Math notations and what are the different fonts for g?**  

In the original submission, the implicit generator uses lowercase letters because the signed distance is a scalar. The explicit generator uses bold letters because the output vertices are vectors. They can also be distinguished by the network parameters: $\phi$ for the implicit generator and $\theta$ for the explicit generator. We used g because both of them are generative models. In the revised manuscript, we have made changes to the notations to make them clearer:  

- $f^{\phi}$ denotes the implicit generator, where $\phi$ are the network parameters. Yes, the reason for $\mathbb{R}^3 \times \mathcal{Z}$ is that it computes the signed distance value at the specific 3D coordinate for the shape defined by a latent code.  

- $f^{\phi}(\mathbf{x}, \mathbf{z})=0$ denotes the implicit surface (i.e. the iso-surface where the signed distance value is 0). $\mathbf{x}$ is the 3D query location and $\mathbf{z}$ is the latent code.  

- $\mathbf{g}^{\phi}(\mathbf{z})$ denotes the discretized mesh of the implicit surface $f^{\phi}(\mathbf{x}, \mathbf{z})=0$.  

- $h^{\psi}$ denotes the encoder in the VAE, where $\psi$ are the network parameters.  

- $\mathbf{m}^{\theta}$ denotes the mesh generator in the third stage, where $\theta$ are the network parameters.  

We have also provided these explanations in the revised main paper.

Reviewer comment (Writing):
Symbol x is introduced in section 4.1 without describing what it is.

Author response:
> **Q: Math notations and what are the different fonts for g?**  

In the original submission, the implicit generator uses lowercase letters because the signed distance is a scalar. The explicit generator uses bold letters because the output vertices are vectors. They can also be distinguished by the network parameters: $\phi$ for the implicit generator and $\theta$ for the explicit generator. We used g because both of them are generative models. In the revised manuscript, we have made changes to the notations to make them clearer:  

- $f^{\phi}$ denotes the implicit generator, where $\phi$ are the network parameters. Yes, the reason for $\mathbb{R}^3 \times \mathcal{Z}$ is that it computes the signed distance value at the specific 3D coordinate for the shape defined by a latent code.  

- $f^{\phi}(\mathbf{x}, \mathbf{z})=0$ denotes the implicit surface (i.e. the iso-surface where the signed distance value is 0). $\mathbf{x}$ is the 3D query location and $\mathbf{z}$ is the latent code.  

- $\mathbf{g}^{\phi}(\mathbf{z})$ denotes the discretized mesh of the implicit surface $f^{\phi}(\mathbf{x}, \mathbf{z})=0$.  

- $h^{\psi}$ denotes the encoder in the VAE, where $\psi$ are the network parameters.  

- $\mathbf{m}^{\theta}$ denotes the mesh generator in the third stage, where $\theta$ are the network parameters.  

We have also provided these explanations in the revised main paper.

Reviewer comment (Writing):
The meaning of d in equation 3 should be clarified, including whether it is some displacement between a vertex and an infinitesimally close corresponding vertex, whether d is of unit length and then scaled down by epsilon, or whether d should be the small displacement that doesn't need to be further multiplied by epsilon.

Author response:
> **Q: What does d stand for in equation 3, some displacement between a vertex and an infinitesimally close corresponding vertex?**  

Yes, $\mathbf{d}_i^{\mathbf{v}}(\mathbf{z}) $  in equation (3) denotes the displacement between a vertex and an infinitesimally close corresponding vertex. In the original submission, we further multiplied it by $\epsilon$ because this simplified equation (6), (7) and (8) by avoiding the constant $\epsilon$. However, you are correct that this is not necessary. In the revised submission, we have followed your suggestions and did not scale it by $\epsilon$. The final expressions of the regularization terms remain the same.

Reviewer comment (Writing):
The meanings of several capital letter symbols (C, F, G, E) introduced at some point in the paper are hard to follow.

Author response:
> **Q: Explanation of capital letter symbols (C, F, G, E).**  

We provide detailed expressions of $C$ and $F$ in the appendices of the main paper. $C$ and $F$ are derived from the matrix representation of Equation (3).  

The expression of $G$ is given in equation (7). It is just a component of $\mathbf{d}_i^{\mathbf{v}}(\mathbf{z})$ that is independent of the perturbation direction $\mathbf{v}$.  

$E$ is derived in equation (8) and serves as a metric to measure the local rigidity and local conformality distortions between an implicit surface and its perturbation. The expression of $E$ can also be found in equation (8) and, like $G$, is independent of the perturbation direction $\mathbf{v}$.

Reviewer comment (Writing):
Acronyms should be spelled out at their first usage.

Author response:
> **Q: Citation, acronyms and typos.**  

We have added the new ARAP reference and corrected the acronyms and typos in the title.

Reviewer comment (Presentation):
The first mention of ACAP is not accompanied by a description and reference; it may be better to remove it from section 3 until it is properly introduced in section 4.

Author response:
> **Q: Citation, acronyms and typos.**  

We have added the new ARAP reference and corrected the acronyms and typos in the title.

Reviewer comment (Writing):
MP-pseudo inverse should be referred to as Moore-Penrose inverse.

Author response:
> **Q: Citation, acronyms and typos.**  

We have added the new ARAP reference and corrected the acronyms and typos in the title.

Reviewer comment (Experiments):
The authors claim that the implicit approach is necessary since learning a mesh generator from scratch is too difficult and error-prone. While this may be true, it is not actually demonstrated within the ambit of the proposed pipeline. There is the 3D-CODED comparison, but that is an entirely different pipeline.

Author response:
We conducted ablation studies for the joint shape matching experiments on DFAUST. The results are shown in the table below. GenCorres-NoImp does not utilize the implicit approach (1st stage) and directly registers the template to each target shape as initialization. GenCorres-NoInit omits both the 1st and 2nd stages, using only the 3rd stage. The results show that GenCorres is superior to both GenCorres-NoImp and GenCorres-NoInit, thanks to better initialization. Without latent space interpolation to generate intermediate shapes, GenCorres-NoImp often fails to register the template directly to a vastly different target shape, leading to poor initialization. GenCorres-NoInit, which does not employ any correspondence initialization, tends to get stuck in undesired local minima. Note that the success of training 3D-CODED relies on a very large dataset (23000 human meshes with a large variety of realistic poses and body shapes). 

Evaluations of JSM on DFAUST using geodesic errors of the predicted correspondences (in cm).

|                  | mean  | median |
|:---------------- |:-----:|:------:|
| GenCorres        | 1.30  | 1.13   |
| GenCorres-NoImp  | 7.92  | 7.73   |
| GenCorres-NoInit | 12.11 | 11.84  |

Reviewer comment (Experiments):
Testing the necessity of the implicit approach would constitute an "ablation" that is at least half a research project by itself, but still, the reviewer wonders if the authors have already done experiments to verify this which are not included in the paper.

Author response:
We conducted ablation studies for the joint shape matching experiments on DFAUST. The results are shown in the table below. GenCorres-NoImp does not utilize the implicit approach (1st stage) and directly registers the template to each target shape as initialization. GenCorres-NoInit omits both the 1st and 2nd stages, using only the 3rd stage. The results show that GenCorres is superior to both GenCorres-NoImp and GenCorres-NoInit, thanks to better initialization. Without latent space interpolation to generate intermediate shapes, GenCorres-NoImp often fails to register the template directly to a vastly different target shape, leading to poor initialization. GenCorres-NoInit, which does not employ any correspondence initialization, tends to get stuck in undesired local minima. Note that the success of training 3D-CODED relies on a very large dataset (23000 human meshes with a large variety of realistic poses and body shapes). 

Evaluations of JSM on DFAUST using geodesic errors of the predicted correspondences (in cm).

|                  | mean  | median |
|:---------------- |:-----:|:------:|
| GenCorres        | 1.30  | 1.13   |
| GenCorres-NoImp  | 7.92  | 7.73   |
| GenCorres-NoInit | 12.11 | 11.84  |

Reviewer comment (Reproducibility):
Code (and preprocessed data) would be really helpful so that others can understand and verify the claimed contributions. The reviewer asks whether it will be released.

Author response:
In the newly submitted supplementary material, we provide an additional experiment that applies the implicit generator to a new human dataset featuring various clothes and hairstyles. We have also included the corresponding code for this experiment along with a readme file. More code and data from the main paper will be released later.

Reviewer comment (Novelty):
A couple of relevant papers may be worth mentioning since they are in the overall spirit of this paper:  
- Muralikrishnan et al., "GLASS: Geometric Latent Augmentation for Shape Spaces", CVPR 2022 (learning generative models from sparse landmarks guided by ARAP regularization -- does require input with correspondences though so it's sort of complementary to this paper)  
- Bednarik et al., "Temporally-Coherent Surface Reconstruction via Metric-Consistent Atlases", ICCV 2021 (using metric distortion energies for consistent reconstruction, and hence joint matching, of time-varying shape sequences)

Author response:
We have added the relevant papers and corrected the typos.

Reviewer comment (Writing):
Minor: p3: Chamber --> chamfer (spelling, and should be lowercase).

Author response:
We have added the relevant papers and corrected the typos.