Paper: Intrinsic Mesh CNNs (ICLR-2024)

Reviewer comment (Novelty):
The authors focus on a limited body of work in the geometric deep learning literature, namely, three early formulations of intrinsic convolutions. Furthermore, they only cite one reference regarding all the theoretical framework of gauge equivariant CNNs, Bronstein et al., 2021. Combined, these two factors mean the paper ignores important references on gauge equivariant CNNs published before (Cohen et al.) or at the same time (Weiler et al.) the book cited, as well as other related works on convolutions for surfaces and 3D shapes (e.g., Tangent Convolutions for Dense Prediction in 3D in CVPR 2018, MeshCNN, and others).

Author response:
The theoretical contribution of our work focuses on the discovery of previously unknown but insightful
properties of intrinsic surface convolutions which depend only on the definition of priors and is as such
more fundamental and independent of gauge transformations. Therefore, while we think our theory
also applies to gauge-equivariant intrinsic surface convolutions, this topic exceeds the scope of this
work.

Reviewer comment (Theory):
The mathematical derivations contain typos (the limit of Gaussian functions for decreasing variance should be for n -> 0 not n -> +oo), and the use of the notation $\Delta_{\theta}$ and $\Delta_{\theta, w}$ should be introduced. The authors should introduce the change of integration domain from [0, 1]^n to BR(0) before Theorem 1. The theorem itself is a direct application of Fubini's theorem and as such is not a particularly strong theoretical result of the paper.

Author response:
We thank the reviewer for pointing out typos in our manuscript and will of course correct those, as well as add
the full definition for the anisotropic Laplacian $\Delta_{\alpha \theta}$ into the appendix to ensure completeness.

Reviewer comment (Theory):
As noted by the authors themselves, the Dirac distribution is not a function. This weakens the link with the theoretical results presented before.

Author response:
While mathematically possible to expand the domain of priors to transition kernels and with that
allowing for several new priors such as the Dirac prior, this redefines the Riemann integrals into
Lebesgue integrals which come with a significant amount of technical overhead, as stated in the manuscript,
and whose description would not help to understand our contribution.
Thereby, our choice of describing the Dirac prior allows to compare it to all previous models that use
priors, which can be expressed in terms of Riemann integrals, for example by comparing their sets of
learnable features. This, in turn, again shows the potential of reformulating IMCNNs in terms of using a
prior rather than a patch operator.

Reviewer comment (Novelty):
The authors claim to "we show that existing definitions for surface convolutions only differ in their prior assumptions about local surface information". It is known in the community, and the authors show it for the formulation of Monti only, who had already shown their formulation encompasses the previous Geodesic CNN and Anisotropic CNN.

Author response:
We generalize the framework of Monti et al., 2017, thereby obviously allowing for the definition of
models mentioned by Masci et al., 2015, and Boscaini et al., 2016, but additionally also the model
of Bronstein et al., 2021, and new models that use priors in the sense Theorem 1.

Reviewer comment (Theory):
Can the theoretical framework be strengthened by reformulating it in terms of distributions and extended to the class of test functions?

Author response:
While the discussion about distributions and test functions definitely is possible, we are convinced
that the choice of transition kernels is more natural as we think of signals on the manifold in a
statistical fashion. If we assume that our IMCNNs only learn test functions, we would restrict our
research on a comparably limited domain.

Reviewer comment (Experiments):
The network architecture used in the experiments uses angular max pooling. Can the authors clarify why this is needed? Isn't it a step backwards compared to the work of Monti, or other mesh CNNs such as FeastNet that do not require angular max pooling?

Author response:
Angular max-pooling (AMP) is required as a means to resolve the angular coordinate ambiguity
problem, a direct consequence of gauge-ambiguity, in practice. It was also used by Monti et al., 2017.
FeastNet (Verma et al., 2018), while representing interesting work, is a fundamentally different
approach to surface convolutions.

Reviewer comment (Experiments):
How did the authors choose the priors compared against the Dirac prior?

Author response:
The normal distribution has also been used by Monti et al., 2017. The other priors portray valid instances
for our class of IMCNNs and were selected for further comparison.

Reviewer comment (Theory):
My primary concern with this paper is that it’s not clear to me why it is called a prior and how using a different prior helps. It simply seems like different instances of the patch operator, akin to the Gaussian in Masci et al. 2015.

Author response:
The prior can be seen as a formal tool for encoding assumptions that you have about the signal on the surface.
Our contribution formally describes the effects of priors. Key to the description is the introduction
of the notion of *learnable features* in Section 4 of our manuscript. Depending on the chosen prior, the set of learnable
features of an IMCNN can be more or less comprehensive. Given a problem, like for example classification, we search for
features within the set of learnable features that cause the IMCNN to perform well. If the set of learnable
features is smaller, we expect that good features within that set are found faster. 
However, a small set of learnable features might exclude features which lead to better performances.
Larger sets of learnable features, in turn, imply more unstable training as searching for good features
becomes more complicated due to the increased amount of possible features to learn.

Reviewer comment (Theory):
Are the shot descriptors orientation invariant? Does a global gauge exist?

Author response:
SHOT-descriptors are rotation-invariant as they are computed with the help of relative angles between
vertex normals, which do not vary under the application of isometries. However, our work
does not discuss SHOT-descriptors. Therefore, we kindly refer to the original paper (Salti et al.,
2014) for further questions about SHOT-descriptors.

Reviewer comment (Theory):
Any linear map from a scalar signal \( s: M \to \mathbb R \) on an \( n \)-manifold, to another scalar signal can be represented by a function \( w : M \times M \to \mathbb R \), via the integral \( (s \star w)(x) = \int_M w(x, y)s(y)dy \). Typically, for a convolution-style operator, one does not want the parameters to depend on \( x \), so one chooses a frame/gauge \( \omega \) on the manifold and arrives at the "intrinsic manifold convolution" of definition 1 of the manuscript, with a parameter \( t: \mathbb R^n \to \mathbb R \).

Author response:
> Any linear map $s: M \to \mathbb{R}$ from a scalar signal on a $n$-manifold, to another scalar signal can be represented by a function [...]

The formulation of the intrinsic manifold convolution follows a different intention. Intrinsic manifold 
convolutions (Bronstein et al., 2021) represent a generalization from standard Euclidean convolutions
to convolutions on Riemannian manifolds. For this we require a notion of relative direction between
points within the manifold. Bronstein et al., 2021, proposes to use tangent vectors. We 
associate tangent vectors to coordinates by selecting one from multiple possible gauges for each
tangent plane. Eventually, we map said tangent vectors via the exponential map
onto the surface to retrieve manifold points on which the surface signal is defined.

Reviewer comment (Theory):
Monti et al. (2017) [4] recognize that the gauge cannot be chosen uniquely, so they choose \( J \) frames and apply a convolution on each choice of frame, with different parameters for each, i.e., choosing a parameter function \( t : \mathbb R^n \to \mathbb R^J \).

Author response:
> Monti et al (2017) [4] recognize that the gauge can not be chosen uniquely, so choose $J$ frames apply [...]

Monti et al., 2017, proposes a general algorithmic framework for the construction of previous geometric CNN
architectures such as, but not limited to, geodesic CNNs (Masci et al., 2015). Within said framework, when
specifying for the geodesic CNNs, $J$ refers to the amount of neighbors around surface point $x$,
at which we calculate the convolution. In the continuous case, the sum becomes an integral over the disc
$B_R(\textbf{0})$, that contains geodesic polar coordinates which portray the input to the selected gauge. 
Thereby, $R$ is pre-specified radius that typically refers to the injectivity radius of the earlier discussed
exponential map.

Reviewer comment (Theory):
In theorem 1, in the equation \( (s \star t)_{\Delta \theta, w}(u) \), the authors do something very different from the above approaches: instead of computing the output signal at one point by a single integral (over either the manifold or over the tangent plane), they compute a convolution as two integrals over the tangent plane.

Author response:
> However, what's done in theorem 1 in the equation of, is very different from any of the above. [...]

Our convolution is equivalent to how Monti et al., 2017, and Bronstein et al., 2021, would compute
intrinsic surface convolutions. We have shown this in Theorem 1. The proof is provided in the appendix
and comes down to the application of Fubini’s theorem. The function $t : \mathbb{R}^n \to \mathbb{R}$
represents the template that shall be learned, the function $w : \mathbb{R}^n \times \mathbb{R}^n \to \mathbb{R}$
references a hand-crafted prior.

Reviewer comment (Theory):
The authors use one parameterization \( \mathbb R^n \to \mathbb R \), and another parameterization \( \mathbb R^n \times \mathbb R^n \to \mathbb R \).

Author response:
> However, what's done in theorem 1 in the equation of, is very different from any of the above. [...]

Our convolution is equivalent to how Monti et al., 2017, and Bronstein et al., 2021, would compute
intrinsic surface convolutions. We have shown this in Theorem 1. The proof is provided in the appendix
and comes down to the application of Fubini’s theorem. The function $t : \mathbb{R}^n \to \mathbb{R}$
represents the template that shall be learned, the function $w : \mathbb{R}^n \times \mathbb{R}^n \to \mathbb{R}$
references a hand-crafted prior.

Reviewer comment (Theory):
The authors appear to suggest that this is similar to what Monti et al. (2017) [4] do, but this seems very different. As the authors themselves note in theorem 1, this notation is completely redundant and can be reduced to a single integral.

Author response:
> However, what's done in theorem 1 in the equation of, is very different from any of the above. [...]

Our convolution is equivalent to how Monti et al., 2017, and Bronstein et al., 2021, would compute
intrinsic surface convolutions. We have shown this in Theorem 1. The proof is provided in the appendix
and comes down to the application of Fubini’s theorem. The function $t : \mathbb{R}^n \to \mathbb{R}$
represents the template that shall be learned, the function $w : \mathbb{R}^n \times \mathbb{R}^n \to \mathbb{R}$
references a hand-crafted prior.

Reviewer comment (Theory):
The equation for \( (s \star t)_{\Delta \theta, w}(u) \) appears to do two convolutions with different parameters, which should indeed reduce to a single convolution.

Author response:
> However, what's done in theorem 1 in the equation of, is very different from any of the above. [...]

Our convolution is equivalent to how Monti et al., 2017, and Bronstein et al., 2021, would compute
intrinsic surface convolutions. We have shown this in Theorem 1. The proof is provided in the appendix
and comes down to the application of Fubini’s theorem. The function $t : \mathbb{R}^n \to \mathbb{R}$
represents the template that shall be learned, the function $w : \mathbb{R}^n \times \mathbb{R}^n \to \mathbb{R}$
references a hand-crafted prior.

Reviewer comment (Novelty):
In the rest of section 3, the authors make the unsurprising observation that if one of the two convolutions contains a Dirac delta, that convolution is an identity operation, and the double convolution reduces to just the second convolution.

Author response:
> The authors appear to suggest that this is similar to what Monti et al (2017) [4] does, but this appears to me as very different. [...] So what's the point of analyzing the double convolution, which no one uses? It's very different from what [4] proposes, so how does it bridge a gap between anything?

As noted before, Theorem 1 states that the framework of Monti et al., 2017, is equivalent to our
notation. Selecting Dirac as a prior does not make the intrinsic surface convolution an identity
operation, but returns the formulation for intrinsic surface convolutions of Bronstein et al., 2021.
That is, selecting Dirac as a prior within our reformulation of Monti et al., 2017, which we have
proven to be equivalent, closes the theoretical gap between the algorithmic framework of
Monti et al., 2017, and the theoretical derivation of the intrinsic surface convolution from
Bronstein et al., 2021.

Reviewer comment (Novelty):
It is unclear what the point is of analyzing the double convolution, which no one uses. It is very different from what [4] proposes, so it is unclear how it bridges a gap between anything.

Author response:
> The authors appear to suggest that this is similar to what Monti et al (2017) [4] does, but this appears to me as very different. [...] So what's the point of analyzing the double convolution, which no one uses? It's very different from what [4] proposes, so how does it bridge a gap between anything?

As noted before, Theorem 1 states that the framework of Monti et al., 2017, is equivalent to our
notation. Selecting Dirac as a prior does not make the intrinsic surface convolution an identity
operation, but returns the formulation for intrinsic surface convolutions of Bronstein et al., 2021.
That is, selecting Dirac as a prior within our reformulation of Monti et al., 2017, which we have
proven to be equivalent, closes the theoretical gap between the algorithmic framework of
Monti et al., 2017, and the theoretical derivation of the intrinsic surface convolution from
Bronstein et al., 2021.

Reviewer comment (Novelty):
The authors cite [1] for the gauge equivariant convolution. This should be a citation to [2] and also [3] in the context of meshes.

Author response:
> The authors cite [1] for the gauge equivariant convolution. This should be a citation to [2] and also [3]
in the context of meshes.

We are aware of the literature by Taco et al., 2019, and De Haan et al., 2020. We thank the reviewer
for pointing out that these citations are missing and will of course add them to the manuscript.

Reviewer comment (Experiments):
The authors should compare to [3] in their experiments. Those authors found close to 100% correspondence at 0 geodesic distance for the FAUST experiment, much better than the numbers reported in the present manuscript.

Author response:
> The authors should compare to [3] in their experiments. Those authors found close to 100 correspondence
at 0 geodesic distance for the the FAUST experiment [...]

The theoretical contribution of our work focuses on the discovery of previously unknown but insightful
properties of intrinsic surface convolutions and not on extensions of them such as gauge-equivariant
convolutions.

Reviewer comment (Writing):
It is very confusing to use \( w \) (w) and \( \omega \) (omega) in the same equation, the first referring to weights and the latter to the gauge. An alternative notation should be chosen.

Author response:
> It’s very confusing to use $w$ (w) and $\omega$ (omega) in the same equation, the first referring to weights and the
latter to the gauge.

Thank you for your feedback on the readability of our paper. We can rephrase $w(·)$ to $p(·)$, 
highlighting that $w(·)$ does not contain trainable weights but represents a handcrafted prior.

Reviewer comment (Theory):
Instead of decoupling $w_t(\cdot)$ into $w(\cdot)$ (the prior) and $t(\cdot)$ (the template) and learning the template, one may simply treat $w_t(\cdot)$ as the learnable parameter. If we are allowed to discretize $w_t(\cdot)$ with a sufficient amount of parameters, parameterizing $w_t$ is flexible enough. Making a (not quite accurate) analogy to regular Euclidean CNNs: it seems to me what the paper presents is to pre-convolve some handcrafted $H$ with the original convolutional kernel $K$, with $H$ being something very restrictive in the sense of both capacity and optimization. It therefore does not surprise me too much that the Dirac prior (equivalent to directly parameterizing $w_t$) is better than all other variants.

Author response:
Learning the parameters directly has been suggested by Bronstein et al., 2021. Pre-convolving the
template $t(·)$ with a handcrafted kernel $w(·)$ has implicitly been done in what previously has been
referred to as the patch operator (Masci et al., 2015; Monti et al, 2017). Our paper proposes a
reformulation of the parametric framework from Monti et al, 2017, which stresses the theoretical
implications of handcrafted kernels $w(·)$ by giving rise to the notion of learnable features. This
notion not only explains why the handcrafted kernel encodes prior knowledge, which in previous
work has merely been referred to as either interpolation coefficients, weighting function or kernel
(Masci et al., 2015; Monti et al, 2017), but also allows us to observe a partial order on priors in
the context of intrinsic mesh CNNs. Therefore, we now know that different priors can vary in their
comprehensiveness. One very interesting example is given by our proposed Dirac prior, which on the
one hand, when inserted into our reformulation, yields the theoretical definition for intrinsic surface
convolutions of Bronstein et al., 2021, and on the other hand renders the pre-convolution irrelevant.

Reviewer comment (Experiments):
Continuing the point above, I believe there is no reason not to learn the priors at the same time. Doing this may lead to some additional benefits in optimization.

Author response:
Learning the parameters directly has been suggested by Bronstein et al., 2021. Pre-convolving the
template $t(·)$ with a handcrafted kernel $w(·)$ has implicitly been done in what previously has been
referred to as the patch operator (Masci et al., 2015; Monti et al, 2017). Our paper proposes a
reformulation of the parametric framework from Monti et al, 2017, which stresses the theoretical
implications of handcrafted kernels $w(·)$ by giving rise to the notion of learnable features. This
notion not only explains why the handcrafted kernel encodes prior knowledge, which in previous
work has merely been referred to as either interpolation coefficients, weighting function or kernel
(Masci et al., 2015; Monti et al, 2017), but also allows us to observe a partial order on priors in
the context of intrinsic mesh CNNs. Therefore, we now know that different priors can vary in their
comprehensiveness. One very interesting example is given by our proposed Dirac prior, which on the
one hand, when inserted into our reformulation, yields the theoretical definition for intrinsic surface
convolutions of Bronstein et al., 2021, and on the other hand renders the pre-convolution irrelevant.

Reviewer comment (Experiments):
The benchmark seems to be quite toy. I feel that if the paper includes empirical results on the tasks presented by MeshCNN, it will be much more convincing.

Author response:
While we agree that there exist more interesting benchmarks, we want to clarify that our benchmark
is no artificial toy benchmark. For instance, compared to Hanocka et al., 2019, our input meshes
have 41328 edges. Their benchmark used considerable smaller meshes with roughly 750 edges for 
mesh classification and roughly 2250 edges for mesh segmentation. Additionally, in difference to
Hanocka et al., 2019, we have trained our IMCNN to predict dense point-correspondences. Predicting
dense point-correspondences is commonly regarded as a notoriously hard problem in the computer
vision community (Van Kaick et al., 2011). Lastly, the benchmark is a common benchmark in the
background papers (Masci et al., 2015; Boscaini et al., 2016; Monti et al, 2017) .

Reviewer comment (Evaluation):
How much time does the proposed implementation with non-Dirac priors consume, compared to Monti et al., 2017?

Author response:
The Dirac-prior effectively gets rid of a matrix multiplication, therefore requiring less computation
than the IMCNNs which use a specific prior.

Reviewer comment (Experiments):
How much difference does it make to change the hyperparameters, including the ones for the GPC coordinate systems and template discretization, in Table 1?

Author response:
In order to get an intuition for what difference the hyperparameter make, it is helpful to take a
look on Figure 4 in the appendix. While it is difficult to quantify how much difference varying
hyperparameter cause in general, we expect certain aspects to hold true. For instance, the GPC-
system radius should be chosen smaller or equal to the injectvity radius of the exponential map.