Paper: Fully Hyperbolic Convolutional Neural Networks for Computer Vision (ICLR-2024)

Reviewer comment (Evaluation):
Across all the experiments, the improvement of the proposed HCNN Lorentz/HECNN Lorentz over the standard Euclidean neural network is usually small (65.96 vs. 65.19 on Tiny-ImageNet), so it is not clear what is the practical advantage of the proposed method over standard Euclidean neural network.

Author response:
Citing our response to reviewer Nvg1:

“[...] The improvements of our HECNN compared to the other provided models on CIFAR100 and Tiny-Imagenet are significant. We provide a table with the p-value for the respective unpaired t-tests below:

|                 | CIFAR-100 | Tiny-ImageNet |
|-----------------|:-----------:|:---------------:|
| Euclidean       | <0.0001   | <0.0001       |
| Hybrid Poincaré | 0.0002    | 0.0006        |
| Hybrid Lorentz  | 0.0009    | 0.0072        |
| HECNN            | -         | -             |
| HCNN            | 0.0008    | 0.0359        |

We also provide a table with the models vs Euclidean performance here:

|                 | CIFAR-100 | Tiny-ImageNet |
|-----------------|:-----------:|:---------------:|
| Euclidean       | -         | -             | 
| Hybrid Poincaré | 0.053     | 0.1827        | 
| Hybrid Lorentz  | 0.0277    |  0.0002       |
| HECNN           | <0.0001   | <0.0001       | 
| HCNN            | 0.0087    |  0.0002       |

These indicate very high statistical significance for HECNN vs all other models and high significance for Lorentz models vs Euclidean. This is also important as no HNN from the literature had outperformed Euclidean models on standard classification before.”

 We also try to emphasize the advantageous properties of HNNs. The proposed Lorentz models show a much larger performance gap for adversarial robustness tasks and when using lower dimensional embeddings. One more thing to note is that this work is also geared toward providing the building blocks for further hyperbolic work for vision models. These tools could be used to create specialized approaches later on.

Reviewer comment (Evaluation):
There is a lack of in-depth analysis of the experimental results. For example, the results show that HECNN is the best when the embedding dimension is low. However, the reason of such an improvement is not clear. Similarly for the image generation task.

Author response:
The lack of extensive motivation and analysis was a result of the page limit. We tried to improve it in the revised manuscript (see, e.g., Section 5.1 and Appendix D.1 for hyperbolicity motivation). We now try to include the motivation, hypothesis, and more analysis for every experiment.

Reviewer comment (Experiments):
It is known that hyperbolic space is well suited for hierarchical dataset, however, none of the experiments clearly demonstrate this.

Author response:
We acknowledge that there are no explicit hierarchy experiments. However, most hierarchies in images are intrinsic (on a localization level). To reflect this, we added the measured hyperbolicity of each dataset, the lower the $\delta_{rel}>0$ the higher the hyperbolicity/tree-like structure of image features (see Table 2 and Appendix D.1). Additionally, all datasets include hierarchical relations between object classes (e.g., superclass label in CIFAR-100 (https://www.cs.toronto.edu/~kriz/cifar.html)). Only CIFAR-10 might have too few classes for any significant object class relations.

We revised the experiments section to make our points clearer.

Reviewer comment (Evaluation):
The authors show that fully hyperbolic neural networks is better than the hybrid version, however, would fully hyperbolic neural networks incur more parameters than the hybrid version? Also, how about the time complexity comparison?

Author response:
Overall, HCNNs have no major impact on the number of parameters. 

However, time complexity is certainly still an issue. The convolutional layer is the main bottleneck both in terms of computation time and memory requirements. This is generally because we lose the heavily optimized closed-source CUDA convolution implementations when we replace PyTorch’s built-in convolution. To verify this, we reimplemented the Euclidean convolution with the same naive operation breakdown (Unfold + matrix multiplication)  and it is around 5.5x slower when compared to the built-in convolution (averaged over 2000 runs with a 3x3 kernel). Thus, most of the computational overhead of our HCNN appears to be implementation-related. We verify this through the use of PyTorch 2.0’s inbuilt “compile” which is able to drop the training time of the hyperbolic mode by around 40% by attempting to find mergeable operations in the computation graph before runtime (as we can see in Appendix D).
To verify this we also measure the FLOPs of all 3 convolution types (nn.Conv2d, reimplemented conv2d and Lorentz conv2d) as reported by the pytorch profiler. This was for an input of size 128x64x64x3 and kernel size 3x3 with 0 padding. All were approximately around 1.7 GFLOPs (with the lorentz conv being slightly less even) despite the drastic time and memory differences.

Reviewer comment (Evaluation):
I find the evidence for the actual usefulness of hyperbolic representations not convincing, in contrast to what the authors present in the abstract and introduction by saying things like "demonstrate the superiority of our HCNN framework". This is not to downplay the results, I still think it is valuable that these original approaches are developed and explored, however, I think some conclusions could be nuanced and perhaps more precisely formulate.

Author response:
> 1. - 4. The main results are very close and do not show clear “superiority” (see CIFAR-10). So, some conclusions could be nuanced and perhaps more precisely formulated. Additionally, almost any neural network can get great results if you engineer it well enough.

We agree that our claims and conclusions should be more nuanced and tried to implement your suggestions into the revised version.

In terms of classification results in Table 1,
- CIFAR-10 is presented as more of a toy example. Accuracy on CIFAR-10 is nearly saturated, and it becomes difficult to get improvements with the same model size. 
- The improvements of our HECNN compared to the other provided models on CIFAR100 and Tiny-Imagenet are significant. We provide a table with the p-value for the respective unpaired t-tests below:

|                 | CIFAR-100 | Tiny-ImageNet |
|-----------------|:-----------:|:---------------:|
| Euclidean       | <0.0001   | <0.0001       |
| Hybrid Poincaré | 0.0002    | 0.0006        |
| Hybrid Lorentz  | 0.0009    | 0.0072        |
| HECNN            | -         | -             |
| HCNN            | 0.0008    | 0.0359        |

We also provide a table with the models vs Euclidean performance here:

|                 | CIFAR-100 | Tiny-ImageNet |
|-----------------|:-----------:|:---------------:|
| Euclidean       | -         | -             | 
| Hybrid Poincaré | 0.053     | 0.1827        | 
| Hybrid Lorentz  | 0.0277    |  0.0002       |
| HECNN           | <0.0001   | <0.0001       | 
| HCNN            | 0.0087    |  0.0002       |

These indicate very high statistical significance for HECNN vs all other models and high significance for Lorentz models vs Euclidean. This is also important as no HNN from the literature had outperformed Euclidean models on standard classification before.

 One more thing to note is that this work is also geared toward providing the building blocks for further hyperbolic work for vision models. These tools could be used to create specialized approaches later on.

Concerning over-engineering, we aimed to avoid this by using the best known hyperparameters of the Euclidean model during the training of our hyperbolic models. We also make the model architectures as one-to-one as possible.

Reviewer comment (Experiments):
It is not always clear what the hypotheses are. It seems like it is "just interesting" to try and build a fully hyperbolic network, without clearly specifying why this would be useful. In particular, the experiments should be accompanied with clear hypotheses of what is expected (why would the Lorentz model be better than the Poincare model if both are isometric representations of the same thing? Why would one expect HE or H model be better compared to the other? Why would the H model be more robust to adversarial attacks?). It would help if the experiments are more clearly motivated, and explain to what questions these experiments provide answers.

Author response:
> 2. & 4. The hypothesis why to build a fully hyperbolic model is not clear. Experiments should be more clearly motivated. Some very narrowly defined experiments could be defined to test for desirable properties of the network (if this is possible).

The lack of extensive motivation was a result of the page limit. We tried to improve it in the revised manuscript (see, e.g., Section 5.1 and Appendix D.1 for hyperbolicity motivation). We now try to include the motivation, hypothesis, and more analysis for every experiment.

Known desirable properties of HNNs are robustness to adversarial attacks and effectiveness at lower feature dimensionalities. Our experiments (Table 2, Figure 3) are designed to test these and now word them better.

Reviewer comment (Evaluation):
The results, in particular in Table 1, all look the same to me. Almost all results lie within each others uncertainty margins and hence drawing conclusions based on these results is dubious, especially if they as strongly state as on some instances in the paper.

Author response:
> 1. - 4. The main results are very close and do not show clear “superiority” (see CIFAR-10). So, some conclusions could be nuanced and perhaps more precisely formulated. Additionally, almost any neural network can get great results if you engineer it well enough.

We agree that our claims and conclusions should be more nuanced and tried to implement your suggestions into the revised version.

In terms of classification results in Table 1,
- CIFAR-10 is presented as more of a toy example. Accuracy on CIFAR-10 is nearly saturated, and it becomes difficult to get improvements with the same model size. 
- The improvements of our HECNN compared to the other provided models on CIFAR100 and Tiny-Imagenet are significant. We provide a table with the p-value for the respective unpaired t-tests below:

|                 | CIFAR-100 | Tiny-ImageNet |
|-----------------|:-----------:|:---------------:|
| Euclidean       | <0.0001   | <0.0001       |
| Hybrid Poincaré | 0.0002    | 0.0006        |
| Hybrid Lorentz  | 0.0009    | 0.0072        |
| HECNN            | -         | -             |
| HCNN            | 0.0008    | 0.0359        |

We also provide a table with the models vs Euclidean performance here:

|                 | CIFAR-100 | Tiny-ImageNet |
|-----------------|:-----------:|:---------------:|
| Euclidean       | -         | -             | 
| Hybrid Poincaré | 0.053     | 0.1827        | 
| Hybrid Lorentz  | 0.0277    |  0.0002       |
| HECNN           | <0.0001   | <0.0001       | 
| HCNN            | 0.0087    |  0.0002       |

These indicate very high statistical significance for HECNN vs all other models and high significance for Lorentz models vs Euclidean. This is also important as no HNN from the literature had outperformed Euclidean models on standard classification before.

 One more thing to note is that this work is also geared toward providing the building blocks for further hyperbolic work for vision models. These tools could be used to create specialized approaches later on.

Concerning over-engineering, we aimed to avoid this by using the best known hyperparameters of the Euclidean model during the training of our hyperbolic models. We also make the model architectures as one-to-one as possible.

Reviewer comment (Experiments):
It would be great if some very narrowly defined experiments could be defined to test for desirable properties of the network (if this is possible). Because, the performances are all so close to each other which I think has to do with the fact that almost any neural network can get great results if you engineer it well enough, even if the code contains bugs or design flaws! However, in order to define such experiments one first has to clearly specify what unique properties are obtained by the HCNNs, which I couldn't really get from the paper, nor did I find concrete research questions that are being addressed.

Author response:
> 2. & 4. The hypothesis why to build a fully hyperbolic model is not clear. Experiments should be more clearly motivated. Some very narrowly defined experiments could be defined to test for desirable properties of the network (if this is possible).

The lack of extensive motivation was a result of the page limit. We tried to improve it in the revised manuscript (see, e.g., Section 5.1 and Appendix D.1 for hyperbolicity motivation). We now try to include the motivation, hypothesis, and more analysis for every experiment.

Known desirable properties of HNNs are robustness to adversarial attacks and effectiveness at lower feature dimensionalities. Our experiments (Table 2, Figure 3) are designed to test these and now word them better.

Reviewer comment (Presentation):
In the Lorentz model section, Figure 4 is reference. It is not nice to refer to a figure which isn't even included in the document. Moreover, the proper way of refering to figures in order of appearance (e.g. first refer to figure 1, then 2 then 3 etc. Not starting with fig 4). I think if this figure is that important it should be included in the main body. Moreover, it would have been nice to include the appendix in the main pdf, not as a separate document.

Author response:
> 5. In the Lorentz model section, Figure 4 is referenced but not in the main document. Appendix should be in the main document.

We agree with you and moved the figure to Section 3. We also moved the Appendix to the main document.

Reviewer comment (Writing):
Page 3, on the Lorentz model. What is "the upper sheet" and what is meant with a "two-sheeted hyperboloid"?

Author response:
> 2. & 3. What is "the upper sheet" and what is meant with a "two-sheeted hyperboloid"? In Equation 1 it says $x_t>0$, why not $x_t \geq \sqrt{-1/K}$?

As an example consider the 3D two-sheeted hyperboloid with $K=-1$. The hyperboloid is then the set of points that satisfies the equation $x^2+y^2-z^2=-1$. This gives us the shape we see here: https://nmd.web.illinois.edu/quadrics/hyper2.html.

When we say upper sheet, we mean the set of points that lie above the $z=0$ plane in this case, this is also why we write $x_t>0$ in Eq. (1). While it is correct to write $x_t \geq \sqrt{-1/K}$, using $x_t>0$ seems to be the conventional representation of the upper sheet model in the literature.

Reviewer comment (Theory):
In Equation 1 it says \(x_t > 0\). Is this because \(x_t \geq \sqrt{-1/K}\), and if so, why not specify that instead?

Author response:
> 2. & 3. What is "the upper sheet" and what is meant with a "two-sheeted hyperboloid"? In Equation 1 it says $x_t>0$, why not $x_t \geq \sqrt{-1/K}$?

As an example consider the 3D two-sheeted hyperboloid with $K=-1$. The hyperboloid is then the set of points that satisfies the equation $x^2+y^2-z^2=-1$. This gives us the shape we see here: https://nmd.web.illinois.edu/quadrics/hyper2.html.

When we say upper sheet, we mean the set of points that lie above the $z=0$ plane in this case, this is also why we write $x_t>0$ in Eq. (1). While it is correct to write $x_t \geq \sqrt{-1/K}$, using $x_t>0$ seems to be the conventional representation of the upper sheet model in the literature.

Reviewer comment (Writing):
Section 4.1 Formalization of the convolution layer... It says "an affine transformation between a linearized kernel and a ..." an affine transformation is a transformation of something, not between two things right? One could say the transformation of a vector is parametrized by a kernel (and a bias), or am I misunderstanding something here?

Author response:
> 4. Is saying  "...an affine transformation between a linearized kernel and a ..." correct?

Yeah, you are right. We can refer to a transformation as “being between” the original vector and the resultant vector, but in this context, you are definitely correct. We’ve changed it in the revised edition to “We define the convolutional layer as a matrix multiplication between a linearized kernel and a concatenation of the values in its receptive field, following \citet{shimizu-et-al-2020}.”.

Reviewer comment (Presentation):
On the same note, equation 3: The definition of LFC seems essential to me, why is it hidden in the appendix and not the main body? This layer is the main work-horse of the neural network, I imagine.

Author response:
> 5. Why is the definition of the LFC hidden in the Appendix?

We had a few issues conforming with the page limit on the paper and since the LFC closely follows the definition by Chen et al. (2021), we chose to add it to the appendix. We could add it back into the main paper but we would have to sacrifice other content in its stead. However, we have moved the Appendix to the main document to make it more visible.

Reviewer comment (Writing):
Next, "the correct local connectivity is established", what does that mean?

Author response:
> 6. What does  "the correct local connectivity is established" mean?

This simply means that the features are multiplied with the correct kernel parameters (considering, e.g., stride and padding). We used the word “connectivity” because convolutional layers are usually visualized by a sliding kernel connecting to the feature map.

Reviewer comment (Writing):
Equation 5, why are there additional \(|\ |\) (vertical bars). I count three of them on each side. Two for the norm, which makes sense, but then another outside of it. Please remove it unless there is a particular reason for it.

Author response:
> 7. Eq. 5: Why are there additional | | (vertical bars)?

Eq. 5 defines the Lorentzian centroid and was introduced in Eq. 11 of Law et al. (2019). They note “[...] where $| \lVert a \rVert_{\mathcal{L}} | = \sqrt{| \lVert a \rVert^2_\mathcal{L} | }$ is the modulus of the imaginary Lorentzian norm of the positive time-like vector a.” So the additional vertical bars are more of a mathematical technicality.

Reviewer comment (Theory):
In 4.4 you mention "... provides best empirical performance", that is nice. But what is theoretical lost with the approach of simply adding the spatial parts? Is this mathematically allowed? It feels like a hack that could theoretically break certain properties of the network (not sure about this though). I suppose since addition is not defined you need to define an alternative, and this one seems as good as anything, but until this point the motivation has been primarily the idea of parallel transport and Frechet mean type "additions".

Author response:
> 9. Regarding residual connection in Section 4.4: What is theoretically lost with the approach of simply adding the spatial parts? Is this mathematically allowed? Until this point, the motivation has been primarily the idea of parallel transport and Frechet mean type "additions".

As you said, one of the issues with the residual connection is the lack of actual addition operation in the hyperbolic space. Initially, we thought of replacing it with parallel transport but found that it was both unstable and expensive (see Appendix D). The idea of space addition is similar to that presented by Chen et al. (2021); it is a pseudo-hyperbolic operation. One can consider it mathematically closer to being a transformation parametrized by the residual value. It would then be able to fall under the boost and rotation operations covered by Chen et al. (2021).

Reviewer comment (Writing):
In section 5.1 you write "replace only the resnet encoder blocks with the highest hyperbolicity", what is "hyperbolicity? What is \(\delta_{rel}\), and why would different blocks have different hyperbolicity? Many details are missing here.

Author response:
> 10. What is "hyperbolicity”? What is $\delta_{rel}$, and why would different blocks have different hyperbolicity? Many details are missing here.

We agree that details on hyperbolicity are missing. That’s why we added a short summary to Appendix D.1.

Hyperbolicity was the main talking point in the paper of Khrulkov et al. (2020). The authors first produce image embeddings from a vision backbone and then measure the “distance distortion” between them. This is done by presenting the data in $\delta$-slim triangle and finding the smallest $\delta$ that validates the triangle inequality postulate using the Gromov product. $\delta_{rel}$ is basically the calculated scale invariant value of $\delta$. This is a very brief explanation, and we refer to the original authors for a much more in-depth analysis.

Khrulkov et al. do this measurement only for the embeddings of the final block. We repeat this measurement on the embedding sets outputted by each of the other blocks as well. These different blocks could have different hyperbolicities because of the underlying features learned by the encoder, the data dimensionality, and model shallowness, which could, in turn, affect the representative capacity of the embeddings, etc…

Reviewer comment (Reproducibility):
Also, when working with hybrid models, how does one convert a hyperbolic representation to an Euclidean one?

Author response:
> 11. How does one convert a hyperbolic representation to an Euclidean one?

There are two main ways to extract Euclidean values from a hyperbolic network. The first is projecting the values onto the tangent space. Since the tangent space is the Euclidean approximation of the surface, the projected values are entirely Euclidean. The second option is to use distance measures calculated through hyperbolic methods as these are then only values and not “hyperbolic.” Based on prior works (hybrid VAEs), we used the first option.

Reviewer comment (Evaluation):
Main result section: "This suggests that the Lorentz model is better suited for HNNs than the Poincaré model", was this expected, why or why not?

Author response:
> 12. Main result section: "This suggests that the Lorentz model is better suited for HNNs than the Poincaré model", was this expected, why or why not?

Prior works showed that the Lorentz model is more numerically stable than the Poincaré ball. We hypothesized this would reduce the inaccuracies and rounding errors that could lead to a degradation in performance.

Numerical stability is an important issue in HNNs. A comprehensive study conducted by Mishne et al. (2022) reveals the following key insights:

1. The main issue lies in the exponential growth of the hyperbolic space’s volume w.r.t. the radius, and that embeddings are usually pushed towards infinity/the boundary. This introduces numerical instability and rounding errors, particularly when encountering unrepresentable values in floating-point arithmetic (intermediate results and numbers can only be represented with fixed precision, e.g., 32-bit).
2. Using the Poincaré ball leads to a “hard” constraint on the representation capacity, where points with a certain distance from the origin will collapse to the boundary. In contrast, using the Lorentz model leads to a “soft” constraint, where some operations can still be performed correctly.
3. The Poincaré ball is more susceptible to gradient vanishing. In the Poincaré model, we are often dealing with significantly smaller magnitudes of numbers compared to the Lorentz model. This leads to a bigger gradient vanishing problem, particularly when points are positioned close to the boundary and rounding errors accumulate.

Reviewer comment (Experiments):
Regarding robustness experiments. Why is this a relevant experiment?

Author response:
> 14. Regarding robustness experiments. Why is this a relevant experiment?

In the literature, two prominent advantages of using hyperbolic models are robustness to attacks and much better performance given lower-dimensional feature maps within the model. That is why we perform both of these experiments. It helps verify these findings and also further highlights these desirable properties.

We included the motivation in the revised manuscript.

Reviewer comment (Writing):
Just above the conclusion "As these structures cannot be found for the hybrid model, ... little impact ..." I do not understand this sentence, could this be clarified?

Author response:
> 15. Just above the conclusion "As these structures cannot be found for the hybrid model, ... little impact ..." I do not understand this sentence, could this be clarified? Thank you!

We meant that the equidistant clusters we found in the latent embedding space of the fully hyperbolic model were not present in the hybrid model. In fact, the shape of the clusters barely changed between the hybrid and the Euclidean model, although the hybrid model learns embeddings in hyperbolic space. This could indicate that including only one hyperbolic layer (as is done in the baselines) is not enough to meaningfully change the internal feature embedding structure of the model. Therefore, its behavior is not really “hyperbolic”.