Paper: Alignment and Outer Shell Isotropy for Hyperbolic Graph Contrastive Learning (ICLR-2024)

Reviewer comment (Theory):
The explanation of Theorem 1 is wrong. Theorem 1 roughly says that distance among points in hyperbolic space can be approximated by a tree, but does NOT say its converse. However, your explanation is "any tree can be embedded into a Poincaré disk with low distortion." This is the converse of Theorem 1.

Author response:
* **Re. 6**, we are happy to rework that. We simply hint here at the connection of hyperbolic spaces to hierarchical representations - something every paper on hyperbolic DL talks about, be it Section 3.1 on the following paper or Eq. 11 of the second following paper.  
> Hyperbolic Image Embeddings, CVPR 2019  

  > Hyperbolic Entailment Cones for Learning Hierarchical Embeddings  

  We agree, Theorem 1 says that distance among points in hyperbolic space can be approximated by a tree subject to the right    mapping f. Indeed, it does not say the converse that any tree can be embedded in hyperbolic space.   
  \  
  \  
  To be absolutely honest, we are more than happy to remove the entire Theorem 1 as its removal changes nothing for the paper (as in, nothing in the paper relies on plugging it into Theorem 1).  
  \  
  \  
  When we talk about the dimensional collapse in the hyperbolic space, we know the $\ell_2$ norms of feature vectors in the ambient space  are related to the tree depth (of course that depth depends also on the negative curvature $c$). We know that the region within the shell (or ring to be more precise) in the ambient space should have uniform distribution as the collapse is something happening due to encoder being unable to fill certain parts of ambient space along the shell (to say it roughly). So, these are the properties we really care about here.  
  \  
  \  
  As such, they affect the tree depth, and its layout in the ambient space but we really do not want to go into the entire topology of the tree graph, as that is way beyond simple points we are trying to illustrate.

Reviewer comment (Writing):
Many important definitions are missed. What are $p_\mathrm{pos}$ and $p_\mathrm{data}$ in Equation (1)? Also, $p$ and $D$ are not defined in Equation (9).

Author response:
# 8. p and D are not defined in Eq. (9).  
There is no D in Eq. 9. There is d defined at the top of section 2.2. p is also explained in the line above Eq. 9.

Reviewer comment (Theory):
The motivation of introducing the term $\mathcal{L}_{U}^\mathcal{T}$ is to force the uniformity. However, since the normal distribution decays exponentially in the tangent space and the surface area of a hyper-ball in hyperbolic space exponentially grows with respect to its radius $r$, the density of the normal distribution decays double exponentially. So the concentration effect is much more dominant than the uniformity effect. The authors have not discussed it.

Author response:
Dear Reviewer,  
\  
\  
One thing that would be useful for us to understand is what you mean by **the disadvantage caused by the strong concentration effect caused by the KL divergence from the isotropic Gaussian distribution could surpass the benefit you receive from by forcing the learning results to be uniform by the KL divergence term**.  
\  
\  
The way we see it is as follows. We applied the exp map transformation from the tangent space to the ambient space. We encourage isotropic Gaussian in the tangent space. The distribution of that isotropic Gaussian (after pushing it through exp map transformation) into the ambient space is in Figure 7.  
\  
\  
Thus, our Eq.9 and 10 promote such a distribution of features in the ambient space.  

If one samples the actual distribution:  
* the highest density is placed along the outer shell ring region (along the Poincare disc boundary)  
* the density along the ring (chose some suitable radius) is uniform and higher than on other rings formed closer towards the centre  
* the density towards centre of Poincare disk decreases (faster than linearly)  

This is precisely what we desire.  
\  
\  
So we are trying to understand what **the disadvantage caused by the strong concentration effect** is on your mind because obviously we are keen to address the issue but we kind of do not see what the issue exactly is. The only potential issue that we see is that **we may for example prefer density to decrease faster than currently is as one moves from the boundary of Poincare disc towards its centre.** Is that what the reviewer meant?  
\  
\  
Or reviewer means that KL on a covariance formed from features on the tangent space may have some unexpected effects when trying to align $\bf \Sigma$ with the identity matrix? (we guess rev. may be talking about this?)  

Best regards,  
\  
Authors

Reviewer comment (Writing):
In Section 3.1., it says "The N pairs of node embeddings,... as in Appendix B." However, the reason is not described.

Author response:
# 5. In Section 3.1., it says "The N pairs of node embeddings,... as in Appendix B." However, the reason is not described.  

As our encoder generates embeddings in the hyperbolic space, it is essential to project (constrain) these embeddings onto the Poincaré ball as in many other papers:  
> Kernel Methods in Hyperbolic Spaces, IJCV 2023

Reviewer comment (Presentation):
Figure 4 does not make sense since the author states that 4b is the best while 4b ignores the topology of the original tree. If we could ignore the topology of the original graph, we should randomly map them to the lattice in Euclidean space. Also, if we consider the hyperbolic distance, 4d seems more uniform than 4b.

Author response:
# 6. Figure 4 does not make sense: author states that 4b is the best while 4b ignores the topology of the original tree. If we could ignore the topology of the original graph, we should randomly map them to the lattice in Euclidean space.  

The analysis does not assume prior knowledge of the original tree structure: why should it? **Fig 4.  represent potential embedding outcomes generated by the algorithm not some one and only solution** with topology A or B. Our objective is to evaluate/compare these figures to discern which one shows a lesser degree of dimensional collapse.  

Topology of input graph has little to do with its embeddings. One may embed an images into hyperbolic space which will enjoy the tree structure. too. Does rev. know underlying image space topology to connect both? We do not.

Reviewer comment (Evaluation):
The proposed method does not scale well with respect to the dimensionality, since it involves the determinant calculation. It might not be critical in real applications where the dimensionality is not large.

Author response:
# 7. The proposed method does not scale well w.r.t. the dimensionality, as it involves determinant calculation.  

We politely disagree.  Determinant by Cholesky factorization is fast. Overhead is a tiny fraction even with large d. The following table shows time w.r.t. d:  

|d|time(ms)|  
|-|-|  
|5| 0.03|  
|10|0.04|  
|20|0.06|  
|40|0.12|  
|80|0.39|  
|200|2.22|  
|500|22.1|  
|1000|119.7|  

Determinant is  widely used by many  papers with large $d$ and thousands/millions of samples. It is computationally reasonable choice (**and faster approximations of it exist**):  
> Learning Log-Determinant Divergences for Positive Definite Matrices (TPAMI 2021)

Reviewer comment (Writing):
What are $p_\mathrm{pos}$ and $p_\mathrm{data}$ in Equation (1)?

Author response:
# 9. What are $p_{pos}$ and $p_{data} $ in Eq. (1).  

These symbols follow standard terminology of Wang & Isola (2020) cited just above Eq. 1. We will explain them accordingly.

Reviewer comment (Writing):
There are many typos in the paper, please correct them.

Author response:
Thank you. Of course, we have now had another iteration of proofreading. We will fix all typos.

Reviewer comment (Presentation):
Figure 1 and 2 and their respective analysis in the text are not clear. Figures and their explanations could be improved / completed. As such, I would even suggest to move them in the supplementary material to have enough space to complete them.

Author response:
Duly noted. Limited space was exactly the reason why these \`dimensional collapse\' cases are not described better by us.

Reviewer comment (Writing):
Definition 2 could be made clearer.

Author response:
Thank you. We simply followed here standard literature here (not our contribution). Nonetheless, we will improve it accordingly.

Reviewer comment (Presentation):
Figure 4: modes of collapse should be put on the subplots. Clean tree should be isolated.

Author response:
Duly noted.

Reviewer comment (Presentation):
Figure 5: in the text you say that you randomly drop edges and nodes. But in the figure it seems that you rather mask some node features not completely remove nodes.

Author response:
Thank you. Indeed, we use edge drop and attribute mask just as the majority of other methods do. We have now corrected the text.

Reviewer comment (Novelty):
Mappings from hyperbolic spaces to Gaussian distributions were already studied in the Machine Learning literature (e.g., [A]). This should be clear in the paper. Also, it seems to mitigate the theoretical contribution stated in Theorem 2.

Author response:
Thank you. That is a very interesting paper. It appears our expression differs from that in [A]. Nonetheless, there is a connection and we will gladly acknowledge it.

Reviewer comment (Experiments):
Optimization in Riemannian framework is clearly an active field of research whose advances seem to be disregarded by authors (e.g., [B], [C]). Could you further justify your choice for RSGD and perform simple benchmarks with other more recent approaches?

Author response:
Kindly notice that **the optimization on Riemannian manifolds is not the topic of our paper**, at all. We simply use RSGD just as a baseline optimization tool, just as the paper [B] does use RSGD as a baseline. Nonetheless, it is a very interesting pointer as RADAM works a little bit better with a quicker convergence. Table below presents our current results:


|Method  |Disease         | Airport         | PubMed          |CiteSeer         |Cora            |
|---|---|---|---|---|---|
|HyperGCL+RSGD| 94.50 $\pm$0.43  |   93.55 $\pm$1.11 |  85.14$\pm$ 0.38  |  73.43 $\pm$ 0.35  |  84.47 $\pm$ 0.46 |  
|HyperGCL+RADAM| **94.61 $\pm$ 0.51**  |   **93.68 $\pm$ 0.97** |  **85.40 $\pm$ 0.25**  |  **73.68 $\pm$ 0.22**  |  **84.77 $\pm$0.62** |

Reviewer comment (Experiments):
Various clearly competitive approaches are not benchmarked (e.g., GraphMAE [D] and more importantly CCA-SSG [E]). The latter also conducts a reflection to circumvent dimensional collapse when operating in an Euclidean setting. The uniformity loss essentially comes down to enforcing embedding covariances to be close to an identity matrix like the one of an isotropic Gaussian distribution. [E] also establishes relations between CCA-SSG and Mutual Information maximization under Gaussian assumptions so in substance it clearly seems like an Euclidean counter to HyperGCL. As such, clear comparisons should be present in the paper. Interestingly, it seems that these methods need a higher embedding dimension to compete/outperform HyperGCL, which was benchmarked for a fixed embedding dimension of 50 but where the curvature parameter needs to be fine-tuned.

Author response:
Thank you. We are more than happy to add GraphMAE to discussions and comparisons. However, **GraphMAE is a Masked Auto-Encoder (MAE) rather than a Contrastive Learning (CL) approach**. 
\
\
Currently, in self-supervised learning, **two families (masked AE and CL) are used as complementary techniques** as they all exhibit very different strengths, i.e., different behavior for linear probing vs. fine-tuning etc.) In fact, they even produce entirely different characteristics of embedding spaces, as shown in several studies.
\
\
A fair way would be not to compare our method with GraphMAE, but in fact combine GraphMAE with our CL, as in current works that combine both MAE and CL. However, this is clearly beyond the scope of our work, as that alone could be a separate paper. Thus, we are not devising mixed Graph MAE+CL methods as in the LGP paper:
> Layer grafted pre-training: Bridging contrastive learning and masked image modeling for label-efficient representations, ICLR 2023


Thank you, **in comparisons with CCA-SSG [E] our approach appears better**:

| | Cora | Citeseer | Pubmed |
|-|-|-|-|
|CCA-SSG [E] |  84.2 |  73.1 | 81.6  |
|HyperGCL+RSGD  |  **84.47**   |  **73.43** |  **85.14** |  
|HyperGCL+RADAM|  **84.77** |  **73.68**  |  **85.40** |  

Kindly notice we compared in our paper with the Euclidean counterpart of HyperGCL too in Table 3 (the first row of table imposes uniformity). They all performed worse just as CCA-SSG performs worse.

In fact, **we observed that typical soft-uniformity on covariances still suffers from the dimensional collapse  as there is nothing to ensure the infinite penalty when an eigenvalue of covariance collapses to zero.** That is, if other loss components benefit at the cost paid by the soft-penalty, in CCA-SSG the collapse will happen.

**That is why CCA-SSG performs worse than our approach that enjoys logdet penalty.**(our logdet sets an infinite penalty under  eigenvalue of covariance going to zero - thus, collapse as in nullifying an eigenvalue under logdet is too costly to occur).

Reviewer comment (Experiments):
What are the hyperparameters involved in the perturbation strategy? Is there a validation of these parameters and if so, what is the sensitivity of the method to these hyperparameters?

Author response:
The choice of perturbation hyperparameters is heuristic, we grid-search on the hyperparameters (edge drop ratio and attribute mask ratio) on the validation set. Table below presents selected hyperparameters:
		
|Disease |Airport |Cora |PubMed |CiteSeer|
|---|---|---|---|---|
|Edge drop ratio        |0.1      |0.2        |0.2       |0.1          |0.3|
|Attribute mask ratio   |0.0      |0.0        |0.4       |0.3          |0.2|

Reviewer comment (Experiments):
Could you provide the sensitivity analysis to the curvature parameter on the node classification part?

Author response:
|c |Disease |Airport|
|---|---|---|
|c = 0.05  |92.74$\pm$0.22 |92.15  $\pm$ 1.31|
|c = 0.1  |94.20$\pm$0.26 |93.53  $\pm$ 1.10|
|c = 0.2  |93.60$\pm$0.14 |91.98  $\pm$ 1.13|
|c = 0.3    |92.80$\pm$0.16 |91.02  $\pm$ 1.02|

Reviewer comment (Experiments):
No experiments in semi-supervised learning settings. As HyperGCL seems to provide discriminant embeddings keeping low-dimensional embeddings, I would tend to believe that they would better suit semi-supervised learning than Euclidean contrastive graph learning methods. As CCA-SSG seems on par with HyperGCL (see point 8), it is not obvious that the overall hyperbolic setting is better than the Euclidean one.

Author response:
Thank you. In fact, CCA-SSG performs much lower on Disease and  Airport than our HyperGCL:

|Disease | Airport |Cora |PubMed |CiteSeer|
|---|---|---|---|---|
|CCA-SSG   |69.21$\pm$0.3 |81.98$\pm$0.4  |  84.2$\pm$0.4 |73.1$\pm$0.3 | 81.6 $\pm$0.4|
|GraphMAE  |70.61$\pm$0.5  |82.79$\pm$0.4 |  84.2$\pm$0.4 |73.2$\pm$0.4 | 81.1 $\pm$0.4|
|HyperGCL  |**94.50 $\pm$0.4** |**93.55$\pm$1.1** |  **84.4 $\pm$0.4** |**73.4 $\pm$ 0.3** | **85.1 $\pm$ 0.3**|

Reviewer comment (Evaluation):
Lack of clarity or hindsight with respect to the evaluation: no clear justifications for the choice of supervised evaluation. No fully unsupervised evaluations proposed, which would suit the learned topology.

Author response:
Supervised evaluation follows the typical GNN evaluation setting described in many papers, e.g.:
> Semi-Supervised Classification with Graph Convolutional Networks ICLR 2016

The self-supervised setting follows: 
> COSTA: Covariance-Preserving Feature Space Augmentation for Graph Contrastive Learning KDD 2022

Reviewer comment (Experiments):
No sensitivity analysis with respect to the encoder, nor a clear comparison between performances of this GNN backbone in a fully supervised setting versus the 2-step strategy used by authors to evaluate HyperGCL embeddings. Such analysis could relate to the common concerns in the GNN literature, e.g., (i) expressivity, simply considering several GNN layers using Jumping Knowledge-based backbones; (ii) homophily vs. heterophily via [F], whose supervised models exhibit considerably higher classification performances than those reported in Table 1.

Author response:
Since we could not find the code to reproduce the result in [F],  we compare different GNN backbones (e.g., GCN, GAT, JKNet) on Disease/Airport with supervised setting and SSL setting (2-step) with HyperGCL:


|Method |Disease (Supervised) | Airport (Supervised) | Disease (SSL HyperGCL(2-step)) | Airport (SSL HyperGCL(2-step)) |
|-|-|-|-|-|
|GCN   |69.61$\pm$0.5 |81.40 $\pm$ 0.4 |  94.5 $\pm$0.4| 93.57$\pm$1.1|
|GAT   |70.40$\pm$0.4 |81.50  $\pm$ 0.3 |  93.8 $\pm$0.3| 92.72$\pm$1.2|
|JKnet |68.80$\pm$0.4 |80.53  $\pm$ 0.3 |  92.6 $\pm$0.4| 92.44$\pm$0.9|

Reviewer comment (Reproducibility):
Do you normalize embeddings or just center them?

Author response:
We do not normalize embeddings, we only center them.

Reviewer comment (Evaluation):
The choice of a linear classifier to evaluate clearly non-linear embeddings is not so obvious (I know that it is not questioned in most graph contrastive learning papers). Could you provide complementary evaluations with a non-linear classifier, e.g., 2-MLP with ReLU activation?

Author response:
| |Disease | Airport |Cora |PubMed |CiteSeer|
|---|---|---|---|---|--|
Linear classifier         |94.50 $\pm$0.4 |93.55$\pm$1.1 |  84.4 $\pm$0.4 |73.4 $\pm$ 0.3 | 85.1 $\pm$ 0.3 |
2-MLP with ReLU   |94.82 $\pm$0.5 |93.98$\pm$1.4 |  84.3 $\pm$0.5 |73.8 $\pm$ 0.2 | 85.3 $\pm$ 0.5 |

In general, there is some benefit to a non-linear classifier. Although in high-dimensional feature space features are mostly linearly separable. For Cora, there is even a drop in performance on 2-MLP with ReLU which we suspect may be a sign of overfitting.

Reviewer comment (Novelty):
The paper's weakness lies in the observed marginal improvement in performance over prior methods. While the idea is intriguing, the practical impact of the proposed framework in terms of tangible performance gains may be limited.

Author response:
Thank you. We do not consider the improvements to be marginal, given that the model has achieved State-of-the-Art (SOTA) performance across six graph datasets and three recommendation datasets.

**While datasets may display indicators nearing saturation** (broader problem in graph learning), suggesting limited potential for large improvement gains on their own, our method demonstrates the ability to continually enhance performance. 
\
\
This constantly present improvement underscores the superiority of our approach, even in scenarios where immediate gains might seem difficult for other methods to achieve.

|Method  |Disease         | Airport         | PubMed          |CiteSeer         |Cora            |
|---|---|---|---|---|---|
|COSTA   | 67.12 $\pm$ 0.39  |   81.19 $\pm$ 0.40 |  84.31 $\pm$ 0.37  |  70.77 $\pm$0.24  |  82.14 $\pm$0.62 | 
|HGCL    | 93.42 $\pm$ 0.82  |   92.35 $\pm$1.51 |  83.14 $\pm$0.58  |  72.11 $\pm$0.64  |  82.37 $\pm$ 0.47 | 
|HyperGCL| 94.50 $\pm$ 0.43  |   93.55 $\pm$ 1.11 |  85.14 $\pm$0.38  |  73.43$\pm$ 0.35  |  84.47 $\pm$0.46 |   

Kindly notice **in all cases we have between 1-2\% gain demonstrating the main point of our easy-to-compute add-on.**
\
\
**We have run also the statistical t-test** given we have deviation and numbers of runs. Except for Airport, in all cases our gains are **statistically significant** within 95\% confidence intervals.

Reviewer comment (Evaluation):
The absence of clear computational complexity advantages over existing methods is a point of consideration. Without distinct efficiency benefits, it becomes challenging to justify the adoption of the proposed approach, especially if it offers similar performance to existing methods. Addressing this aspect would enhance the paper's practical appeal.

Author response:
The only overhead introduced in our method is to compute determinant. However, the determinant computation can be efficiently achieved with the Cholesky factorization even with larger feature dimensionality $d$. 
\
\
The following table show the time in ms w.r.t to dimension. 

d   | time(ms)|
--- | ---  |
5   | 0.030|
10  | 0.040|
20  | 0.061|
40  | 0.123|
80  | 0.393|
200 | 2.223|
500 | 22.18|
1000| 119.7|

In contrast to the naive approach in Eq. 6, such as calculating the pair-wise Riemannian distance between all point combinations, our method enjoys less computation.
\
\
As can be seen, **our method just ads few milliseconds per sample which is negligible compared with the cost of running the backbone.**

Cholesky has in practical terms complexity lower than the complexity of SVD $\mathcal{O}(\min(d^2 n, n^2 d))$. Kindly notice matrix determinant has very efficient parallelized implementation in PyTorch and is frequently used in deep learning papers:
> Learning Log-Determinant Divergences for Positive Definite Matrices (TPAMI 2021)

Reviewer comment (Reproducibility):
The results tables (Table 1/2) look a bit "too good to be true": HyperGCL take the first place for every single dataset. Did the authors obtain all the values themselves (by running all the algorithms on all datasets)? Or are the values for other algorithms taken from the respective publications? This should be clarified.

Author response:
Thank you. These are the results we obtained. If they are good, we are glad.

In Table 1, the results have been replicated using their official code.
\
In Table 2, the results are sourced from the HRCF paper.

Reviewer comment (Novelty):
According to Table 1, HGCL is also a hyperbolic graph contrastive learning method. The authors should explain how it is different from HyperGCL. Is it the only hyperbolic graph contrastive learning method in existence?

Author response:
* HGCL treats node embedding in the hyperbolic space as one view and the node embedding in the Euclidean space as another. It employs typical contrastive learning objective on these two views to learn node embeddings. 

* Our method utilizes both views in the hyperbolic space and aims to mitigate the dimension collapse through alignment and outer shell isotropy, which yield a good improvement.

Reviewer comment (Writing):
Page 3, Definition 2: should \mathbb D^n_c be D^d_c? Previously you only used D^d_c.

Author response:
Indeed, thank you for noticing this. We have now fixed accordingly.