Paper: Graph Neural Networks on Symmetric Positive Definite Manifold (ICLR-2024)

Reviewer comment (Novelty):
Missing at least one recent paper on the same topic with equivalent results on the same experiments (Hyperbolic Representation Learning : Revisiting and Advancing, M. Yang et al. ICML'23). Moreover, the numerical results for the methods presented in both papers are consistent, but not identical, with the globally higher performance reported in Yang et al.

Author response:
Thanks for sharing with us this up-to-date work. There are three main reasons for the slight discrepancies between its results and the ones we reported: 1) Variations in experimental settings, as it employs higher-dimensional representations (8, 64, 256 dimensions), whereas we chose matrix features of 5 and 10 dimensions, corresponding to node representations of 15 and 55 dimensions, respectively. 2) Differences in the reported metrics, as it presents accuracy for Cora, whereas our four datasets are uniformly reported in terms of F1-score. 3) Differences in datasets like Citeseer and PubMed . Furthermore, to our knowledge, its source code is currently not publicly available, making it challenging for us to consider it as our comparative method. Nonetheless, we will cite it and add relevant discussion in the revised version.

Reviewer comment (Evaluation):
In the same vein, it would be good to discuss why geometric learning seems to underperform on PubMed and Cora and gives scores for other types of GNN on the same experiments (best competitor). A quick search gives scores on PubMed (91.4 points method from 2021) and Cora (90.16 points method from 2020) that are largely superior, with a caveat on the experimental setting.

Author response:
Thank you for your thoughtful comments. We understand your concern regarding the seemingly lower performance of geometric learning on PubMed and Cora. Allow us to provide a more detailed explanation:
- **Problem Setting Clarification:**
It's crucial to emphasize that the default development of GNNs is rooted in Euclidean space assumptions, and successful methods within this paradigm often presume settings in Euclidean space. However, these assumptions may not universally hold, prompting our exploration of alternative geometric frameworks, specifically hyperbolic and SPD geometries. Our endeavor surpasses the constraints of Euclidean geometry, delving into more intricate geometric attributes.

- **Addressing Fundamental Problems:**
Our approach is distinct in that it tackles a more fundamental problem within geometric learning. While state-of-the-art GNN methods developed in Euclidean space can be extended to hyperbolic or SPD manifolds, our work focuses on exploring the foundational geometric embedding space upon which GNNs rely. This emphasis on fundamental geometric properties allows us to understand the limitations and potentials of these spaces in the context of graph representation learning.

- **Representative Baselines:**
We exclusively compare our approach against representative methods endowed with geometric properties, specifically Euclidean and hyperbolic geometries. The comparison is intentionally limited to methods that **share a similar architectural foundation**, allowing for a more focused assessment of the inherent advantages and challenges associated with our proposed SPD geometry.

In essence, our work extends beyond the conventional Euclidean assumptions of GNNs, exploring alternative geometric spaces to address fundamental challenges in graph representation learning. We acknowledge the potential superiority of state-of-the-art methods in Euclidean space, and we appreciate your suggestion to include additional context in our discussion. We will ensure that the revised manuscript provides a more comprehensive explanation of the motivation behind our geometric choices and their implications on the reported performance.

Reviewer comment (Theory):
The case of edge-features is not addressed.

Author response:
Thank you for your nice suggestion. Indeed, edge features constitute a pivotal concern. However, this manuscript concentrates on the exploration of latent geometric spaces, with further investigations into its higher echelons earmarked for our subsequent endeavors.

Reviewer comment (Theory):
Is the initial mapping limited to a linear map, what is its influence?

Author response:
There is no absolute constraint. In truth, the objective of the initial mapping is to alter the dimensionality $n$ of input features to accommodate the matrix dimension $m$ (ensuring $n = m(m+1)/2$). The utilization of a linear mapping is the most direct and efficient means for this purpose.

Reviewer comment (Theory):
What is the expressive power of SPD-GNN?

Author response:
To avert unnecessary misconceptions, it is imperative to elucidate that, within the context of this manuscript, "expressiveness" pertains explicitly to the representation in geometric embedding and does not delve into an exploration of the expressive power of GNNs in addressing the graph isomorphism problem. Although the latter serves as the default interpretation of expressive power within the GNN context.

Reviewer comment (Novelty):
Many of the contributions in this work are, in fact, already present in the literature, but the discussion regarding prior work is insufficient. The feature transformation with Stiefel matrices and the non-linear activation with the "ReLU-like" layer have already been proposed in [3]. However, at the point of defining these layers, the article does not clearly cite [3] as a reference but rather kind of presents them as original contributions. Additionally, the Riemannian/Cholesky approach is directly borrowed from [2], again without clear referencing when introduced. Moreover, there are significant interactions between the approach proposed here and that of [1], which is only discussed in the appendix. It is crucial to compare with [1] in the experiments and position this method in relation to it, as both seek GNNs where nodes are embedded in the space of SPD matrices. Consequently, it becomes challenging to precisely distinguish what constitutes contributions or ideas borrowed from other articles.

Author response:
Thank you for providing insightful comments and pointing out areas for improvement in our manuscript. While it is true that some aspects of our work share common ground with prior research, we contend that our study brings a unique contribution through generalizing the fundamental components of GNNs onto SPD manifold for learning node embeddings. Nevertheless, have carefully addressed each of your concerns in the revised manuscript.
- **Feature Transformation**:
The feature transformation utilizing Stiefiel bilinear mapping [3] is indeed a classical methodology, as described in our related work. However, it is crucial to note that, within the domain of graph neural networks and graph embedding [1,4], Isometry maps for SPD matrices have been the prevailing methodology. Our contribution lies in introducing dimensionality transformations atop Isometry maps for crafting our feature transformation layer. We have conducted a comprehensive ablation study, including relevant comparisons, to substantiate the superiority of our approach over IsometryQR. We appreciate your suggestion to enhance references in the subsection on feature transformation, particularly by incorporating [3].
- **Non-linearity**:
Regarding non-linearity and the choice of elevating eigenvalues, our contribution lies in propounding a non-linear methodology grounded in the log-Cholesky metric. In contrast to methods based on eigenvalue decomposition [3], our approach demonstrates efficiency by seamlessly integrating into the neighborhood aggregation layer (Eq.17) within the SPDGNN framework. This integration avoids the computational overhead associated with standalone eigenvalue decomposition.

- **Riemannian/Cholesky Approach**:
We respectfully disagree with the interpretation that our Riemannian/Cholesky approach is borrowed from [2]. The Log-Cholesky metric, as a Riemannian metric on the SPD manifold, has found widespread application in various domains. [2] focuses on a distinct task related to Skeleton-Based Interaction Recognition and employs Cholesky space differently. In contrast, our methodology extends the operations of GNN onto the SPD manifold based on the log-Cholesky metric. Additionally, we have duly cited [2] in the related work pertaining to SPD neural networks.

- **Comparison with [1]**:
We have addressed your suggestion to compare our method with [1] in the experiments. In the revised version, we have included SPD4GCN [1] as a baseline and reported the results of performance and efficiency. 

|          | **Method** | **Disease** | **Airport** | **PubMed** | **Cora**   |
|----------|-------------|-------------|-------------|------------|------------|
| **Euclidean** | GCN   | 69.7$\pm$0.4 | 81.4$\pm$0.6 | 78.1$\pm$0.2 | 81.3$\pm$0.3 |
|              | GAT   | 70.4$\pm$0.4 | 81.5$\pm$0.3 | 79.0$\pm$0.3 | 83.0$\pm$0.7 |
|              | SAGE  | 69.1$\pm$0.6 | 82.1$\pm$0.5 | 77.4$\pm$2.2 | 77.9$\pm$2.4 |
|              | SGC   | 69.5$\pm$0.2 | 80.6$\pm$0.1 | 78.9$\pm$0.0 | 81.0$\pm$0.1 |
| **Hyperbolic** | HGCN  | 82.8$\pm$0.8 | 90.6$\pm$0.2 | 78.4$\pm$0.4 | 81.3$\pm$0.6 |
|               | HAT   | 83.6$\pm$0.9 |      --      | 78.6$\pm$0.5 | 83.1$\pm$0.6 |
|               | LGCN  | 84.4$\pm$0.8 | 90.9$\pm$1.7 | 78.6$\pm$0.7 | **83.3**$\pm$**0.7** |
|               | HYPONET | 96.0$\pm$1.0 | 90.9$\pm$1.4 | 78.0$\pm$1.0 | 80.2$\pm$1.3 |
|   **SPD**            | SPD4GCN | 91.1$\pm$3.5 | 65.8$\pm$3.4 | 78.1$\pm$0.6 | 80.2$\pm$1.4 |
|               | SPDGNN | **96.9**$\pm$**0.9** | **94.9**$\pm$**1.3** | **79.3**$\pm$**0.7** | 80.5$\pm$3.2 |

We can observe that the performance of our SPDGNN surpasses that of [1] significantly, owing to our more comprehensive architecture.

|**Method** | **Disease**| | **Airport**| | **PubMed** | |**Cora**  | |
|----------------|-------|-------|-------|-------|-------|-------|-------|-------|
|                | TR(s) | IN(s) | TR(s) | IN(s) | TR(s) | IN(s) | TR(s) | IN(s) |
| **HGCN**       | 0.011 | 0.010 | 0.014 | 0.005 | 0.018 | 0.006 | 0.011 | 0.007 |
| **SPD4GCN**    | 1.068 | 1.144 | 3.115 | 3.265 | 18.002| 18.479| 2.779 | 2.702 |
| **SPDGNN**     | 0.091 | 0.061 | 0.086 | 0.035 | 0.163 | 0.073 | 0.074 | 0.026 |

We can observe a significant enhancement in the efficiency of SPDGNN compared to [1], attributed to our metric selection and component optimizations as discussed in Section 2. While current SPD methods incur a certain increase in time overhead compared to hyperbolic methods due to various factors, including Cholesky decomposition, Riemannian gradient descent, and hyperplane classifier, the further exploration of its efficiency is warranted by the rich geometric properties.

Reviewer comment (Theory):
One crucial point that doesn't seem to be discussed is the algorithmic complexity compared to standard approaches. From a memory perspective, the approach seems already highly expensive since each node is represented as a matrix with roughly \( n^2 \) parameters. Furthermore, in terms of algorithmic complexity, just the aggregation layer requires performing approximately \( m \) Cholesky decompositions and computing an outer product with each of the representations (equation 12). So I doubt that this architecture is really applicable.

Author response:
We appreciate your insightful comments. In response to your comments, we have made the following clarifications and adjustments: 

- **Memory Perspective**:
The utilization of Symmetric Positive Definite (SPD) matrices implies a symmetric matrix, wherein the lower triangular elements of an m×m matrix can be elegantly unfolded into a vector of dimension m(m+1)/2. In our experimental configuration, a 5×5 SPD matrix is mapped to a 15-dimensional representation following conventional methodologies. As a result, the matrix representation of nodes incurs no additional memory overhead.

- **Algorithmic Complexity**:
In terms of algorithmic complexity, as detailed in Section 2 (Rationale for the Choice of the Log-Cholesky Metric), it is highlighted that the computation of Riemannian exponential and logarithmic maps, as employed in [1], necessitates the evaluation of an infinite series, incurring substantial computational costs. In contrast, our adoption of the log-Cholesky metric effectively bypasses these intricacies, although it introduces the computational overhead of Cholesky decomposition.

To empirically assess the efficiency of our proposed SPDGNN relative to SPD4GCN, we conducted a comparison of training and inference times for each epoch. The results, as provided in Q1, unequivocally demonstrate that SPDGNN significantly outperforms SPD4GCN. This observation serves as a clear indication of the superior efficiency of our approach in both the training and inference phases, reinforcing its practical advantages in terms of computational performance.

Reviewer comment (Writing):
I find that the article is generally quite confusing and not very clear. There are numerous and often ambiguous notations (for example, the notations for the Cholesky map and its inverse are almost identical), making it challenging to read. Additionally, the writing is quite heavy, with many vague and non-rigorous statements that don't convey a clear meaning. Here are a few examples:  
- "many complex graph data exhibit a profound non-Euclidean potential for analysis" (what is a "profound non-Euclidean potential"?)  
- "the SDP manifold [...] captures the hierarchical structure of datasets in hyperbolic subspaces while retaining Euclidean characteristics."  
- "In typical GNNs, a fundamental assumption is implicitly linked to linear classifiers, as they heavily rely on the Euclidean geometry of \( \mathbb{R}^n \)" (this is not true; you can have Euclidean geometry but use non-linear classifiers).

Author response:
We are sorry for causing some unnecessary difficulty for readers. We have carefully polished our draft. The notations for the Cholesky map and its inverse are followed the symbols as defined in the original article [5]. we will address the specific vague and non-rigorous statements as follows:

- many complex graph data exhibit a profound non-Euclidean potential for analysis $\rightarrow$   many complex graph datasets manifest characteristics that surpass the purview of conventional Euclidean analysis. 

- the SDP manifold [...] captures the hierarchical structure of datasets in hyperbolic subspaces while retaining Euclidean characteristics.  $\rightarrow$  Embedding graphs into the SPD manifold offers significant advantages, as it allows for the concurrent modeling of hierarchical structures in hyperbolic subspaces and grid structures in Euclidean subspaces.

- In typical GNNs, a fundamental assumption is implicitly linked to linear classifiers, as they heavily rely on the Euclidean geometry $\rightarrow$ In conventional GNNs, the routine utilization of linear classifiers critically hinges on the presupposition that data adheres to the principles of Euclidean geometry.  (The previous expression does cause ambiguity. What we seek to convey is that the default deployment of a linear classifier relies on the foundational assumption of Euclidean geometry. In the absence of the Euclidean geometric structure, the linear classifier lacks cogency, prompting our adoption of a geometric hyperplane classifier. )

Reviewer comment (Reproducibility):
The experimental setting in the article appears to lack clarity. The reported results include variances, but it's unclear on how many train/test splits these are based, neither if there is any cross-validation. Additionally, it's not specified whether the competing models were retrained for these experiments. If they were retrained, there is no information on how the retraining was conducted. Alternatively, it's unclear whether the performances were directly taken from the original articles. More transparency and details regarding the experimental setup and data handling would be beneficial for readers trying to understand and reproduce the results.

Author response:
Thanks for your valuable suggestions. We have added the information of experimental settings in the revised version. 

- **Data splits.** Disease dataset into 30\%, 10\%, and 60\%, and the nodes in Airport dataset into 70\%, 15\%, and 15\%. For Cora and PubMed datasets, we used 20 labeled examples per class for training. The above splits are the same as those used in [6]. 

- **Reported resutls.** The presented averages and standard deviations were derived from 10 independent runs. 

- **Performance reproduction.** To ensure fairness, we reproduce the outcomes of the first two categories as reported in [7]. Meanwhile, the results of SPD4GCN (https://github.com/andyweizhao/SPD4GNNs) and our SPDGNN are obtained using the consistent experimental settings and data split, aligning with the details in [6,7].

Reviewer comment (Presentation):
The Figure 2 is not so clear; what is \( V \) on the Figure? I assume that the manifold is in green while the hyperplane is in dark grey, but it is very hard to see this “hyperplane” and to interpret it.

Author response:
We are sorry for causing difficult for readers. We have revised the annotation and caption of Fig.2.  A 3D example of the SPD hyperplane on $\mathcal{S}^2_+$. The gray convex conical point cloud delineates the $\mathcal{S}^2_+$ space. The green points, sampled within $\mathcal{S}^2_+$, outline the SPD hyperplane. Notably, this hyperplane is orthogonal to the red normal vector $\mathbf{V}  \in T_\mathbf{P} \mathcal{S}^2_+$ and passes through the point $\mathbf{P} \in \mathcal{S}^2_+$.

Reviewer comment (Presentation):
The exposition of this paper can be significantly improved. I feel a significant lack of overall quality in the structure and messaging of this paper. The abstract is particularly too verbose and unclear. To this aid, Figures 2 and 3 could be annotated and captioned more clearly to convey the message of the experiment.

Author response:
We regret any inconvenience caused to the readers and have meticulously refined our manuscript, particularly focusing on the logic and expression in the abstract, as outlined in our General Response. Additionally, we appreciate your suggestions regarding the clarity of Figures 2 and 3. 
- In Figure 2, we present a 3D illustration of the SPD hyperplane, parameterized by the normal vector $\mathbf{V}$ and a bias point $\mathbf{P}$. The gray convex conical point cloud outlines the SPD space, while the green points, sampled within SPD space, delineate the SPD hyperplane. 
- In Figure 3, we present a comparative analysis of 3D illustrations, depicting hyperplanes and node embeddings learned through Euclidean and SPD MLR, respectively. Unlike the flat Euclidean hyperplanes, the SPD hyperplanes encompass both flat and curved hyperplanes, corresponding to the geometric structures of Euclidean and hyperbolic spaces, respectively. These comprehensive hyperplanes effectively discriminate node embeddings from distinct catalogs (colors), demonstrating the efficacy of SPD MLR in concurrently incorporating both Euclidean and hyperbolic geometries for classification.

Reviewer comment (Experiments):
Some important baselines appear to be lacking like Zhao 2023 and Lopez et al 2021. I am especially critical of the lack of comparison with Zhao et.al 2023 which seems to propose an identical formulation (i.e. GNNs using SPD manifold - but the specific components are different to this paper).

Author response:
Thank you for your valuable suggestions. Considering the scope of [1] does not align with the domain of graph neural networks, while [2] represents a direct extension for it, we exclusively choose SPD4GCN [2] as our comparative method. Noteworthy distinctions between our approach and [2] include: 

- **Metric**: Our proposed operations are grounded in the Log-Cholesky metric, whereas [2] relies on the Gyrocalculus from [1]. 

- **Architecture**: In addition to the fundamental neighborhood aggregation layer in GNNs, we introduce dimension-varying linear transformation and SPD MLR layers, setting our approach apart from [2]. 

We have meticulously reproduced the experimental results of [2] using their source code, adhering to consistent experimental settings and data partitioning akin to other baselines. Due to variations in experimental settings, there are certain discrepancies compared to the original results; however, the overall trends remain consistent, particularly in the case of suboptimal results observed on the Airport dataset. These results have been integrated into the revised version of the manuscript. 

|          | **Method** | **Disease** | **Airport** | **PubMed** | **Cora**   |
|----------|-------------|-------------|-------------|------------|------------|
| **Euclidean** | GCN   | 69.7$\pm$0.4 | 81.4$\pm$0.6 | 78.1$\pm$0.2 | 81.3$\pm$0.3 |
|              | GAT   | 70.4$\pm$0.4 | 81.5$\pm$0.3 | 79.0$\pm$0.3 | 83.0$\pm$0.7 |
|              | SAGE  | 69.1$\pm$0.6 | 82.1$\pm$0.5 | 77.4$\pm$2.2 | 77.9$\pm$2.4 |
|              | SGC   | 69.5$\pm$0.2 | 80.6$\pm$0.1 | 78.9$\pm$0.0 | 81.0$\pm$0.1 |
| **Hyperbolic** | HGCN  | 82.8$\pm$0.8 | 90.6$\pm$0.2 | 78.4$\pm$0.4 | 81.3$\pm$0.6 |
|               | HAT   | 83.6$\pm$0.9 |      --      | 78.6$\pm$0.5 | 83.1$\pm$0.6 |
|               | LGCN  | 84.4$\pm$0.8 | 90.9$\pm$1.7 | 78.6$\pm$0.7 | **83.3**$\pm$**0.7** |
|               | HYPONET | 96.0$\pm$1.0 | 90.9$\pm$1.4 | 78.0$\pm$1.0 | 80.2$\pm$1.3 |
|   **SPD**            | SPD4GCN | 91.1$\pm$3.5 | 65.8$\pm$3.4 | 78.1$\pm$0.6 | 80.2$\pm$1.4 |
|               | SPDGNN | **96.9**$\pm$**0.9** | **94.9**$\pm$**1.3** | **79.3**$\pm$**0.7** | 80.5$\pm$3.2 |

We can observe that the performance of our SPDGNN surpasses that of [2] significantly, owing to our more comprehensive architecture.

Reviewer comment (Experiments):
I miss any comparisons on runtime or complexity with previous methods. Again, Zhao et al 2023 seems significant baseline to compare with and report.

Author response:
Thank you for your valuable suggestions. Considering the scope of [1] does not align with the domain of graph neural networks, while [2] represents a direct extension for it, we exclusively choose SPD4GCN [2] as our comparative method. Noteworthy distinctions between our approach and [2] include: 

- **Metric**: Our proposed operations are grounded in the Log-Cholesky metric, whereas [2] relies on the Gyrocalculus from [1]. 

- **Architecture**: In addition to the fundamental neighborhood aggregation layer in GNNs, we introduce dimension-varying linear transformation and SPD MLR layers, setting our approach apart from [2]. 

We have meticulously reproduced the experimental results of [2] using their source code, adhering to consistent experimental settings and data partitioning akin to other baselines. Due to variations in experimental settings, there are certain discrepancies compared to the original results; however, the overall trends remain consistent, particularly in the case of suboptimal results observed on the Airport dataset. These results have been integrated into the revised version of the manuscript. 

|          | **Method** | **Disease** | **Airport** | **PubMed** | **Cora**   |
|----------|-------------|-------------|-------------|------------|------------|
| **Euclidean** | GCN   | 69.7$\pm$0.4 | 81.4$\pm$0.6 | 78.1$\pm$0.2 | 81.3$\pm$0.3 |
|              | GAT   | 70.4$\pm$0.4 | 81.5$\pm$0.3 | 79.0$\pm$0.3 | 83.0$\pm$0.7 |
|              | SAGE  | 69.1$\pm$0.6 | 82.1$\pm$0.5 | 77.4$\pm$2.2 | 77.9$\pm$2.4 |
|              | SGC   | 69.5$\pm$0.2 | 80.6$\pm$0.1 | 78.9$\pm$0.0 | 81.0$\pm$0.1 |
| **Hyperbolic** | HGCN  | 82.8$\pm$0.8 | 90.6$\pm$0.2 | 78.4$\pm$0.4 | 81.3$\pm$0.6 |
|               | HAT   | 83.6$\pm$0.9 |      --      | 78.6$\pm$0.5 | 83.1$\pm$0.6 |
|               | LGCN  | 84.4$\pm$0.8 | 90.9$\pm$1.7 | 78.6$\pm$0.7 | **83.3**$\pm$**0.7** |
|               | HYPONET | 96.0$\pm$1.0 | 90.9$\pm$1.4 | 78.0$\pm$1.0 | 80.2$\pm$1.3 |
|   **SPD**            | SPD4GCN | 91.1$\pm$3.5 | 65.8$\pm$3.4 | 78.1$\pm$0.6 | 80.2$\pm$1.4 |
|               | SPDGNN | **96.9**$\pm$**0.9** | **94.9**$\pm$**1.3** | **79.3**$\pm$**0.7** | 80.5$\pm$3.2 |

We can observe that the performance of our SPDGNN surpasses that of [2] significantly, owing to our more comprehensive architecture.

|**Method** | **Disease**| | **Airport**| | **PubMed** | |**Cora**  | |
|----------------|-------|-------|-------|-------|-------|-------|-------|-------|
|                | TR(s) | IN(s) | TR(s) | IN(s) | TR(s) | IN(s) | TR(s) | IN(s) |
| **HGCN**       | 0.011 | 0.010 | 0.014 | 0.005 | 0.018 | 0.006 | 0.011 | 0.007 |
| **SPD4GCN**    | 1.068 | 1.144 | 3.115 | 3.265 | 18.002| 18.479| 2.779 | 2.702 |
| **SPDGNN**     | 0.091 | 0.061 | 0.086 | 0.035 | 0.163 | 0.073 | 0.074 | 0.026 |

We can observe a significant enhancement in the efficiency of SPDGNN compared to [2], attributed to our metric selection and component optimizations as discussed in Section 2. While current SPD methods incur a certain increase in time overhead compared to hyperbolic methods due to various factors, including Cholesky decomposition, Riemannian gradient descent, and hyperplane classifier, the further exploration of its efficiency is warranted by the rich geometric properties.

[1] Lopez et al., Vector-valued distance and gyrocalculus on the space of symmetric positive definite matrices, NeurIPS 2021.

[2] Zhao et al., Modeling Graphs Beyond Hyperbolic: Graph Neural Networks in Symmetric Positive Definite Matrices, ECML PKDD 2023.