Paper: Aligning Persistent Homology with Graph Pooling (ICLR-2024)

Reviewer comment (Writing):
The notations in the paper are confusing and need to be improved.

Author response:
We have revised them accordingly.

Reviewer comment (Theory):
The authors use Forman-Ricci curvature as the filtration parameter and their reason is that it “incorporates edge weights and graph clusters to better capture the topological features of the coarsened graphs”. In fact, Forman-Ricci curvature is one type of discrete Ricci curvature models. The other discrete Ricci curvature model is Ollivier-Ricci curvature (thus it may be better to call it Forman-Ricci curvature instead of Forman curvature). Essentially, all these discrete Ricci curvature models only characterize the “local” geometric information of the graph or simplicial complex. It is not sure why the authors choose to use it as the filtration parameter.

Author response:
While it is true that curvature characterizes local geometric information, we use it solely as the filtration function. Filtration is the fundamental concept in persistent homology (PH), providing a means to capture the graph's global structures [1]. In this study, the objective is to select a fixed filtration function to compute the Wasserstein distance between persistence diagrams, serving as an evaluation metric for their topological similarities. Thus, we opt for utilizing the Forman-Ricci curvature as the filtration function, given its suitability for our scenario involving graphs with edge weights and cluster structures. Consequently, it is better equipped to capture the topological features of the coarsened graphs. Another advantage of using Forman-Ricci curvature is its computational efficiency. While alternative options like the Ollivier-Ricci curvature are also viable, they generally result in a heavier computational workload.

Reviewer comment (Writing):
The authors state that “The core of PH is the computation of filtration, which presents a challenging task due to its complexity”. What is the meaning of “computation of filtration”? Is it to design a proper filtration parameter, to construct a series of simplicial complexes from the filtration process, or to calculate the persistence of homological generator from the filtration process?

Author response:
We apologize for the lack of clarity in our presentation. In this section, we aim to convey the challenge of selecting the appropriate filtration for achieving optimal performance. We have revised this part as "The core of PH is notion of filtration, which presents a challenging task to choose the right filtration."

Reviewer comment (Writing):
Page 4, for the “sequence of nested subgraphs”, the subindex is from 0 to n, this implies that nodes are added into the filtration process one by one, which is clearly not the case. To avoid the confusion, it is better to use a different way of notations.

Author response:
Here the subindex does not have to be continuous from 0 to n. We have revised the notations to eliminate any potential ambiguity.

Reviewer comment (Theory):
Page 5, in equation (5), the operation of min and max is in terms of what? If it is for the elements of the matrix, the min(A^l) will result in a number (scale).

Author response:
It is for the elements of the matrix to get the maximum and minimum values of A, so it will result in a scalar. Equation (5) represents a regular min-max normalization process.

Reviewer comment (Theory):
Page 5, in equation (6), note that “D_1” is defined to be “ph()”, but in page 4, “ph()” is defined to be collections of “D_0”, “D_1”, etc. Further, $A^l$ is defined as the Hadamard product of $A^l$ and $D_1[1]- D_1[0]$. Here $A^l$ is a matrix and $D_1[1]- D_1[0]$ is a vector. They may have very different dimensions.

Author response:
Please note that the persistence diagram $\mathcal{D}$ is extended to match cardinality of the number of edges. Therefore, Eq. (6) represents the element-wise product of the edges in $\mathbf{A}'^{(l)}$. We apologize for the lack of clarity in our presentation. The notations of Eq. (6) have been updated to eliminate any potential ambiguity.

Reviewer comment (Theory):
While the concept is quite promising, the implementation falls short in terms of its strength. The authors have attempted to utilize persistent homology to maintain consistent topological information during the pooling process; however, the approach they propose doesn't align with their intended outcome. To achieve their objective, the integration of persistent homology should be fundamentally different.

Author response:
PH is utilized to incorporate topological information into the edge weights, which are then utilized as inputs to the GNN for obtaining node features. As a result, the utilization of PH influences the final node representations, which, in turn, impact the filtration values in the subsequent pooling layer, ultimately resulting in varied performance on downstream tasks.

Reviewer comment (Theory):
The theorem is a bit irrelevant here as the existence of a filtration function in this context does not mean much for practical purposes.

Author response:
The existence of such filtration function implies that theoretically TIP has stronger expressive power than conventional dense pooling methods (such as DiffPool, MinCutPool, and DMoNPool) when it comes to distinguishing non-isomorphic graphs. It does not generally translate into better predictive performance, though. Expressive power serves as an indicator of the method's ability in graph classification tasks [1]. By employing a learnable filtration function to enhance flexibility, the theorem validates the potential capability of our method [2][3]. Additionally, we apply our mechanism to a collection of graph pooling methods and observe consistent and substantial performance gain over several popular datasets in graph classification tasks, which is consistent with our theorem.

Reviewer comment (Experiments):
For classification results, it would provide more valuable insights into the performance of the proposed model if the authors compared it with State-of-the-Art (SOTA) GNN results rather than basic models. This approach would offer a more meaningful assessment of the model's capabilities.

Author response:
We provide additional comparisons with Graph Substructure Networks (GSN) [4], which is a GNN model with a topologically-aware message passing scheme based on substructure encoding. The results are presented in Table 2 and shown below. GSN is highly related to our method TIP, as they both take the graph topology into consideration.

|             | NCI1               | PROTEINS           | IMDB-BINARY        | IMDB-MULTI         | OGBG-MOLHIV        |
|--------------|:------------------:|:------------------:|:------------------:|:-------------------:|:-------------------:|
| GSN          |   83.50 ± 2.00    |   74.59 ± 5.00    |   76.80 ± 2.00    |   52.60 ± 3.60     |   76.06 ± 1.74     |

Reviewer comment (Theory):
Use of PH is a bit problematic here. You are using sublevel filtration with a node function as far as I see. Since you are using single filtration, edge weights have no importance here. Therefore, the concepts of resampling and persistence injection, which involve varying the edge weights, may not align well with the method's underlying principles.

Author response:
PH is utilized to incorporate topological information into the edge weights, which are then utilized as inputs to the GNN for obtaining node features. As a result, the utilization of PH influences the final node representations, which, in turn, impact the filtration values in the subsequent pooling layer, ultimately resulting in varied performance on downstream tasks.

Reviewer comment (Theory):
The notion of introducing topological information into the pooling process is intriguing, but it does present a certain contradiction. The original graph may possess numerous topological features, yet the objective of graph pooling is to transition to a smaller, more compact graph. As a result, expecting the coarsened graph to retain similar topological information from the original one is not very meaningful, especially when using PH with sublevel filtration.

Author response:
There is significant empirical evidence that substructures are often intimately related to downstream tasks in chemistry, biology, and social network analysis [1]. Furthermore, cycles in graphs have been receiving increasing attention [1][2][3]. These studies highlight the significance of cycles in graphs and suggest that enhancing GNNs' structure-awareness can enhance their expressivity and usefulness for downstream tasks. Thus, our paper aims to preserve cycle structures during the pooling process. The effectiveness of graph pooling relies heavily on the coarsened graphs. Accordingly, we assume that preserving cycles in the coarsened graphs is beneficial for downstream tasks in network science and bioinformatics fields. In this paper, we utilize persistent homology to incorporate these crucial topological features into graph pooling. Experimental results validate the effectiveness of our model, confirming the validity of our assumption.

[1] Giorgos Bouritsas, Fabrizio Frasca, Stefanos Zafeiriou, and Michael M Bronstein. Improving graph
neural network expressivity via subgraph isomorphism counting. IEEE Transactions on Pattern
Analysis and Machine Intelligence, 45(1):657–668, 2022.

[2] Jiaxuan You, Jonathan M Gomes-Selman, Rex Ying, and Jure Leskovec. Identity-aware graph
neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp.
10737–10745, 2021.

[3] Yinan Huang, Xingang Peng, Jianzhu Ma, and Muhan Zhang. Boosting the cycle counting power
of graph neural networks with iˆ2-gnns. In The Eleventh International Conference on Learning
Representations, 2022.

Reviewer comment (Experiments):
One potential solution in pursuit of this goal involves using TDA to guide the graph pooling process. However, this approach often demands a completely different filtration type, such as the Vietoris-Rips (or power) filtration, which can be computationally expensive. Balancing the desire for topological preservation with the computational cost remains a significant challenge in this context.

Author response:
Thanks for your quick comments, and I think maybe we misunderstood your concern before. Here we provide detailed explanations to address your concern.

Our motivation primarily stems from the observed deficiency in current graph pooling methods, specifically their inadequate preservation of topological structures, and we can amend this problem through PH. We are not utilizing PH to directly represent the graph's topology, but rather employing it as a guiding principle for the pooling process. Such guidance makes sense as we observe there is an alignment between PH and graph pooling process (Fig. 1(b)). While PH itself does not inherently represent the graph's topology, it introduces the crucial concept of "persistence" to characterize the significance of edges in a topological context. Through persistence injection, akin to a pooling process, edges with low persistence are filtered out, aligning with the idea of graph coarsening. 

Moreover, the influence of persistence on message passing within GNNs resembles an attention-like mechanism. This enhances the GNN's focus on meaningful topological structures, contributing to the increased likelihood of preserving the original graph's topology in the coarsened graph.

Our theoretical and experimental results corroborate the rationale behind our approach. The integration of PH significantly augments the expressive power of graph pooling methods to distinguish isomorphic graphs. Figures 3 and 4 underscore that our method indeed enhances topology preservation compared to traditional pooling methods that fail to retain any topology. Notably, in graph classification tasks across multiple datasets, our method consistently demonstrates substantial improvements.

Reviewer comment (Experiments):
In the experiment part, since TU datasets are suffering from high standard deviation, I recommend adding more popular benchmarks such as ZINC and the OGB datasets. In addition, the running time should be reported on different benchmarks.

Author response:
We provide additional experiments on OGBG-MOLHIV from the OGB datasets. The comparison results are shown in Table 2 and the response to Reviewer XtaZ, where our method can significantly improve the baseline pooling methods. Additionally, to compare the running time, we perform experiments on a platform with an AMD EPYC 7542 CPU and a single NVIDIA 3090 GPU. We utilize the default settings from the graph classification experiments and record the running time of 50 epoches training on different datasets.  The running time (in seconds) is reported in Appendix E.3 and shown below, which reveals that the incorporation of TIP does not impose a substantial computational burden. 

|             | NCI1   | PROTEINS | ENZYMES |
|--------------|:------:|:--------:|:-------:|
| DiffPool     | 209.48 |  56.55   |  30.61  |
| DiffPool-TIP | 365.98 | 113.65   |  61.02  |
| MinCutPool   | 145.99 |  38.22   |  27.34  |
| MinCutPool-TIP | 341.93 | 102.31   |  55.54  |
| DMoNPool     | 124.89 |  35.07   |  19.35  |
| DMoNPool-TIP | 351.42 | 101.86   |  57.91  |

Reviewer comment (Theory):
[Theory] Theorem 1 states that distinguishing graphs based on 1-dimensional persistence diagrams is more expressive than 1-WL. **I believe this is false**. For instance, let $G$ and $G'$ be two graphs comprising one and two isolated nodes, respectively. The 1-dimensional diagrams for these graphs are identical (and empty) for any filtration function. However, 1-WL distinguishes graphs with different numbers of nodes. In addition, since the paper leverages filtration functions on node features, for graphs that share the same single feature vector (or same color), 1-dim diagrams cannot go beyond Betti-1, which is clearly less expressive than 1-WL.

Author response:
In the resampling process in Section 4.2, we forgot to include some implementation details. In Eq. (5), we add **self-loops** to the graph for the diagonal part of $\mathbf{A}^{\prime (l)}$ . For the off-diagonal part of $\mathbf{A}^{\prime (l)}$, we only resample the upper triangular matrix to ensure symmetry. As mentioned at the end of Section 3, the self-loop edges are assigned a dummy tuple value $(f(e), f(e))$. Therefore, for graphs $G_1$ and $G_2$ comprising one and two isolated nodes respectively, their corresponding 1-dim persistence diagrams contain one and two data points respectively, making them still distinguishable. While there are cases where 1-dim persistence diagram and 1-WL have the same expressive power, there are also cases where 1-dim persistence diagram can distinguish while 1-WL cannot, e.g. $G_1$ is a graph containing two non-overlapping triangles (2 cycles) and $G_2$ is a hexagon (1 cycle). 
    We would like to thank Reviewer XtaZ for pointing out the deficiencies in our proof. To address this, we have made revisions to Section 4.2 to include additional implementation details and Theorem 1 in Section 4.3.

Reviewer comment (Theory):
[Methodology] Since 1-dim persistence diagrams have limited persistence (the death times are identical), it seems the proposal mainly relies on cycle-preserving pooling (do the authors agree?). At a conceptual level, I am not convinced that this design choice is a desideratum for general-purpose graph pooling and/or graph classification tasks. Overall, I found the motivation for adopting only 1-dimensional PD weak.

Author response:
Our primary motivation is to preserve cycle structures during the pooling process. This motivation stems from the significant empirical evidence that substructures, such as cycles, are often intimately related to downstream tasks in fields like chemistry, biology, and social network analysis. Additionally, there has been an increasing focus on cycles in graphs, as evidenced by several notable works [1, 2, 3]. These works emphasize the significance of cycles in graphs and highlight the potential for structure-aware GNNs to enhance expressivity and improve performance in downstream tasks.

   The effectiveness of graph pooling heavily relies on the quality of the coarsened graphs. Therefore, we assume that preserving cycles in the coarsened graphs can be beneficial for network science and bioinformatics tasks. In this paper, we leverage persistent homology (PH) to incorporate these essential topological features into graph pooling. Experimental results confirm the efficacy of our model, validating our assumption. Regarding the use of 1-dim persistence diagrams exclusively, our rationale is that the majority of graphs are connected with only one connected component. Additionally, we performed experiments comparing the use of 1-dim persistence diagrams and found no improvements in our model. In response to this, we have enhanced the description of the motivation for retaining cycle structures in Section 4.2.

Reviewer comment (Experiments):
[Experiments] Due to the lack of a strong/principled motivation, I would expect to see more empirical evidence to support the proposed method. The paper only considers TU Datasets, and therefore does not exploit recent efforts (e.g., OGB) to strengthen the evaluation of GNNs. Moreover, except for the Enzymes dataset, the empirical gains do not look significant (with differences in mean accuracies usually within one std, overall) compared to pooling-free approaches. Lastly, the ablation study should consider other datasets and pooling layers.

Author response:
We conduct additional experiments on OGBG-MOLHIV from the OGB datasets. The comparison results, presented in Table 2 and shown below, demonstrate that our method significantly enhances the performance of the baseline pooling methods.  It's worth noting that our objective is to enhance graph pooling methods through the alignment of PH with graph pooling. Furthermore, we include additional results on the ablation study in Appendix E.2, where we add ablation study of DiffPool-TIP, MinCutPool-TIP and DMoNPool-TIP on a wider range of datasets.

|             | OGBG-MOLHIV      |
|--------------|-----------------|
| GCN          | 75.04 ± 0.84    |
| GIN          | 76.03 ± 0.84    |
| GraphSage    | 76.18 ± 1.27    |
| TOGL         | 77.21 ± 1.33    |
| Graclus      | 76.81 ± 0.70    |
| TopK         | 76.28 ± 0.67    |
| DiffPool     | 76.60 ± 1.04    |
| DiffPool-TIP | 77.75 ± 1.18    |
| MinCutPool   | 75.60 ± 0.54    |
| MinCutPool-TIP | 77.18 ± 0.83   |
| DMoNPool     | 76.30 ± 1.34    |
| DMoNPool-TIP | 76.70 ± 0.62    |

Reviewer comment (Writing):
Could you elaborate on how PH coarsens graphs (as in Figure 1)?

Author response:
As outlined in Section 3, PH leads to a sequence of nested subgraphs of the form $G = G_n  \supseteq \ldots G_{k} \ldots \supseteq G_0 = \emptyset$, where $G_k = (V_k, E_k)$ is a subgraph of $G$. These nested subgraphs can be interpreted as a form of coarsening, gradually filtering out specific nodes or edges through PH, as visualized in Figure 1(a). Here, what we want to express is that PH and graph pooling both seek to coarsen/sparsify a given graph in a hierarchical fashion: while PH gradually derives persistent sub-topology by adjusting the filtering parameter, GP obtains a sub-graph by performing a more aggressive cut-off.

Reviewer comment (Presentation):
Do the persistence ratios in Fig. 1(b) include 0-dim persistence diagrams? If so, it seems misleading to use Fig(b) as a motivation and only exploit 1-dimensional PH information for pooling.

Author response:
No, we exclusively utilize 1-dim persistence diagrams.

Reviewer comment (Presentation):
I believe it would be helpful to see examples of (1-dim) persistence diagrams before and after applying TIP. For instance, I am curious to see the diagrams for the ring and grid2d datasets.

Author response:
We have added a new section, labeled Appendix E.1, to present the persistence diagrams of the ring and grid2d datasets before and after applying TIP. This addition aims to provide a visual representation of the impact of TIP on the datasets.

Reviewer comment (Evaluation):
In section 4, the paper uses vectorized 1-dim diagrams to measure topological similarity. However, Fig. 5 shows learning curves using the Wasserstein distance. I was confused. The same happens with the filtration functions --- Forman curvature vs. learnable filtration function --- it seems the former is only used for evaluation purposes (table 1). These choices should be clarified.

Author response:
In Fig. 5, our objective is to demonstrate the topological similarity between the coarsened graph and the original graph. This similarity is measured using the Wasserstein distance.  To ensure the validity of the evaluation metric, it is important to have a fixed filtration function rather than a learnable one. In this study, we utilize the Forman curvature on each edge of the graph as the fixed filtration. The Forman curvature incorporates edge weights and graph clusters, enabling a more accurate capture of the topological features of the coarsened graphs. Further clarification on this matter has been revised in Section 5.2.

Reviewer comment (Writing):
What do you mean by persistent sub-topology?

Author response:
It means substructures that have meaningful topology, such as cycles.

Reviewer comment (Presentation):
The caption of Fig. (6) says: "a pair of isomorphic graphs that cannot distinguished by 1-WL but can be distinguished by TIP". It is not clear what the paper wants to convey here (including the plot and the caption).

Author response:
In this context, we aim to illustrate that there are certain scenarios where TIP surpasses 1-WL, thereby demonstrating its superior expressive power in comparison to conventional dense pooling methods such as DiffPool, MinCutPool, and DMoNPool.