Paper: How well does Persistent Homology generalize on graphs? (ICLR-2024)

Reviewer comment (Novelty):
While the question addressed in this paper is important, its contribution appears to be relatively modest, especially when compared to the recent work by Morris et al. (2023). In that study, the authors delve into the predictive power of Graph Neural Networks (GNNs) through the lens of VC-dimension, and the authors of the current paper adapt their methods to the context of Persistent Homology. While the theoretical results are intriguing, their practical relevance in the machine learning domain remains questionable.

Author response:
**Our focus and analysis are orthogonal to Morris et al. (2023).** Thanks for the chance to clarify the relationship between our Proposition 2 and the result by Morris et al. (in Proposition 1). We highlight that we only use the work by Morris to show an analogous result --- we don't build upon it or use it to prove anything. Also, our work reports a PAC-Bayesian analysis, which is unrelated to Morris et al. We added a plot to depict the dependence between the Lemmas used to prove our Proposition 2. We hope this clarifies the orthogonality of our work wrt to Morris'.

**Our theoretical results are important in their own right.** Understanding the generalization behavior of ML algorithms, i.e., analyzing their ability to predict well once trained,  is one of the main goals of machine learning in general and statistical machine learning in particular. From that perspective, our theoretical results are already significant. 

**Practical implications of our work.** We introduce a new algorithm by regularizing PersLay (please see the new Section 3.4). The regularization term directly exploits the dependence of parameters established in our bounds. In our experiments with several real-world experiments, we show that this method generally improves performance over the standard (i.e., unregularized) PersLay algorithm across many real datasets, often significantly. Our empirical results strongly substantiate the practical benefits of our theoretical analysis that leads to a new algorithm with better performance.

Reviewer comment (Theory):
The provided bounds are often complex and abstract, making them challenging to compute in real-world applications. Given the extensive, nine-page proof section, which requires a thorough review, the paper might be better suited for a journal focused on statistics or applied topology rather than an ML venue. The heavy theoretical content, with limited applicability, calls into question the practical utility of the paper's findings in the ML community.

Author response:
**Our theoretical results are important in their own right.** Understanding the generalization behavior of ML algorithms, i.e., analyzing their ability to predict well once trained,  is one of the main goals of machine learning in general and statistical machine learning in particular. From that perspective, our theoretical results are already significant. 

**Practical implications of our work.** We introduce a new algorithm by regularizing PersLay (please see the new Section 3.4). The regularization term directly exploits the dependence of parameters established in our bounds. In our experiments with several real-world experiments, we show that this method generally improves performance over the standard (i.e., unregularized) PersLay algorithm across many real datasets, often significantly. Our empirical results strongly substantiate the practical benefits of our theoretical analysis that leads to a new algorithm with better performance.

**"Theoretical issues in deep learning" is an explicitly identified relevant topic for ICLR (https://iclr.cc).** Although there has been a surge in PH-based methods for machine learning, our understanding of their theoretical properties is vastly under-explored, especially for graph-structured data. Therefore, we believe that our work represents an important contribution to the topological deep learning community.

Reviewer comment (Evaluation):
In Figure 4, while the width is increasing, empirical values of the generalization gap stay very low. How do we explain this?

Author response:
First, we note that the empirical gap is a function of many quantities, including those other than width. More importantly, we have observed strong Pearson correlation coefficients between the empirical and theoretical gap across datasets, validating our analysis.

Reviewer comment (Evaluation):
Could you provide insights into the relationship between the generalization gap and the training set size?

Author response:
The generalization gap decreases with, i.e., is inversely proportional to, the square root of the training set size. Theorem 2 reports this relationship where $m$ is the training set size.

Reviewer comment (Theory):
(Q2) Section 3.1: It is not very clear how Section 3.1 fits into the whole story of your paper, in particular since you write that expressivity and generalization can be at odds with each other, but you do not elaborate further. This issue is pronounced more in (the very nice) Figure 2, which is missing Proposition 1 and 2, and Lemma 2 and 3. What is the reason that Morris et al., 2023 and Rieck, 2023 that you rely on in Section 3.1, are not discussed in the Related work? Why do the experiments not validate these theoretical results too?

Author response:
> "Section 3.1: It is not very clear how Section 3.1 fits into the whole story of your paper, in particular since you write that expressivity and generalization can be at odds with each other, but you do not elaborate further."

**Expressivity vs. generalization, and the relevance of Section 3.1** Thank you for the opportunity to clarify this.

Indeed, enhancing expressivity typically comes at the expense of  generalization. Morris et al. (2023) established such a result for Graph Neural Networks, showing that the  VC-dimension of GNNs with $L$ layers is lower bounded by the maximal number of graphs that can be distinguished by 1-WL (Weisfeiler-Leman test for isomorphism).  High VC-dimension directly translates to poor generalization, whereas by definition   greater the number of graphs that can be distinguished greater the expressivity. Thus, Morris et al. showed the tension between expressivity and generalization in the context of message passing GNNs that are at most as expressive as the 1-WL test.

However, these results do not apply to PH. In section 3.1, we fill this gap by establishing the conflict between expressivity and generalization for any PH method, building on the results about expressivity of PH by another influential paper due to Rieck (2023). Importantly, we connect the generalization of PH via VC-dimension to expressivity in terms of the WL hierarchy (i.e., 1-WL and more expressive higher order WL tests). Our results in Section 3.1 hold particular significance since (topological descriptors obtained from) PH methods are increasingly being used to augment the capabilities of message-passing GNNs, so being able to analyze one in terms of the other is a step forward.

The result in Section 3.1 provides a lower bound on the generalization for any PH method, and thus complements the lower bound on expressivity by Rieck (2023) - both in terms of the WL/Folklore WL (FWL) hierarchy. Albeit important, these bounds do not take into account the underlying data distributions. Therefore, in the rest of the paper, we develop a data-dependent PAC-bayes bound on generalization of PersLay, a flexible and widely used PH method.

Importantly, unlike the lower VC-dim bounds in Section 3.1 that are hard to estimate, we provide an upper bound on the generalization error in terms of more amenable parameters that directly lead to a new regularized PersLay method with demonstrable empirical benefits on real-world datasets.

> "This issue is pronounced more in (the very nice) Figure 2, which is missing Proposition 1 and 2, and Lemma 2 and 3."

Thanks for catching this. We have added a subplot to Figure 2, depicting the dependence between the Lemmas (2 and 3) used to prove Proposition 2.

> "What is the reason that Morris et al., 2023 and Rieck, 2023 that you rely on in Section 3.1, are not discussed in the Related work?"

Thanks for pointing this out. We have added a related works sections (Section 1.2), where we also discuss these works.

Reviewer comment (Presentation):
(Q3) Table 1: In the Discussion, you write the following: “In Table 1, the study provides a valuable resource by depicting the resulting bound dependencies on various parameters. This information is instrumental in estimating the overhead introduced by PH in the generalization performance of conventional multi-layer perceptron.” Immediately I was hoping to see some discussion in this direction, but I am not sure if the next two paragraphs are related to Table 1? Where does the $\sqrt{\ln b}$ appear, and where do we see $h \sqrt{\ln h}$? References to particular lemmas, theorems or tables can improve readability.

Author response:
> "you write the following: “In Table 1, the study provides [...] bound dependencies on various parameters. This information is instrumental in estimating the overhead introduced by PH in the generalization performance of conventional multi-layer perceptron.” Immediately I was hoping to see some discussion in this direction, but I am not sure if the next two paragraphs are related to Table 1?"

**Overhead due to PersLay.** Thanks for your bringing our attention to this. Based on your comment, we have improved the discussion regarding the discrepancy between our bound for the PersLay Classifier and the bounds for feedforward networks and GNNs. The dependence on width and spectral norms of the weights directly stems from Theorem 2 by absorbing non-related  quantities within the big-O notation. Analyzing the width dependence we conclude that the upper bound for PersLay increases from $O(\sqrt{h \ln h})$ (in feedforward networks and GNNs) to $O(h\sqrt{\ln h})$ when $q = \Theta(h)$. So, to produce a tighter generalization bound,
we recommend choosing $q = o(h)$. We have also clarified this in the revised version.

>  "Where does the lnb appear, and where do we see hlnh? References to particular lemmas, theorems or tables can improve readability."

**Regarding $\ln b$ and $h \ln h$.** Here, $b$ denotes the radius of the $\ell_2$-ball in which the points of the persistence diagrams lie. The dependence on $b$ is subsumed within the quantity $M$ that appears in Theorem 2 (please, see Lemmas 4 and 5).  The additional dependence (compared to feedforward networks and GNNs) of our bound on the maximum input norm is proportional to $\sqrt{\ln b}$, which is upper bounded by $b$. Thus, PH does not incur any overhead due to this term, and so we have removed this now from the discussion section for improved exposition. 

For $h \ln h$, again, this appears in Theorem 2. Please note that, unlike the dependence on $b$, PersLay incurs an overhead proportional to $O(\sqrt{h})$ when $q = \Theta(h)$, as clarified above.

Reviewer comment (Theory):
In general, can you provide some intuition about what makes the generalization bounds for PH different from other models, and/or what properties of PH do you use to obtain your theoretical results? Is it crucial that the input is a graph?

Author response:
> "In general, can you provide some intuition about what makes the generalization bounds for PH different from other models, and/or what properties of PH do you use to obtain your theoretical results? Is it crucial that the input is a graph?"

The main sources of difference come from the choice of the persistence diagram vectorization (Lemmas 4 and 5) and from the combination of perturbations in vectorization and linear layers (Lemma 6).

Regarding the applicability to graphs, we note that our results in Section 3.1 regarding the VC dimension applies to graph data while  the PAC-Bayes bound applies more generally to diagrams extracted from any input.

Reviewer comment (Evaluation):
Moreover, could you summarize the “key insights about the limits and power of PH methods”? What do we learn from your paper about the generalization ability of PH?

Author response:
> "Moreover, could you summarize the “key insights about the limits and power of PH methods”? What do we learn from your paper about the generalization ability of PH?"

Some **key insights from our analysis** are:
- There is an inherent tension between generalization and expressivity of any PH method in the context of graph representation learning (both can be lower bounded in terms of the WL hierarchy). In particular, enhancing expressivity leads to an increase in the VC-dimension, thereby worsening the ability to generalize (Section 3.1). 
- A flexible PH method, namely PersLay, has provably good generalization performance. In particular, the dependence of generalization on margin, sample complexity (i.e., size of the training dataset), and maximum norm across points in the persistence diagrams resembles and is comparable to that for feedforward neural networks (FNNs, Neyshabur et al., 2018) and graph neural networks (GNNs, Liao et al., 2020). 
- A key difference between generalization of PersLay and FNNs/GNNs is the dependence on the model width and the spectral norm of the model weights. Specifically, PersLay has slightly worse dependence in terms of these quantities; however, this gap can be regulated by modulating the dimensionality of the embedding of the persistence diagram before treating with the linear layers. 
- From a practical viewpoint, the theoretical dependence of generalization on parameters can be leveraged to introduce a regularized method in a structural risk minimization setup that leads to improved empirical performance on several real-world datasets.

Reviewer comment (Reproducibility):
(Q4) “We report additional results across different epochs and hyper-parameters in the supplementary material.” These results are not included?

Author response:
> "“We report additional results across different epochs and hyper-parameters in the supplementary material.” These results are not included?"

Apologies for this. We did not find significant variability in these experiments and ended up not including them.

Reviewer comment (Presentation):
(Q5) Figure 5 is not very informative. Could it be replaced by including the results for the line and triangle point transformations (and their correlations) into Figure 3 and Figure 4?

Author response:
> "Figure 5 is not very informative. Could it be replaced by including the results for the line and triangle point transformations (and their correlations) into Figure 3 and Figure 4?"

Thanks for the suggestion. We will remove Figure 5 and replace it with additional plots (like Fig. 3 and 4) for line and triangle transformations. We will update the paper with this change by the end of this discussion period.

Reviewer comment (Writing):
(Q6) Notation: The notation could be improved, what is the current logic? For example, you could e.g. use small case/capital case/Greek alphabet for graph nodes/sets/functions, and then be consistent. Often, you use the same notation for different things: e.g., S for both training set and the upper bound in Lemma 6, m for the size of training data and the maximal number of distinguishable graphs, omega for the PersLay weight function and for the hypothesis parameters, etc. Could you include a notation table? For instance, it took me quite some time to find what b is when seeing it appear at the end of Section 3. As you will see, a lot of the minor comments below would likely be resolved with a table summary of improved notation.

Author response:
> "Notation: The notation could be improved, what is the current logic? For example, you could e.g. use small case/capital case/Greek alphabet for graph nodes/sets/functions, and then be consistent. Often, you use the same notation for different things: e.g., S for both training set and the upper bound in Lemma 6, m for the size of training data and the maximal number of distinguishable graphs, omega for the PersLay weight function and for the hypothesis parameters, etc. Could you include a notation table? For instance, it took me quite some time to find what b is when seeing it appear at the end of Section 3. As you will see, a lot of the minor comments below would likely be resolved with a table summary of improved notation."

**We have revised the presentation**. We have rewritten sections of the paper for clarity. In particular, we have
- clarified the relevance of the VC-Dimension bounds in the context of generalization and expressivity of any PH method (Section 3.1);
- fixed typos and included a notation table in the Appendix;
- expanded the notation, including the abbreviation (kFWL for k Folklore WL);
- clarified which results apply to PH in general vs those that apply to PersLay (now called PersLay Classifier);
- added a related works section (Section 1.2)
- added the dependence between the different Lemmas for the VC-Dim bound (Figure 2);
- clarified how our PAC-Bayes generalization bound for PersLay differs from the corresponding bounds for graph neural networks and feedforward networks;
- refined the presentation by re-writing parts of sections 2 and 3 for clarity and precision.

**Importantly, we have accepted all your minor suggestions/comments and implemented the changes in the revised manuscript.** We really appreciate your careful reading. 

---

Many thanks for your thoughtful and incisive comments. We hope our response has elucidated several subtle points that you raised and allayed your concerns. We would greatly appreciate your stronger support for this work.

Reviewer comment (Writing):
Which norm do you use for persistence diagrams, i.e., what is |D(G)|?

Author response:
> Regarding the notation |D(G)| ---size of the diagram.

We have updated the paper to clarify this notation. Instead of using $|\mathcal{D}(G)|$ to denote the number of points in the persistence diagrams, we are now using card($\mathcal{D}(G)$) and call it the cardinality (instead of size) of the diagram $\mathcal{D}(G)$. We added the following sentence to the first paragraph of page 3:

*We denote the persistence diagram for a graph $G$ as $\mathcal{D}(G)$ and use $\text{card}(\mathcal{D}(G))$ to represent its cardinality, i.e., number of (birth time, death time) pairs in $\mathcal{D}(G)$.*

 We have also updated the notation table accordingly.

Reviewer comment (Novelty):
This paper merely investigated the generalization one vectorization tool of PD, i.e. Perslay, thus having limited contribution. Researchers who are interested in the generalization of PH on graphs may be more interested in other representations of PD, such as persistence images and deep sets [3], and in models with flexible filtrations [2].

Author response:
**Our analysis subsumes persistence images and deep sets.** We note that the vectorization methods introduced by PersLay are rather general and subsume the suggested persistence images (PI) and also DeepSet-based strategies. Thus, our analysis covers most of the vectorization schemes broadly used in the literature.

**Our work lays a strong foundation for analyzing learnable filtrations.** One way to analyze PH with learnable filtration schemes could be to get upper bounds on perturbation of outputs in terms of the filtration function parameters. This would additionally require an analysis of Wasserstein distances between persistence diagrams obtained with different parameters. We believe that for a specific class of graphs we can get modified upper bounds for perturbation with respect to filtration function parameters that would depend on Wasserstein distance of same order. This additional analysis could be readily integrated into our framework to get generalization bounds for learnable filtrations.