Paper: Understanding Graph Transformers by Generalized Propagation (ICLR-2024)

Reviewer comment (Novelty):
The equivalence between Graph Transformer and the color refinement algorithm in proposition 3.1 are proved in previous work [1].

Author response:
A: Firstly, we would argue that [1] shall be considered as a concurrent work since [1] was officially public in Aug 2023 and our submission was in Sep 2023. It is nearly impossible to track all new works on arXiv because of the tremendous amount of papers posted on arXiv. 

Second, the goal of our work is different from [1]. [1] aims to demonstrate that the proposed positional encoding together with global self-attention is endowed with stronger expressiveness compared to the 1-WL algorithm. 

In contrast, our work aims to illustrate that the adjacency function of various GNNs (e.g., MPNNs, graph transformers, etc.) is the dominant component in the expressiveness of modeling graph-structure, demonstrated with color refinement algorithms, while the feature-conditioning counterpart in self-attention (i.e., attention scores based on features similarity) is not the key factor.

Reviewer comment (Theory):
Section 3.3 looks incomplete. At end it mentions various GNN expressivity hierarchy without clarifying GPNN's connection with them.

Author response:
A: Thank you for pointing out this. We would add more bridging text mentioning the connection to various GNNs. Besides, we did indeed include several examples to show various typical GNNs (including but limited to GCNs, GATs, and graph transformers), as special cases of GPNNs. The proposed GPNN framework allows us to analyze their expressiveness by simply analyzing the adjacency function.

Reviewer comment (Presentation):
Figures are not mentioned in the maintext.

Author response:
A: We would add bridging text for them.

Reviewer comment (Theory):
The connection between CURC and GPNN seems fully intuitive. More formal connection can make me understand better.

Author response:
A: Understanding the "bottleneck" of GNNs has been extensively studied and is known to be a critical factor affecting the performance of graph models [2]. In such models, bottlenecking refers to a situation where a small number of edges or vertices significantly restrict the flow or connectivity of information propagated by GNNs between two nodes. 
The authors' proposal in [2] to use Ollivier-Ricci (OR) curvature as a means to quantify this phenomenon has been a significant advancement in understanding graph structures. 
Due to the limitation of OR, they utilize OR on the input graphs which are unweighted and undirected to study the bottleneck of MPNNs.
However, when the scope goes broader to various GNNs (e.g., MPNNs, graph rewiring, graph transformers), it is insufficient. Different GNNs behave differently given the same input graphs. 
Therefore, we propose a unified framework, GPNN, for various GNNs. Theoretically and empirically, we demonstrate that various GNNs can be viewed as generalized propagations with learned or fixed propagation matrices. Thus, we can study the bottleneck on GNNs directly via the corresponding propagation matrices instead of on the input graphs.
However, the propagation graphs are naturally weighted-directed (for example, the learned attention matrices), where the OR is not applicable.
This drives us to extend the OR to weighted-directed graphs, termed the CURC (Curvature Under Ricci Curvature), to become a valuable tool for studying the bottleneck of graph transformers.

Reviewer comment (Theory):
The definition and properties of CURC seem quite straighforward. However, therefore meaning for GNN are not quite clear to me. How to use these theories to guide the design of GNN?

Author response:
A: We would strongly argue that the definition and properties of CURC are not trivial. Even though curvatures have been utilized to analyze the bottleneck of MPNNs (Topping et al, ICML 2022). However, it is limited on analyzing the input graphs, which need to be simple graphs (i.e., undirected without self-loop) due to the property of curvature (only applicable the symmetric graphs). Considering the behaviors of various GNNs (e.g., MPNNs, graph rewiring, graph transformers, etc.) are not strictly tied to the input graphs: MPNNs and graph transformers, as examples, show different behaviors given the same input graphs. 
To better analyze various GNNs, we propose the unified framework, GPNN, which accommodates various GNNs. Furthermore, we demonstrate that the dominant component in GPNN to the expressiveness is the adjacency functions, which allow us to analyze the expressiveness of various GNNs in a unified approach, ridding of the less relevant parts.
However, the propagation graphs from adjacency functions are naturally weighted and directed (e.g., attention matrices are asymmetric). To better understand the bottleneck of various GNNs under the GPNN framework, we carefully extend the definition of Ricci-curvature to weighted directed graphs without breaking the desired properties of curvature. 

The proposed tools, CURC and GPNN framework, are beneficial to the graph research community, for better understanding the behaviors of GNNs and guiding the future design of GNNs. Open questions can be studied with CURC and GPNN including but limited to: 1. whether the larger average curvatures of the propagation graphs are beneficial for GNNs to learn graphs? 2. do negative curvatures for node pairs contain important information on modeling graph structure?

Reviewer comment (Theory):
The formal connection between CURC and GPNN.

Author response:
A: Please find the answer for W5.

Reviewer comment (Theory):
More clear and detailed implication of CURC properties.

Author response:
A: Thank you for pointing out that the discussion on CURC could be more comprehensive in the main paper. In response, we are committed to refining the terminology and clarifying the indications for reader-friendliness. Additionally, we encourage readers to refer to Appendix B, where CURC is explained in detail. This should provide a more thorough understanding of the topic.

Reviewer comment (Writing):
Generally, the paper is not very friendly to readers and need to be polished, especially section 3 and section 4.

Author response:
> (W1) Generally, the paper is not very friendly to readers and needs to be polished, especially section 3 and section 4.

A: We are currently working on improving the reader-friendliness by introducing clearer notation explanations and adding more bridging language for the revised version.

Reviewer comment (Experiments):
More experiments need to be conducted on larger graph benchmarks (cora, Citeseer, Pubmed and OGB datasets) based on different GTs (except GRIT) to validate the effectiveness and scalability of the proposed method on large graphs.

Author response:
> (W3) More experiments need to be conducted on larger graph benchmarks (cora, Citeseer, Pubmed and OGB datasets) based on different GTs (except GRIT) to validate the effectiveness and scalability of the proposed method on large graphs.

A: We would argue that scalability is actually not the focus of this work; this work aims to analyze and understand why graph transformers are efficacious on various graph learning tasks and which components in the architecture contribute to the efficacy. Therefore, we mainly focus on benchmarks widely utilized in evaluating the expressiveness of graph transformers in previous works (Kreuzer et al., NeurIPS 2021; Rampasek et al., NeurIPS 2022; Ma et al., ICML 2023).
Thus, we would argue that addressing the scalability limitation of graph transformers is not the duty of this paper.

In addition, the suggested node-level benchmarks (e.g., Cora, Citeseer, and Pubmed) are generally not suitable for evaluating the expressiveness of GNNs: even simple GNNs like GCNs, GATs are prone to overfitting, which obfuscates the understanding of expressiveness with the overfitting issue due to the limited amount of training data.

Reviewer comment (Presentation):
The paper is not very easy to follow and some notations are confusing. And some formulas are not numbered. Specifically, the format of Table 3 seems need to be reorganized.

Author response:
> (W4) The paper is not very easy to follow and some notations are confusing. And some formulas are not numbered. Specifically, the format of Table 3 seems need to be reorganized.

A: Same as the reply for W1. We would revise the unclear notations and number all formulas. Regarding the format of Table 3, we would greatly appreciate further suggestions on modification from the reviewers. We just simply followed the format utilized in previous works (Rampasek et al., NeurIPS 2022; Ma et al., ICML 2023).

Reviewer comment (Theory):
It is not clear how the connection of the expressiveness of the model to (only) the function of its adjacency matrix could drive the choice of particular forms for such appropriate functions (which would certainly be a highly practical implication of this work).

Author response:
> (W3) ... how the connection model expressiveness to its adjacency function could drive the choice of such function... practical implication...

A: Driving the choice of adjacency function is probably the most valuable consequence of this work. There are various adjacency function designs exist in the literature (See Appendix D). In general, most of the positional/structrual encoding and rewiring procedures, which brought huge improvements empirically, are an instantiation of adjacency functions. Previously, those are regarded as different directions of research. In this work, a general framework is proposed to setup a unified view on thoes encodings/procedure which brings extra expressiveness beyond 1-WL.

Reviewer comment (Theory):
The paper would be improved if it talked about why we chose Ricci curvature (generalized or not) as a measurement to explore the graph. In geometry, Ricci curvature is a degrade of Riemann curvature, while it is enough to characterize two-dimensional manifolds.

Author response:
> (W2) The paper would be improved if it talked about why we chose Ricci curvature (generalized or not) as a measurement to explore the graph. In geometry, Ricci curvature is a degrade of Riemann curvature, while it is enough to characterize two-dimensional manifolds.

A: This comment is regarding towards **our choice of Ricci curvature**, in the context of exploring graphs in our paper. Your observation about the role of Ricci curvature in geometry is indeed accurate. Ricci curvature, derived from the more general Riemann curvature tensor, is crucial in the characterization of two-dimensional manifolds. However, our paper's focus extends beyond the traditional geometric interpretation of Ricci curvature to its application in graph analysis, particularly within the framework of Graph Transformers.

In our study, we build upon existing research that employs Ollivier-Ricci curvature for analyzing graphs [2]. The choice of Ollivier-Ricci curvature is motivated by its ability to capture the geometric properties of a graph in a way that is particularly relevant to our analysis of Graph Transformers. Specifically, CURC extends the application of Ollivier-Ricci curvature to weighted directed graphs (digraphs), which are representative of the propagation graphs defined in our GPNN framework, unifying various GNNs (e.g., MPNNs, graph rewiring, graph transformers).
The extension to weighted digraphs is not trivial because it allows for a more nuanced understanding of how GNNs behave via propagation graphs rather than on the property of input graphs, 
where the directionality and weights of edges play a critical role. This is particularly pertinent in the context of Graph Transformers, where the learned propagation graphs can differ from the input graphs remarkably.

We believe this extension not only enriches the theoretical understanding of graph curvature in the realm of machine learning and graph neural networks but also provides practical benefits in terms of enhanced model interpretability and performance.

Reviewer comment (Theory):
(Section 4.2, definition 4.2, d^{\epsilon} and (8)) The definition of CURC is questionable, which lies in the so-called asymmetric metric function. As a curvature to measure the curvedness of a manifold (in our case, a graph), its definition should be based on distance (or, say, metric) satisfying positivity, symmetry and triangle inequality. But CURC is based on an asymmetric metric function (which is not a metric), which means the metric could be different from u to v or v to u, which in turn means that the curvature from u to v could be more curved or less curved than the reverse. But CURC is defined as a scale, which contradicts the above.

Author response:
> (W3) ...The definition of CURC is questionable, which lies in the so-called asymmetric metric function ... CURC is based on an asymmetric metric function (which is not a metric) ... But CURC is defined as a scale, which contradicts the above.

A: Thanks for the observations regarding the definition of CURC and its underlying asymmetric metric function in our paper. Your point about traditional curvature measures in geometry relying on symmetric metrics is well-founded. We would like to emphasize that CURC is a **scalar valued function** defined on **node-pairs (source and target)** as stated in definition 4.3, which does not contradict the asymmetric nature of directed distance on weighted-digraphs.

The asymmetric nature of the "distance" used in CURC is a **deliberate and crucial aspect** of our approach, particularly when dealing with propagation graphs in Graph Transformers, which are naturally asymmetric. The asymmetry in the metric function allows CURC to capture the flow of information carried by the **directed and weighted propagation graphs**. This is essential in accurately measuring the propagation dynamics within Graph Transformers. Even though it is proper with symmetric input graphs, the traditional symmetric metrics would not adequately measure the propagation graphs for a better understanding of the behaviors of graph transformers.

Reviewer comment (Writing):
The paper would be improved if it elaborated more on the implications of CURC, KR duality in section 4.3. It is hard to follow the terminology and abstract indications.

Author response:
> (W4) The paper would be improved if it elaborated more on the implications of CURC, KR duality in section 4.3. It is hard to follow the terminology and abstract indications.


A: Thank you for pointing out the areas in section 4.3 where the discussion on CURC and KR duality could be more comprehensive. We acknowledge that the current terminology and abstract indications may be challenging to follow. In response, we are committed to refining the terminology and clarifying the indications for reader-friendliness. Additionally, we encourage readers to refer to Appendix B, where these concepts are explained in detail. This should provide a more thorough understanding of the topic.

Reviewer comment (Novelty):
The definition of a generalized propagation network is more of a rewrite of a graph network with an adjacency function that incorporates propagation and feature mapping (there is still some ambiguity, as mentioned in the first point). It is more on the conveniences to bring out the notion of curvature.

Author response:
> (W5) ... generalized propagation network is a rewrite of a graph network with an adjacency function ... It is more on the conveniences to bring out the notion of curvature.

A: We understand your concern that the current definition might appear as a mere reiteration of a graph network with an enhanced adjacency function. However, our intention was to highlight the GPNN framework as a significant contribution in its own right, serving as a unifying model in the realm of graph deep learning. With the GPNN framework, we can analyze the expressiveness of various GNNs from a unified approach and understand the dominant components in designing more expressive GNNs.

Reviewer comment (Writing):
(7) and lemma B7 in appendix: what does it mean m(x) divided by m(y)? As the definition of m just above, they are vectors.

Author response:
> (Q1)  (7) and lemma B7 in appendix: what does it mean m(x) divided by m(y)? as the definition of m just above, they are vectors.

A: In response to your query about the expression $m(x)$ divided by $m(y)$ in (7) and Lemma B7 of the appendix, it's important to clarify the nature of the mapping $m$. As outlined in Definition 4, $m$ is indeed a mapping from the vertex set $V$ to the interval $(0,1]$. This mapping places $m$ in the space $R^{(0, 1] \times |V|}$, considering the discrete nature of the vertex set. 

The normalized left eigenvector, which we refer to in our paper, resides within this space. Therefore, when we mention $ m(x) $, we are specifically referring to the x-th value of this eigenvector, denoted as $ (\frac{v\_{pf}}{||v\_{pf}||})\_x $. This notation represents the x-th component of the eigenvector $ v_{pf} $ normalized by its norm $ ||v_{pf}|| $. 

When discussing $ m(x) $ divided by $ m(y) $, we are comparing these specific components of the normalized eigenvector, which correspond to different vertices in the graph respectively. This comparison is crucial for the analysis presented in section 7 and Lemma B7, as it relates to the behavior of the network at different vertices.

Reviewer comment (Theory):
CURC measuring GPNN (the paragraph above section 4.3). 'Furthermore, the incorporation of the Perron measure as .....'. How do they relate to bottlenecking within weighted-directed graphs? Any section it refers to?

Author response:
> (Q2) CURC measuring GPNN (the paragraph above section 4.3). 'Furthermore, the incorporation of the Perron measure as .....'. How they relates to bottlenecking within weighted-directed graphs? any section it refers to?

A: The notion of a "bottleneck" in undirected graphs has been extensively studied and is known to be a critical factor affecting the performance of graph models [2]. In such models, bottlenecking refers to a situation where a small number of edges or vertices significantly restrict the flow or connectivity within the network. The authors' proposal in [2] to use Ollivier-Ricci (OR) curvature on input graphs as a means to quantify this phenomenon has been a significant advancement in understanding graph structures. 
When we extend the concept of bottlenecking to weighted-directed graphs, the CURC (Curvature Under Ricci Curvature) becomes a valuable tool. 
This is driven by the fact that different GNNs (e.g., MPNNs, graph rewiring, graph transformers) behave differently given the same input graphs. 
Alternatively, we propose to analyze the propagation graphs of various GNNs under the GPNN framework, which are naturally weighted-directed graphs and not compatible to OR curvature.
In this context, CURC's extension to OR curvature allows for a nuanced understanding of bottleneck phenomena in the propagation graphs. This generalization is crucial as it adapts the concept of bottlenecks, traditionally applied to undirected graphs, to the realm of weighted-directed graphs, thereby providing deeper insights into the bottleneck of GNNs. We will include a more throughout discussion on this topic in the appendix

**The authors hope this clarification addresses the reviewer's query and highlights the novelty and significance of their approach in integrating geometric concepts with advanced graph analytical methods.**