Paper: Node Classification in the Heterophilic Regime via Diffusion-Jump GNNs (ICLR-2024)

Reviewer comment (Presentation):
The main drawback of this paper is its organization and presentation, which can be improved.

Author response:
**W1**. In the paper, we used a couple of figures (Fig. 1 and Fig. 2) to illustrate respectively the concepts of structural heterophily and jumps hierarchy. These two elements summarize the key 
contributions of the approach.

Reviewer comment (Theory):
Some statements/claims from this paper seem problematic and not accurate. E.g., in Eq. (2), seems the propagation matrix/Jumps $J^k$ cannot be strictly termed as filters according to the definition from the graph signal processing [1]. Specifically, if they are constructed according to Eqs (2) and (4), they may not share the same set of eigenvectors with the original adjacency matrix.

Author response:
**W2**. The  $\mathbf{J}^k = (V, E_k, C_k)$ are referred to as "filters" by following MixHop [Abu-El-Haija et al. (2019)]. In MixHop, the filters (which are actually powers of the transition matrix) are not learnable. However, we explicitly use $\mathbf{J}^k = (V, E_k, C_k)$ to emphasize that each edge of the so-called "Structural Filter" is endowed with a learnable weight (see Fig.3 where the tridimensional functions illustrate what extent these coefficients define the support of a filter).

In addition, there is a relationship with graph signal processing: we learn the Fourier Transform of the graph. We are not learning the eigenvectors of the adjacency matrix but those of the normalized Laplacian which are equivalent in reverse order to those of the transition matrix.

Reviewer comment (Novelty):
The novelty of this paper is not outstanding. From a high-level view, the idea of a filter bank (i.e., a set of filters/propagation matrix) is included in many existing works, e.g., [2-5]. Also, the proposed heterophily measure "Structural Heterophily" is related to the existing measure "edge homophily". The difference between the "Structural Heterophily" and edge homophily is that (1) the numerator of the former is the # of different-label connected node pairs, and the numerator of the enumerator of the latter is the # of same-label connected node pairs; (2) the denominator of the former is topology smoothness and the denominator of the latter is the total # of edges.

Author response:
**W3**. Concerning the novelty, we are aware that the idea of learning a filter bank is in the literature: see for instance MixHop. However, the value of our approach is a technique for learning the shape of these filters, i.e. the coefficients $C_k$. To sum up, note that in this paper the term "Structural Filter" refers to a learnable attributed graph.

Regarding structural heterophily, please see the answer to the Q3.

**Q3**. As you noted in W3, structural heterophily is related to edge homophily. However, this relationship is not as straight as you suggest. Edge homophily simply counts the number of homophilic edges, independently of how they are located in the graph. For instance, intra-class heterophilic edges are quite structurally different from those in the bottleneck: the first ones are more difficult to bypass than the natural heterophily arising from the bottleneck. Bottleneck heterophily can not be avoided, but edge heterophily counts these edges and overestimates the difficulty of the problem.

Our proposed metric focuses on those intra-class edges and discards bottleneck edges (see Fig.1 A, where structural heterophily is one). The denominator in our metric is the Fiedler value, whereas the numerator is close to one of the eigenvalues of the Laplacian. As a result, our measure approximates the ratio between two structural variabilities. Which is quite different from counting edges.

Reviewer comment (Theory):
In the 2nd paragraph of Section 2, it mentioned "$l^*=\arg\min l^T\Delta l$, where $l*$ is the smoothest labeling of V after propagating l(B) to l(U) through the edges of the graph". I think this is not accurate. Solely based on the context provided in this paragraph, the $l^*$ should be a vector whose entries are all the same.

Author response:
**Q1**. Yes, you are right, the correct formula is: $\ell^{\ast}=\arg\min_{\ell\neq\mathbf{0},\ell\perp\mathbf{1}} \ell^T\triangle\ell$, as in the beginning page 4.

Reviewer comment (Novelty):
What is the advantage of Structural Heterophily compared with existing homophily/heterophily measures? Also, according to the statement "For $R > 1$ the graph is heterophilic", so only $R=1$ the graph is homophilic? I think the statement is not accurate.

Author response:
**Q3**. As you noted in W3, structural heterophily is related to edge homophily. However, this relationship is not as straight as you suggest. Edge homophily simply counts the number of homophilic edges, independently of how they are located in the graph. For instance, intra-class heterophilic edges are quite structurally different from those in the bottleneck: the first ones are more difficult to bypass than the natural heterophily arising from the bottleneck. Bottleneck heterophily can not be avoided, but edge heterophily counts these edges and overestimates the difficulty of the problem.

Our proposed metric focuses on those intra-class edges and discards bottleneck edges (see Fig.1 A, where structural heterophily is one). The denominator in our metric is the Fiedler value, whereas the numerator is close to one of the eigenvalues of the Laplacian. As a result, our measure approximates the ratio between two structural variabilities. Which is quite different from counting edges.

Reviewer comment (Writing):
On the top of Page 4, it mentioned "unstable states u1, u2," and $\bar{A}_1, \bar{A}_2, \dots$. What are those vectors and adjacency matrices? How to obtain them from the context? In the paragraph next, it mentioned "if we relabel the white node, ....., the new Fiedler vector ul leading to the labeling l does no longer induce a sharp step function". It is confusing since the Fiedler vector should only be determined by the graph topology, and not related to the labeling of nodes. In addition, what is the "sharp step function"?

Author response:
**Q4**. To begin with, $\bar{A}$ is a partition, not an adjacency matrix: $\bar{A} \cap A = \emptyset$. Actually, the white node in Fig.2 belongs to $\bar{A}$. Then, the notation $\bar{A}$: $\bar{A}\subseteq \bar{A}_1\subseteq \bar{A}_2\subseteq\ldots$ indicates the level sets depicted in Fig.2 bottom.

Answering your question, $\mathbf{u}_1,\mathbf{u}_2,\ldots$ lead to energies $\mathbf{u}_1^T\triangle\mathbf{u}_1$, $\mathbf{u}_2^T\triangle\mathbf{u}_2,\ldots$

In addition, we are aware that the Fiedler is independent of any labeling. And this is why it is the denominator of our proposed measure. Homophilic graphs tend to have the same energy (numerator) as the Fiedler vector: labeling meets optimal structural partition. In the heterophilic regime, however, it is very likely to have energies (denominator) larger than the Fiedler value.

Concerning the sharpness of the Fiedler vector, it is well known that the smaller the bottleneck, the sharper the partition. But you are right: the sentence "the new Fiedler vector" should be "the new vector" and it is corrected in the new version, thank you for your awareness.

Reviewer comment (Writing):
In Section 3, what is the relationship/difference between $f_{\Theta}$ and $f_{\theta}$?

Author response:
**Q5**. We wanted to discriminate between the parameters of the MLP $f_{\theta}(\mathbf{A})$  and the parameter $\Theta$ of the full model.

Reviewer comment (Writing):
In a subsection named "Exploration by Parallel Jumping", the Jumps $J^k$ is presented as a tuple with three elements, but according to the context, the Jumps $J^k$ should be a matrix with the same shape as the adjacency matrix, which is confusing.

Author response:
**Q6**.  Jumps $J^k = (V,E_k)$ are graphs, we should indicate that at the end of the first sentence of the paragraph **Exploration by parallel Jumping**. Filters $\mathbf{J}^k = (V,E_k,C_k)$ are just attributed graphs. Thank you again for making this point more clear in the new version.

Reviewer comment (Evaluation):
On page 9, it claimed that "our method needs higher frequency eigenfunctions in order to capture the extreme degree variability of this dataset". Any reference or evidence?

Author response:
**Q8**. This claim is referred to the fact that we are outperformed by FSGNN using 8 hops. Please, note that our method is shallow. We want to emphasize that certain types of graphs require a higher number of labels in order to obtain a more precise classification with a single layer. The discussion of multiple-layered *Diffusion-Jump GNNs* is in progress.

Reviewer comment (Writing):
The paper is poorly written with important notations not defined and explained. This makes it very difficult to judge the contributions of the paper. Examples include the issues raised in the following questions.

Author response:
**W1. Paper poorly written.** What exactly you don't like from the paper? How can we improve it?

Reviewer comment (Writing):
The paper seems to be written in a hurry and not proofread. It is not ready for submission.

Author response:
**W2. In a hurry.** Well, from the abstract to the supplementary stuff there is a formal construction flowing from intuition to inspiring methods and stopping at the contribution. We ask the following scientific question: *Could a GNN learn the eigenvectors of a graph and use them for node classification?* This is not a minor question (no work in the literature does this: eigenvectors are usually pre-computed). As this is not a minor question it deserves a motivation (measuring heterophily in a different way) and the use of the eigenvectors in a node distance. Having the distance to hand we explore in parallel different level-sets to solve the semi-supervised node-classification task.

Reviewer comment (Writing):
The abstract is too long and does not capture concisely the contributions of the paper. This is not how an abstract should be written.

Author response:
**Q1**. Abstract is within the limits of the template. The abstract has 3 stages: motivation (heterophily), technique (related to the state of the art) and optimization problem (trace-ratio) with experimental results.

Reviewer comment (Writing):
What is the difference between $f_{\Theta}$ and $f_\theta$? There are notation confusions throughout.

Author response:
**Q2**. Difference between $f_{\Theta}$  and $f_{\theta}$. $\Theta$ refers to the whole parameters of the GNN whereas $\theta$ refers to the MLP parameters. What other confusios do you see?, please let us know.

Reviewer comment (Writing):
In (1), what is $\ell$ and is this the same as $\ell^*$ defined previously?

Author response:
**Q3**. What is $\ell$? $\ell$ is defined in the first paragraph of section 2; it is a labeling. After that, we define $\ell^{\ast}$. as the optimal (minimal energy) labeling.

Reviewer comment (Writing):
On page 4, from the phrase "Consequently, the jump hierarchy defines a succession of unstable states" onwards, the terms have not been explained clearly. What are "unstable states"? What is an "expansion"?

Author response:
**Q4**. Jump Hierarchy and expansions (page 4). $\bar{A}$ is a partition of the vertex set: $\bar{A} \cap A = \emptyset$. Actually, the white node in Fig.2 belongs to $\bar{A}$. Then, the notation $\bar{A}$: $\bar{A}\subseteq \bar{A}_1\subseteq \bar{A}_2\subseteq\ldots$ indicates the level sets depicted in Fig.2 bottom. This is a refinement. Then, $\mathbf{u}_1,\mathbf{u}_2,\ldots$ lead to energies $\mathbf{u}_1^T\triangle\mathbf{u}_1$, $\mathbf{u}_2^T\triangle\mathbf{u}_2,\ldots$ and they are unstable because their energies are lower bounded by the optimal one (the ground energy). $\mathbf{u}^T\triangle\mathbf{u}$ defined by the Fiedler vector $\mathbf{u}$.

Reviewer comment (Writing):
The section "Exploration by Parallel Jumping" needs to be rewritten for clarity. What are "structural" filters? Why are there $K+1$ of them? How do you choose $K$? The authors seem to want to say that they are learning a graph filter but instead tries to bring in confusing jargon.

Author response:
**Q5**. Structural filters are weighted graphs, i,e, tuples $\mathbf{J}^k=(V,E_k,C_k)$ where $E_k\subseteq E$ and $C_k$ are the learnable coefficients of these edges. By the way, the concept of *structural filter* is introduced in the abstract. 

There are $K+1$ filters because the last one is the original graph to capture the full homophilic case (Homophilic branch: see in the last paragraph of section 4),

Reviewer comment (Novelty):
How is the proposed approach different from graph rewiring?

Author response:
**Q6**. Actually the filters are rewirings.

Reviewer comment (Writing):
Some mathematical formulas and concepts within the paper may be challenging for non-experts. A more simplified explanation or background could make it more accessible to a broader audience.

Author response:
**W1**. We are aware of that and due to space limitations, we have explained the methods inspiring our approach in the Appendix. We also include some experiments with SBMs.

Reviewer comment (Evaluation):
The Diffusion-Jump GNN might introduce computational complexities, especially related to diffusion distances. This could be a concern in resource-constrained settings or applications requiring real-time responses.

Author response:
**W2**. Sure. See answers to Q1 and Q3. 

**Q1**. Dealing with huge graphs is still a challenge that new GNN layers or architectures have to deal with today. In our case, creating a dense diffusion matrix forces us to use an $O(n^2)$ memory space, just as a Transformer or HO-GNNs would do. That said, in the paper we have tried to include large graphs to show that our performance is still competitive and we leave the window open so that in future work, we do not need all the distances but a selection of them. Making our method sparse and consuming less memory. Part of this sparsification effort relies on computing the eigenfunctions of the transition matrix (which involves the sparse adjacency in the trace-ratio problem instead of the Laplacian). 

**Q3**. Yes of course, as we said in Q1, we left the window open so that in the future we can work in a more sparse way and also improve the performance. As to the question of whether we are planning on expanding it for other tasks, the answer is yes, we are currently looking at how these eigenfunctions behave and the distances they generate in link prediction. Since we are observing that they do not react in the same way as in node classification task, because these functions have to capture the nature of the graph, instead of changing it as we do in this work in order to connect distant nodes of the same class. Our preliminary experiments in link prediction show that we can replace the diffusion distance matrix by the concatenation of the matrix of features with the eigenfunctions. This allows us to deal with graphs with $0(10^6)$ edges. What we want to emphasize here is the power of empirical eigenfunctions.

Reviewer comment (Experiments):
The paper does not delve deeply into the stability of training the model. Some GNN models might be susceptible to issues like vanishing or exploding gradients, which could affect the training efficiency of the Diffusion-Jump GNN.

Author response:
**W4**. Concerning the stability, it is important to note here that the diffusion pump has a double role: (a) it generates eigenfunctions, and (b) it reduces the structural energies (if needed) of the modifications proposed by the classification loss (regularization). We implement a shallow network where gradients are under control. However, the interference between the Dirichlet loss and the classification loss is important during the initial epochs. Then, the Dirichlet loss converges to a basic set of eigenfunctions, and finally, after the stabilization of the Dirichlet loss, the classification loss fluctuates until convergence. We agree that we have to delve into this point, and we will do that in future papers since we honestly believe that diffusion pumps can be useful modules in other GNNs.

Reviewer comment (Experiments):
Given the potential computational complexities, how efficient and scalable is the Diffusion-Jump GNN when dealing with large graphs, e.g., those with millions of nodes and edges?

Author response:
**Q1**. Dealing with huge graphs is still a challenge that new GNN layers or architectures have to deal with today. In our case, creating a dense diffusion matrix forces us to use an $O(n^2)$ memory space, just as a Transformer or HO-GNNs would do. That said, in the paper we have tried to include large graphs to show that our performance is still competitive and we leave the window open so that in future work, we do not need all the distances but a selection of them. Making our method sparse and consuming less memory. Part of this sparsification effort relies on computing the eigenfunctions of the transition matrix (which involves the sparse adjacency in the trace-ratio problem instead of the Laplacian).

Reviewer comment (Experiments):
How would the Diffusion-Jump GNN be adjusted or modified when faced with graphs having multiple types of nodes and edges or other intricate structures? Is the method versatile enough for these complex scenarios?

Author response:
**Q2**. We believe that when Diffusion-Jump GNN can deal with graphs that have multiple nodes or edges.
One possible way is to follow the "role2vec" approach (https://arxiv.org/pdf/1802.02896.pdf): compute the transition matrix between types of nodes (for instance) and then rely on this transition matrix. A similar approach can be applied to the line graph of the original graph if there are multiple types of edges. Finally, for a multigraph, we should create a diffusion pump for each particular type of edge.

Reviewer comment (Experiments):
Are there plans to further optimize the Diffusion-Jump GNN for better performance or reduced computational demand? Is there potential to extend this method to other graph tasks like graph classification or link prediction?

Author response:
**Q3**. Yes of course, as we said in Q1, we left the window open so that in the future we can work in a more sparse way and also improve the performance. As to the question of whether we are planning on expanding it for other tasks, the answer is yes, we are currently looking at how these eigenfunctions behave and the distances they generate in link prediction. Since we are observing that they do not react in the same way as in node classification task, because these functions have to capture the nature of the graph, instead of changing it as we do in this work in order to connect distant nodes of the same class. Our preliminary experiments in link prediction show that we can replace the diffusion distance matrix by the concatenation of the matrix of features with the eigenfunctions. This allows us to deal with graphs with $0(10^6)$ edges. What we want to emphasize here is the power of empirical eigenfunctions.