Paper: Global minima, recoverability thresholds, and higher-order structure in GNNs (ICLR-2024)

Reviewer comment (Novelty):
Analysis into heterophily on cSBM datasets have been made in [1][2].

Author response:
Analysis into heterophily on cSBM datasets have been made in [1][2].
- We now cite these sources in the appropriate place. In particular, our work agrees with (sometimes contradictory) prior works by showing that performance on heterophilic data sets can be good or bad depending on the strength of the heterophily and the signal strength from the features. We additionally contribute by presenting the accuracy transition in detail and considering a wider range of architectures (e.g. SAGE, Transformer) and cSBM distributions (e.g. degree correction).

Reviewer comment (Theory):
The theoretical results are limited to 1-2 layer GCNs, and may not be inspiring enough for designing better GNNs.

Author response:
With regards to limitations, we agree that it would be great to handle deep GNNs, but already in the two-layer case, the nonlinearities are unfortunately quite formidable. Fortunately, the 2-layer case is quite commonly used, so we hope that our results are applicable. Similarly, we found the reviewer’s comments about overemphasizing the 2-class empirical results prescient. Our solution has been to change our two figures that used to highlight two-class results to now focus on the more interesting multiclass case. The rest of the discussion section has been rebalanced accordingly to balance binary and multiclass results.

Reviewer comment (Experiments):
The empirical discussion is mostly restricted to 2-class cSBM datasets, so a gap exists between empirical results and real-world scenarios.

Author response:
With regards to limitations, we agree that it would be great to handle deep GNNs, but already in the two-layer case, the nonlinearities are unfortunately quite formidable. Fortunately, the 2-layer case is quite commonly used, so we hope that our results are applicable. Similarly, we found the reviewer’s comments about overemphasizing the 2-class empirical results prescient. Our solution has been to change our two figures that used to highlight two-class results to now focus on the more interesting multiclass case. The rest of the discussion section has been rebalanced accordingly to balance binary and multiclass results.

Reviewer comment (Evaluation):
It would be better if more explanation about how the empirical results are affected by the GNN architecture is provided.

Author response:
It would be better if more explanation about how the empirical results are affected by the GNN architecture is provided.
- Agreed. We have added some additional discussion to the main text and more substantial additional discussion to Appendix C2.

Reviewer comment (Theory):
What's the physical meaning about Theorem 3? It would be better if more explanation is included.

Author response:
We have also added some additional intuition about Theorem 3 in a Remark. Basically, the accuracy estimate correctly combines the relative abundances of various local graph structures and the degree to which they are helpful.

Reviewer comment (Theory):
Intuitively, the high-order structure will be affected by the one-hop structure. On a homophilic graph, its one-hop and high-order neighbors would both be homophilic. How to ensure the high-order impact is completely erased? Does it suggest that designing high-order GNNs is meaningless?

Author response:
Your comments about higher order structure were particularly interesting to us. First, you asked how we ensured that higher order structure is completely erased, particularly since homophily implies some sort of higher order structure automatically. This is true, and we actually had to clarify some of the language of our paper because this sort of higher order structure is impossible to erase while retaining homophily. Rather, the key point is that other higher order structures can be substantially erased, such as recurring motifs, degree-degree correlations, and spatially correlated edge structures. 

The mechanism is a rewiring algorithm where the rewiring ensures that the number of edges between clusters and the degree of each node are respected, but everything else should be randomized. After enough swapping, the resulting graph is basically sampled from a cSBM, which doesn’t have any higher-order structure beyond what is implied by the SBM parameters (e.g. homophily). 

As far as we can tell, this does not mean that designing higher-order GNNs is meaningless, in fact maybe just the opposite. Higher order structure is present in data, and low-order GNNs cannot handle it well, but it seems probable that higher order GNNs might be designed that could specifically handle some of these structures (e.g. recurrent motifs) particularly well. This is, of course, the exact sort of question that we hope our work brings attention to.

Reviewer comment (Theory):
Theoretical results in this paper appear to be somewhat limited, as they focus on computing accuracy at single nodes and rely on fixed GCNs without training. Stronger and more informative findings can already be found in the existing literature, where guarantees for the performance of trained GCNs have been provided (for example, see [1,2]).

Author response:
Thank you for your review, and we are glad that you noticed how we employed symmetry in a new way and found our numerical results interesting. We are sure that the paper is stronger with your feedback. We note that revisions have been highlights in blue for the reviewers convenience.

We are somewhat confused with regards to the limitations you see in our theoretical results and would appreciate any clarification you can offer. In short, it appears that you see weaknesses in that (1) we are focusing on the accuracy at single nodes and (2) we do not consider trained GNNs. With respect to the first objection, we perhaps should clarify that, particularly in Theorem 1 (formerly Theorem 2–we renumbered), the accuracy is actually the generalization accuracy over the entire cSBM family, and similarly with Proposition 1, part 3, so we are a bit unclear on what is meant by focusing on single nodes. Secondly, with respect to trained GNNs, our results in (new) Theorem 1 apply to both trained and untrained GNNs, and we even identify the global minimizer and its accuracy. We do not address gradient descent or similar algorithms, but this does not seem to be the spirit of your comment.

Our best guess for the cause of this disconnect is that our paper is fundamentally about non-linear graph models, whereas the papers you referenced in your review focus on linear models. Other reviewers noted various deficiencies in our presentation in that section that obscured our main point. While our paper does reference linear GNNs, the majority of our theory results (and all of our simulations) focus on the behavior of genuinely nonlinear models. We are sometimes able to bound the performance of the nonlinear models in terms of the corresponding linear models. We have substantially rewritten the theory section to more clearly reflect these facts–apologies if the original submission was unclear. Hopefully that section is a much more enjoyable and enlightening read now. All of this is of course a guess, and we welcome clarification if we are off track.

Finally, we note that we have clarified the final section on the detrimental impact of higher order graph structure on GNN performance, clarifying that this section is pointing out a fundamental weakness of current GNN architectures and pointing out possible directions for future improvements.

Reviewer comment (Novelty):
The author missed a couple of key literature pieces, such as Wu, Chen, Wang and Jababaie ‘A non-asymptotic analysis of oversmoothing in GNN’ in ICLR2023. As a result, some key results of the paper have been shown or can be derived from existing results in the literature. For instance, Theorem 1 of the paper is established in Lemma 1 of Wu, Chen, Wang and Jababaie (although this paper does not make clear the distinction between homophily and heterophily regime, the results of Theorem 1 are only a few lines away from their Lemma).

Author response:
Thank you for carefully reading our paper! We found these comments very interesting and are glad that you found the paper thought provoking. We particularly appreciate the pointers to other literature, not all of which we were aware of. We now reference these works in the appropriate places. We do wish to clarify the relationship between our work and these in this comment as well. Revisions have been highlights in blue.

Specifically, in the paper of Wu et al. ICLR 2023, Lemma 1 is indeed closely related to our Theorem 1. There is a difference in normalization of the graph operator and possibly in the homophily assumption, but we expect that these are not essential differences. Indeed, we would not be surprised if versions of our Theorem 1 were known elsewhere. Therefore, we have reclassified our Theorem 1 as a Proposition instead and explicitly cited the related Lemma 1 from Wu et al. nearby to clarify that similar facts were already in the literature.

Reviewer comment (Novelty):
In Wu et al, the authors noted the ineffectiveness of nonlinearities (in particular ReLU) in Appendix K1, which may also give rise to much of the results of Theorem 2 in the current paper. Therefore, much of the theoretical contribution in this paper has already been established elsewhere.

Author response:
We do, however, feel that our presentation has caused a lack of clarity about the relationship between our last two theorems and the rest of the literature, including Appendix K1 of Wu et al. For example, Proposition 6 in Wu et al.’s appendix states that adding a ReLU activation function after a sequence of linear aggregation layers does not improve classification. This may have a bearing on our Theorem 1 (now Proposition 1), but we actually view it as quite different from our Theorems 2 and 3 (now Theorems 1 and 2), which deal with a nonlinearity interleaved between the linear layers. More generally, the larger purpose of our theoretical section is to show that the nonlinear models cannot perform better than linear models under certain circumstances. We have rewritten section 4.2 with this distinction in mind and hope that it is clearer for the reviewer.

Reviewer comment (Writing):
Pg 3: n_out “the number of nodes in other classes” — do you mean “the number of neighbors in other classes”? Also, n_in and n_out are very dependent on i (the vertex), so it is suggested to write n_in(i), for example, to avoid confusion (since \mathcal{N}(i) is already written elsewhere).

Author response:
Pg 3: n_out “the number of nodes in other classes”, do you mean “the number of neighbors in other classes”? Also, n_in and n_out are very dependent on i - the vertex, so I’d suggest writing n_in(i), for example, to avoid confusion (since you’ve already written \mathcal{N}(i), for example).
- changed to n_in(i)

Reviewer comment (Theory):
Pg 3: Assuming that the means are of opposite sign does not cause loss of generality only in the case when the number of classes is 2. Can the results be generalized to more than 2 classes?

Author response:
Pg 3: Assuming that the means are of opposite sign does not cause loss of generality only in the case when the number of classes in 2. Can the results be generalized to more than 2 classes?
- We have not generalized it here, but we have explored an increased number of classes in the empirical work. Due to the symmetries it is possible the proof would extend.

Reviewer comment (Theory):
Pg 3: First bullet point of Theorem 1 — There seems to be some transpose or dot missing when taking the inner product of vectors (e.g., mW should be either m^\top W or m \cdot W).

Author response:
Pg 3: First bullet point of Theorem 1. There seems to be some transpose or dot missing when taking the inner product of vectors (eg - mW should be either m^\topW or m\cdot W).
- changed to m\cdot W

Reviewer comment (Writing):
Pg 4: Second bullet point of Theorem 1 — What is the notation y[X]? Is it the same as y(X)?

Author response:
Pg 4: Second bullet point of Theorem 1. What is the notation y[X], is it the same as y(X)?
- Fixed, changed all to y[X]

Reviewer comment (Writing):
Pg 4: Third bullet point of Theorem 1 — The term “maximum accuracy” lacks a clear definition in the main paper.

Author response:
Pg 4: Third bullet point of Theorem 1. The term “maximum accuracy” also lacks a clear definition in the main paper.
- Fixed, changed to Maximum expected accuracy for an arbitrary node

Reviewer comment (Theory):
Pg 6: Statement of Theorem 3 — The linear model still has a nonlinear sigma written in it.

Author response:
Pg 6: Statement of Theorem 3. The linear model still has a nonlinear sigma written in it.
- Fixed, we have removed the sigma.

Reviewer comment (Theory):
Equation 7 (page 14) — The first derivative is also 0 when n_in = n_out. Is it possible for some setting of the cSBM such that n_in = n_out with high probability? Perhaps this is mentioned later on below equation 15, but it is not clear what the scope of “we make no claims” is, and if it is wide enough, it should be reflected in the main statement of the main paper. For instance, there are phase transition results for SBM suggesting that if lambda < 1, then the SBM is not distinguishable from the Erdos-Renyi model with average degree d.

Author response:
Equation 7 (page 14). The first derivative is also 0 when n_in=n_out (is it possible for some setting of the cSBM such that n_in = n_out with high probability?) Perhaps this is mentioned later on below equation 15 but it’s not clear what is the scope of “we make no claims” and if it’s wide enough, should be reflected in the main statement in the main paper. For instance, there are phase transition results for SBM that suggest that if lambda < 1 then the SBM is not distinguishable from Erdos-Renyi model with average degree d.
- Thank you for pointing out this discrepancy in the paper. We would like to clarify this both here and in the appendix. If n_in=n_out then regardless of the weight matrix or theta we have that the probability evaluates to .5 (This can be seen in part 2 of proposition 1). Thus every weight matrix is optimal. We have updated the appendix to reflect this fact.

Reviewer comment (Experiments):
the setup is limited and it is not clear whether the linear classifier is good beyond the assumptions

Author response:
Regarding limitations of GNNs when the assumptions do not hold, this is of course a valid point, and we have rewritten the paper to emphasize that the superiority of linear classifiers in general is not one of the claims of the paper. The object of interest here is the nonlinear GNN, and we are using the performance of linear GNNs as a tool to get there. For example, Theorems 2 and 3 both tell us significant facts about nonlinear GNNs. Namely, unless certain asymmetries are included in the data generation model, there is no theoretical reason that the nonlinearities will be helpful. From this we can get tight GNN accuracy bounds purely in terms of linear models (which turn out to be a special case of the nonlinear GNNs, with the right parameter choices). This motivates research into better models than cSBM for the data and, more importantly, a deeper understanding about what significant features of the data should be incorporated into such a model. For example, ideas like homophily and Gaussian features are simply not enough to correctly motivate nonlinear GNNs theoretically. We have significantly enhanced the presentation and discussion in the paper to help get these points across better.

Reviewer comment (Theory):
nice theoretical results but I guess they are highly tied to the assumptions (limited)

Author response:
Regarding limitations of GNNs when the assumptions do not hold, this is of course a valid point, and we have rewritten the paper to emphasize that the superiority of linear classifiers in general is not one of the claims of the paper. The object of interest here is the nonlinear GNN, and we are using the performance of linear GNNs as a tool to get there. For example, Theorems 2 and 3 both tell us significant facts about nonlinear GNNs. Namely, unless certain asymmetries are included in the data generation model, there is no theoretical reason that the nonlinearities will be helpful. From this we can get tight GNN accuracy bounds purely in terms of linear models (which turn out to be a special case of the nonlinear GNNs, with the right parameter choices). This motivates research into better models than cSBM for the data and, more importantly, a deeper understanding about what significant features of the data should be incorporated into such a model. For example, ideas like homophily and Gaussian features are simply not enough to correctly motivate nonlinear GNNs theoretically. We have significantly enhanced the presentation and discussion in the paper to help get these points across better.

Reviewer comment (Theory):
I think the theoretical contributions of the paper are nice but unfortunately not enough and limited to particular assumptions

Author response:
Regarding limitations of GNNs when the assumptions do not hold, this is of course a valid point, and we have rewritten the paper to emphasize that the superiority of linear classifiers in general is not one of the claims of the paper. The object of interest here is the nonlinear GNN, and we are using the performance of linear GNNs as a tool to get there. For example, Theorems 2 and 3 both tell us significant facts about nonlinear GNNs. Namely, unless certain asymmetries are included in the data generation model, there is no theoretical reason that the nonlinearities will be helpful. From this we can get tight GNN accuracy bounds purely in terms of linear models (which turn out to be a special case of the nonlinear GNNs, with the right parameter choices). This motivates research into better models than cSBM for the data and, more importantly, a deeper understanding about what significant features of the data should be incorporated into such a model. For example, ideas like homophily and Gaussian features are simply not enough to correctly motivate nonlinear GNNs theoretically. We have significantly enhanced the presentation and discussion in the paper to help get these points across better.

Reviewer comment (Writing):
Section 4.2 is not well written. It is not clear what the authors want to say and it does not have flow. I strongly recommend rewriting it.

Author response:
First, regarding Section 4.2, we apologize the section was not easier to read, and we have done a significant rewrite, adding in particular guiding Remarks in appropriate places and clarifying the notation. This theorem is really very nice, along with its proof, and we hope the reviewer will take a second look at the much better presentation. In particular, Theorem 2 implies that Theorem 3 is actually a bound on the accuracy of nonlinear GNNs as well. Your comment has substantially increased the quality of this paper, for which we thank you.

Reviewer comment (Writing):
The definition of $\sigma$ in page 4 is missing. It first seems it is a function, but apparently it is a constant?

Author response:
Smaller point: $\sigma$ was defined on page 4, but its usage here without a subscript is apparently unclear. We have added text explicitly clarifying the role of sigma here.