Paper: Size Generalization of Graph Neural Networks on Biological Data: Insights and Practices from the Spectral Perspective (ICLR-2024)

Reviewer comment (Experiments):
As the paper takes a data-driven approach, the main question is whether these empirical results are transferable to other domains, other datasets, other models.

Author response:
**Q4.1 Transferability of results**

As mentioned in the general response, one important insight brought up by this paper is that the covariate shifts caused by variations in sizes are dependent on the data domain. Thus, we restrict our attention to the biological domain. More importantly, most existing datasets for graph classification belong to the biological or chemistry domain.

Reviewer comment (Experiments):
The models and datasets are academic. Is it possible to apply this model to more real-world datasets and showcase how this method can be used in biological scenarios?

Author response:
**Q4.4 Other datasets**

As mentioned in the general response, our selection of datasets is based on the biological domain, and the dataset should not suffer significantly from other issues that blur the focus on size generalization. More details can be found in the datasets section in the general response.

Reviewer comment (Experiments):
GNNs that counts or can aware substructures were not compared.

Author response:
**Q3.4 Comparison with other baselines**

We have added more baselines in Table 4 (including some expressive GNN models). For more details, please refer to the baseline section in the general response and Table 4 in our updated submission. In general, the proposed method SIA still achieves best performance and is model-agnostic. Thank you for your valuable suggestions, we do find that expressive model that excels at detecting cycles have good size generalizability on certain datasets.

Reviewer comment (Theory):
Besides cycles, can there be any substructure that changes the spectrum according to changes in size and number?

Author response:
**Q3.2 Besides cycles, can there be any substructure that changes the spectrum according to changes in size and number?**

There can be. The insights for our investigation into cycles are based on our discernment of variations in the peaks of specific frequencies within the graph spectrum, such as the frequency=1 in unnormalized Laplacian matrices. Many of these specific frequencies align with the frequencies of cycles, with cycles containing 5-7 nodes being particularly evident. As a reminder, the spectrum for a cycle with n nodes is given by the formula 2-2cos(2πj/n). We also computed the spectrum of other substructures such as lines, but their difference in small and large graphs is not consistent across different biological datasets.

Reviewer comment (Experiments):
Where is a comparison with GNNs [1-8] that can understand the structure of the substructure and predict its number relatively accurately or consider out-of-distribution?

Author response:
**Q3.4 Comparison with other baselines**

We have added more baselines in Table 4 (including some expressive GNN models). For more details, please refer to the baseline section in the general response and Table 4 in our updated submission. In general, the proposed method SIA still achieves best performance and is model-agnostic. Thank you for your valuable suggestions, we do find that expressive model that excels at detecting cycles have good size generalizability on certain datasets.

Reviewer comment (Experiments):
The results in Tables 3 and 4 are the results after class imbalance and size imbalance have been corrected. What is the performance in the class imbalance setting of the original data?

Author response:
**Q3.5 performance in the class imbalance setting of the original data**

In our experiments, we find that certain backbone models tend to be very biased towards the major class and thus result in an extremely skewed performance. For instance, the FAGCN and MLP backbone model always outputs a test accuracy (and thus unreliable F1 score) that is almost the same as the proportion of the larger class. In order to prevent such cases from happening, we decided to adopt upsampling.

Reviewer comment (Experiments):
Is size generalizability using cycle applicable to other data domains beyond the biological domain?

Author response:
**Q3.6 Is size generalizability using cycle applicable to other data domains beyond the biological domain?**

It might be also applicable to the chemistry domain. For the other domains, it relies on further investigation of the spectrums. However, most existing datasets for graph classification belong to the biological or chemistry domain.

Reviewer comment (Experiments):
Experiments are insufficient and lack of comparison with related methods, such as the size-generalization methods referenced in the related work. The baselines are not state-of-the-art methods in the relevant field. The authors need to add comparison experiments with methods from most recent years, which are related to this paper. Furthermore, the paper lacks experimental validation from other perspectives, such as the effect of different graph size settings in the training process.

Author response:
**Q2.1 Insufficient experiments and comparison with baselines**

We have added more baselines in Table 4. For more details, please refer to the baseline section in the general response and Table 4 in our updated submission. In general, the proposed method SIA still achieves the best performance and is model-agnostic.

Reviewer comment (Novelty):
The contribution lacks novelty. In this paper, the authors identify cycle structures as a major factor affecting the generalization capacities of GNNs. This finding looks to be a special case of [1]. In Section 3, the authors observe that cycle structures have an impact on the spectrum differences between graphs, but it is difficult to ascertain the effect of graph size and cycle distribution on the generalization capacities of models. Also, the three proposed strategies in Section 4 lack novelty and could be combined into a single algorithm.

Author response:
**Q2.2 Novelty wrt. "From local structures to size generalization in graph neural networks"**

Actually, the empirical findings of our paper are contradictory to the assumptions made by the paper "From local structures to size generalization in graph neural networks". In their paper, they hold the assumption that the degree patterns change with graph sizes. However, in Appendix G of our updated paper, we find that the degree distribution does not show a clear correlation with the graph size on biological datasets. Our main contribution is that we find cycle-related information (existence of cycles, average cycle length) is critical for size generalization for biological data, which has not been proposed before.

Reviewer comment (Writing):
Why is the algorithm description incomplete in the text, such as the section 4.3? If not essential, it could be excluded as the part of the contributions.

Author response:
**Q2.3 Why is the algorithm description incomplete in the text, such as the section 4.3? If not essential, it could be excluded as the part of the contributions.**

This is primarily due to page limits. In fact, we have explained well about the motivation as well as the main function of this algorithm at the same paragraph. We leave the details of implementation in the Appendix for further interest.

Reviewer comment (Evaluation):
Although the analysis shows that there is a connection between spectrum differences with the cycle lengths, there could be some underlying confounders that jointly affect the graph sizes and cycle lengths. For example, in the model by Bevilacqua et al. 2021, the graphon and the size of the graph will jointly affect the cycle lengths.

Author response:
There might be other confounders that jointly affect the graph size and cycle lengths. However, the method by Bevilacqua et al. 2021 cannot fully address the covariate shifts we found. In that paper, they assume that the density of induced k-sized subgraphs is invariant. However, based on our empirical observations, we actually found big cycles that do not appear in small graphs, suggesting that this density invariance does not hold for large cycles. In other words, their model cannot generalize from cycles of small lengths to cycles of large lengths.

Reviewer comment (Theory):
The proposed three solutions are well motivated, while mainly based on empirical observations. To what extent can the three methods resolve the cycle issue? Will the operations affect the expressivity of GNNs?

Author response:
Explicitly including cycle information can help distinguish more graphs, but will not fundamentally affect the expressiveness of the model. To understand this problem better, in Table 4, we also include two expressive models as our baselines: RPGNN[R2] and SMP[R3]. What we find is that the general expressive model can help size generalization to some extent, but is not as good as explicitly providing this information. The model that excels at cycle-related tasks, e.g. SMP, exhibits better size generalizability than other expressive models, such as RPGNN.

Reviewer comment (Experiments):
Why do the experiments adopt a different data split scheme from previous practice such as in Bevilacqua et al. 2021?

Author response:
Firstly, it's important to note that we did not utilize identical datasets as those referenced in Bevilacqua et al. 2021. Secondly, the data files we obtained proved to be incompatible with the current versions of PyTorch and CUDA, preventing us from successfully loading the data into our environments. Thirdly, when attempting the data processing using the identical original code provided in Bevilacqua et al. 2021, we encountered errors. Consequently, we were unable to replicate the exact data split presented by Bevilacqua et al. 2021. Despite this deviation, our experimental setup closely aligns with the overarching concept of partitioning datasets into larger and smaller subsets.

Reviewer comment (Experiments):
How well do the proposed methods perform on more realistic and large datasets such as OGB-molhiv with graph size shifts, and DrugOOD[1]?

Author response:
We sincerely thank the reviewer for giving several dataset examples. Initially, we explored several other biological datasets, such as the HIV dataset you mentioned. However, we encountered a significant class imbalance issue (please refer to the dataset section in the general response for details). For instance, in the HIV dataset, the 50% smallest dataset contains 20054 class 0 samples and 510 class 1 samples, whereas its 10% largest dataset contains 3680 class 0 samples and 433 class samples. Other large-scale datasets from ogb and TU exhibit similar trends. Additionally, for the other dataset mentioned in your review, DrugOOD includes various covariate shifts other than graph sizes. For the one concerning size shift (DrugOOD-lbap-core-ic50-size), it adopts a different idea in splitting the data, where it uses the large datasets in the training and validation set and small datasets in the testing. Due to such differences, we did not conduct experiments on it.

Reviewer comment (Experiments):
Can the proposed methods perform better than previous solutions like Bevilacqua et al. 2021, and Buffelli et al. that are cited in the paper, and [2,3] that are the state-of-the-art in graph size OOD generalizations?

Author response:
Thanks for your suggestion. We have just included the performances of two versions of CIGA (Learning Causally Invariant Representations for Out-of-Distribution Generalization on Graphs, NeurIPS’22) in Table 4 in our second update of the paper. CIGA is indeed a strong baseline, but we can still see that the average improvements across different datasets are not as good as SIA.

Reviewer comment (Novelty):
[4] analyzes the size generalization in link predictions, which is also a related work to discuss.

Author response:
Thank you for your great suggestion! We have added this paper to our related work section.

Reviewer comment (Evaluation):
Can the proposed methods improve the size generalization in algorithmic reasoning tasks?

Author response:
One important view this paper takes is that the size-induced distribution shift depends on the data domain, and that is why contradictory conclusions appear in prior works. Though our main empirical findings apply mainly to biological data, we can adopt a similar spectral analysis to other fields, and identify other patterns that lead to the spectral differences.