Paper: Shape-aware Graph Spectral Learning (ICLR-2024)

Reviewer comment (Theory):
In the presented theorem, the two compared graph filters have the same response norm, which is reasonable but not the case for many existing spectral GNNs. My concern is that, although the theorem is correct, some existing spectral GNNs can still fit the desired shape by further increasing the overall norms of their responses. This is not a technical flaw but needs to be further discussed.

Author response:
1. In Theorem 1, the reason **why we assume two filters have the same response norm is to avoid the trivial solution**. To be more specific, we made two assumptions. (1) before a certain point $m$, $g_1(\lambda_i)<g_2\left(\lambda_i\right)$; after the point $m$, $g_1\left(\lambda_i\right) > g_2\left(\lambda_i\right)$. (2) $g_1$ and $g_2$ have the same output norm.   
If we don’t assume they have the same norm, consider the following scenario: $g_1(\lambda_1)  = g_2(\lambda_1) - e < g_2(\lambda_1) $, and $g_1(\lambda_i)  = g_2(\lambda_i) + E >> g_2(\lambda_i) $, for $i \ge 2$, where $e$ is a very small value ($10^{-5}$) and $E$ is a very large value ($10^{5}$). In this case, assumption 1 is still satisfied. However, because $ E$ is an extremely large value, almost all of the frequencies are involved in the final representations, which are meaningless and contain no information. Therefore, we only compare two filters with the same norm.  

2. **Increasing the overall norm of the output values does not mean that the filter can fit better** because what really matters is the relative difference between high and low frequencies, not their absolute values. Consider we have a certain filter $g$, and then we multiply its output values by 2 to get $g’ = 2g$. Obviously, $||g’||_2^2 = 4||g||_2^2$. However, they have the same expressive power because the output results are just multiplied by a constant.   

3. We adopted these assumptions solely for the purpose of understanding which types of filters are favored by a graph with a specific homophily ratio. However, **in practical applications, we do not impose any restrictions on the norm of the spectral filter**. A filter that incorporates beneficial frequencies and discards harmful frequencies is likely to yield good results. This is proved by our experiments.

Reviewer comment (Experiments):
In the experimental setup, GCN is regarded as a spatial GNN method, which is not that necessary, in my opinion. To my knowledge, GCN is often analyzed as a simplification of Chebyshev polynomial approximator.

Author response:
Thanks for pointing this out! Yes, you are totally correct. GCN can be considered as both spatial and spectral methods. Actually, according to [1], most GNNs can be explained as both spatial and spectral. And this does not affect our analysis and results.

Reviewer comment (Reproducibility):
I am wondering how you calculate the homophilic level ($h$) when there is just a tiny fraction of nodes are labeled. Actually, this is a crucial factor for the explicit shape-aware regularization to work well, according to my understanding of this work.

Author response:
In the initial epoch, we compute the homophily ratio using the given training labels. Then, **in every subsequent epoch, we update this ratio by considering both the original and the predicted labels.** Table 6 compares the actual homophily ratios with estimated homophily ratios by our method. The similarity between these two sets of ratios indicates that our method can estimate the homophily ratio accurately.

Reviewer comment (Experiments):
I am curious about how you conduct HPO. The search space is supposed to be:

```Python
 lr = trial.suggest_categorical("lr", [0.005, 0.01, 0.05])
 temp_lr = trial.suggest_categorical("temp_lr", [0.005, 0.01, 0.05])
 dropout = trial.suggest_categorical("dropout", [0, 0.5])
 dprate = trial.suggest_categorical("dprate", [0, 0.5])
 wd = trial.suggest_categorical("wd", [0.0, 5e-4])
 gamma1 = trial.suggest_categorical("gamma1", [0, 1, 3, 5])
 gamma2 = trial.suggest_categorical("gamma2", [0, 1, 3, 5])
 gamma3 = trial.suggest_categorical("gamma3", [0, 1, 3, 5])
```

It seems that it is intractable to enumerate all possible choices. Instead, I run your code with an HPO toolkit (optuna) with 400 trials, where each trial returns the mean best valid accuracy as the feedback. Other hyper-parameters are kept unchanged (i.e., using your default value `K=5`, `L=2`, and `hidden=64`. The best trial on Cora, CiteSeer, Crocodile, and Gamer are 90.610, 78.797, 76.002, and 61.861, respectively. Obviously, it is not that promising to achieve the reported test accuracy for such configurations, and, as expected, the test accuracy on them are 87.948, 77.293, 74.037, and 61.712, respectively.

Author response:
2. Thanks for noticing this. Except that, the "dropout" and "dprate" should also be tuned, other parts are correct. You may increase the number of trials to get better results. **To ensure better reproductivity,** we have updated the code and done the following things:  
(1) We provide our virtual environment and optimal hyperparameters in scripts.sh to reproduce results in Table 1.  
(2) We give the code to search optimal hyperparameters using Optuna.  
(3) We provide the code to reproduce the results in the case study in Section 3.  
All of these are in the instructions in README.md (https://anonymous.4open.science/r/NewtonNet-8115/README.md). 

1. Note that different environments may lead to difference in results. Did you use the same environment as `requirements.txt`? Make sure you created the same virtual environment according to the `README.md` and use the same random seed.  

2. Different machines may also cause different results. We run our experiments on NVIDIA RTX A6000. Could you provide more details of your environments and machines?

3. To get results on a specific dataset, you don't need to run `bash scripts.sh`. 

For example, if you want to get results on Cora, you can run  
`python main.py --dataname cora --lr 0.05 --temp_lr 0.01 --K 5 --hidden 64 --weight_decay 0.0005 --dropout 0.3 --dprate 0.5 --gamma 0 --gamma2 1 --gamma3 3`


To get results on Crocodile, run  
`python main.py --dataname crocodile --lr 0.005 --temp_lr 0.005 --K 5 --hidden 64 --weight_decay 0.0005 --dropout 0.0 --dprate 0.1 --gamma 3 --gamma2 3 --gamma3 0`   

4. You can search your own optimal hyperparameters with 
```
python search_multi.py --dataname $DATASET
python search_binary.py --dataname $DATASET
```
, where `$DATASET` is the name of the dataset. 
  

5. We have noticed that some of your results are better than our reported results, while others are not as good as ours. We believe this may be caused by randomness across different environments and machines.

Since figures cannot be posted on Openreview, we would be more than happy to discuss further details after the code becomes publicly available.

Reviewer comment (Evaluation):
The effectiveness of the suggested approach might not be universally applicable, as its performance is contingent on the specific characteristics of the dataset, making it less versatile.

Author response:
In Table 1, we demonstrate that our method achieves state-of-the-art performance across 11 diverse datasets. These datasets span various domains (Citation, WebKB, Wikipedia, Social networks) and range from small to large scales. We believe this variety and the consistent performance across these datasets substantiate the effectiveness of our method beyond mere coincidence.

Reviewer comment (Experiments):
The process of fine-tuning hyperparameters could be a resource-intensive and time-consuming endeavor, potentially impeding the scalability of the proposed approach.

Author response:
1. We detail the experimental settings and the process for hyperparameter tuning in Section 6.1 and Appendix D.4. Once the search space is determined, methods like grid search or hyperparameter optimization tools such as Optuna can be utilized to identify the optimal hyperparameters.  
2. The time and space complexity of NewtonNet is analyzed in Appendix B.1, where we demonstrate the excellent scalability of our method. NewtonNet has a better scalability than GCN.

Reviewer comment (Theory):
The foundation of our approach hinges on the presumption that the graph Laplacian matrix adequately encapsulates the graph's inherent structure; however, this assumption may not hold true in all instances.

Author response:
As we discuss in Section 2, the graph Laplacian matrix $\mathbf{L}=\mathbf{I}-\mathbf{D}^{-1 / 2} \mathbf{A} \mathbf{D}^{-1 / 2}$ encapsulates all information from the adjacency matrix $\mathbf{A}$. This is the same across all graphs.

Reviewer comment (Evaluation):
Interpretability poses a concern, as the proposed approach lacks a straightforward means of explanation. This could curtail its utility in domains where interpretability is a paramount consideration.

Author response:
The focus of our paper is not on interpretability, but rather on exploring the relationship between homophily/heterophily and graph spectral learning.

Reviewer comment (Experiments):
The susceptibility of the proposed approach to data noise and outliers may undermine its performance when confronted with real-world datasets.

Author response:
As mentioned in response to RW1, our experiments are conducted on 11 varied real-world datasets. Our research does not specifically address noisy data or outlier detection; instead, it concentrates on examining the relationship between homophily/heterophily and graph spectral learning.

Reviewer comment (Experiments):
A significant limitation arises from the substantial volume of labeled data that the proposed approach necessitates to attain satisfactory performance, which can be impractical in situations where acquiring labeled data is a challenging and expensive endeavor.

Author response:
Our proposed method does not require a substantial volume of labeled data. The experiments illustrated in Figure 3 reveal that NewtonNet demonstrates a higher percentage of improvements under conditions of limited labels compared to scenarios with higher training ratios. This indicates that NewtonNet performs effectively even with a limited number of labels available.

Reviewer comment (Presentation):
On the choice of K. In the second line under Eq.7, the authors write that they set K=4 in this paper. According to Fig.2 and the description of experimental settings in Appendix D.4, you want to write K=5, right?

Author response:
1. Thanks for your careful reading. Yes, it’s a typo. We have revised the paper accordingly and submitted the new version.  
2. We appreciate your question and realize our explanation might not have been as clear as intended:  
(1) We assure you that **we did not use any prior knowledge from the test sets in our experiments**. We strictly follow the common practice of training the model on the training set, selecting the best model (hyperparameters) based on the validation set, and then evaluating and reporting the results on the test set.  
(2) **$K$ is indeed a hyperparameter.** Regarding the hyperparameter $K$,  to simplify our approach and reduce the search space, we initially did not tune $K$ and chose $K=5$ for the results presented in Table 1. However, to analyze the impact of $K$, we varied it over the set {2, 5, 8, 11, 14} as shown in Fig. 8. This investigation revealed that $K=5$ yields the best outcomes for the Cora and Chameleon datasets. However, it doesn’t mean that $K=5$ is the best choice for other datasets. Additionally, we conducted further experiments on amazon-ratings, which shows that $K=8$ provides better results. Generally, as $K$ increases from a small number the performance will first increase and then drop when $K$ is too large. A number of $K$ between 5 to 8 works well.  

|                  | **amazon-ratings** |
|:----------------:|:------------------:|
|  NewtonNet (K=2) |      54.43±0.2     |
|  NewtonNet (K=5) |      55.06±0.5     |
|  NewtonNet (K=8) |    **55.24±0.4**   |
| NewtonNet (K=11) |      54.81±0.3     |
| NewtonNet (K=14) |      53.47±0.6     |

(3) To give more facts, if you refer to our code (Line 13 of https://anonymous.4open.science/r/NewtonNet-8115/config.py), it can be seen that $K$ is set as a hyperparameter.

Reviewer comment (Experiments):
In Appendix E.1, the authors conduct a sensitivity analysis on K, and find that NewtonNet's performances on the Cora and Chameleon datasets peak at K=5. Such an analysis is weird since the accuracies on test datasets are in fact used as prior knowledge. A valid way is to set K as a hyperparameter.

Author response:
A1. We do define $K$ as a hyperparameter. The only thing is that we fixed $K=5$ to reduce the search space and simplify the settings. However, according to your suggestions, we re-ran the experiments with $K$ varying among [2, 5, 8, 11, 14] and found some improved results. Due to limited time, we only conducted experiments on Cora, Citeseer, Pubmed, Chameleon, Squirrel, and Crocodile. The results show that on Cora, Citeseer, and Chameleon, the best results are still achieved with $K=5$ and are the same as Table 1. However, on Pubmed, Squirrel, and Crocodile, better results were observed since we expanded the search space. These are summarized in the following table.  

| **Dataset** | **Acc (%)** | **Optimal $K$** |
|:-----------:|:-------:|:-------------:|
|    Pubmed   |  90.28  |       2       |
|   Squirrel  |  63.37  |       14      |
|  Crocodile  |  76.80  |       8       |

Due to time constraints, we conducted only a limited number of trials with Optuna. We believe to achieve better results with more trials. We intend to complete all the experiments and update the results in the next version of our work. Thank you for your valuable suggestions.

Reviewer comment (Experiments):
On the weak-supervised experiments. According to Ref.1, the Chameleon and Squirrel datasets are problematic in that they use duplicated nodes. This would be problematic, especially in weak-supervised settings. I am curious how this experiment would perform on other datasets.

Author response:
**RW2:** We run experiments on your suggested datasets and show the results here. We will add this table and reference in the camera-ready version.

|           | **roman-empire** | **amazon-ratings** |  **tolokers** |
|:---------:|:----------------:|:------------------:|:-------------:|
|    MLP    |     66.33±0.9    |      48.85±0.2     |   75.36±0.4   |
|    GCN    |     61.96±0.8    |      49.90±0.1     |   77.13±0.4   |
|   Mixhop  |    _74.38±0.5_   |      51.44±0.6     | **80.62±0.5** |
|   APPNP   |     73.50±0.7    |     _51.80±0.6_    |   75.02±0.2   |
|   GPRGNN  |     71.09±0.3    |      47.54±0.5     |   73.83±0.6   |
| ChebNetII |     70.78±0.5    |      46.24±0.7     |   71.86±0.6   |
| NewtonNet |   **75.34±0.3**  |    **55.24±0.4**   |  _79.01±0.7_  |

Reviewer comment (Theory):
Can the regularization be used for other polynomial filtering functions?

Author response:
1. We would like to kindly remind the reviewer that we discussed this on page 7, under the section “Compared with other approximation and interpolation methods,” and in Appendix C. To make it more clear, **shape-aware regularization is applicable exclusively to interpolation methods, as it necessitates direct control over the output values of interpolated points** (specifically, $\mathbf{t}$ in Equation 7). Only interpolation methods directly reflect the output values of the polynomial. Methods like GPRGNN, ChebNet, and BernNet, which use approximation methods, are not compatible with shape-aware regularization.  

2. **While ChebNetII employs interpolation methods, its interpolated points are fixed**, limiting our ability to tailor any interpolated points to fit various shapes of spectral filters. This limitation led us to propose Newton interpolation for learning the spectral filter, which is not only an interpolation method but also allows for the random selection of interpolated points.  

3. NewtonNet can choose any points as interpolated points, including Chebyshev nodes. However, when conducting experiments, we found that Chebyshev nodes with regularization did not yield as satisfactory results as equispaced points. In fact, the performance was similar to ChebNetII. Consequently, we opted for equispaced points in our experiments.  

4. Additionally, **the concern of Runge's phenomenon primarily arises with high polynomial powers.** According to Wikipedia[1], “Runge's phenomenon is a problem of oscillation at the edges of an interval that occurs when using polynomial interpolation with polynomials of high degree …” Our experiments demonstrate that NewtonNet with K=5 delivers good performance across various datasets. Opting for **a lower polynomial degree not only mitigates the Runge phenomenon but also enhances the speed and efficiency of our method.**

Reviewer comment (Theory):
The authors explain in Appendix E.1 that the reason for their choice of K to be 5 (typically set as 10 or tuned in a range in other polynomial filters) is that when K is larger, the polynomial becomes susceptible to the Runge phenomenon according to He et al. (2022). Since the Runge phenomenon is caused by the selection of equal-paced interpolation points, why not use Chebyshev nodes for interpolation?

Author response:
1. We would like to kindly remind the reviewer that we discussed this on page 7, under the section “Compared with other approximation and interpolation methods,” and in Appendix C. To make it more clear, **shape-aware regularization is applicable exclusively to interpolation methods, as it necessitates direct control over the output values of interpolated points** (specifically, $\mathbf{t}$ in Equation 7). Only interpolation methods directly reflect the output values of the polynomial. Methods like GPRGNN, ChebNet, and BernNet, which use approximation methods, are not compatible with shape-aware regularization.  

2. **While ChebNetII employs interpolation methods, its interpolated points are fixed**, limiting our ability to tailor any interpolated points to fit various shapes of spectral filters. This limitation led us to propose Newton interpolation for learning the spectral filter, which is not only an interpolation method but also allows for the random selection of interpolated points.  

3. NewtonNet can choose any points as interpolated points, including Chebyshev nodes. However, when conducting experiments, we found that Chebyshev nodes with regularization did not yield as satisfactory results as equispaced points. In fact, the performance was similar to ChebNetII. Consequently, we opted for equispaced points in our experiments.  

4. Additionally, **the concern of Runge's phenomenon primarily arises with high polynomial powers.** According to Wikipedia[1], “Runge's phenomenon is a problem of oscillation at the edges of an interval that occurs when using polynomial interpolation with polynomials of high degree …” Our experiments demonstrate that NewtonNet with K=5 delivers good performance across various datasets. Opting for **a lower polynomial degree not only mitigates the Runge phenomenon but also enhances the speed and efficiency of our method.**

Reviewer comment (Experiments):
In practice, incorporating homophily information might offer the most significant benefits when the datasets are of limited size. For sufficiently large datasets, the graph filters will be fine-tuned automatically according to the graph homophily level.

Author response:
1. Our approach does not explicitly incorporate homophily information. We rely solely on the graph structure and predicted labels to estimate the homophily ratio.   
2. Could you kindly point out that is there any reference for the claim “For sufficiently large datasets, the graph filters will be fine-tuned automatically according to the graph homophily level”? To our understanding, there is no clear evidence that graph filters adapt more effectively to larger graphs. Even if the graph is large, if there are not sufficient labels, we still do not know much about the homophily/heterophily information.  In such scenarios, the graph filter's fine-tuning is hindered by insufficient information. Our experiments, illustrated in Figure 3, demonstrate that under conditions of scarce labels, NewtonNet shows a more significant improvement compared to situations with higher training ratios on both large and small datasets. For instance, in the Chameleon dataset, NewtonNet shows a notable improvement over BernNet by 28.2% when the training ratio is 0.1; however, this improvement is 3.2% when the training ratio is 0.6.  
However, **if we do not provide guidance to the spectral filter, it will learn some undesired information on both small and large graphs.** Actually, letting the graph filter be fine-tuned automatically on any graph is exactly what we aim to achieve by proposing shape-aware regularization. The experiments in Table 1 and Figure 3 validate that NewtonNet successfully learns beneficial frequencies autonomously on both small and large datasets, supporting our approach.

Reviewer comment (Presentation):
I recommend that the complexity analysis in Appendix B.1 be included in the main body of the paper.

Author response:
Thanks for your suggestion. In the new version submitted, we added a complexity analysis part in Section 4.