Paper: Rethinking the Smoothness of Node Features Learned by Graph Convolutional Networks (ICLR-2024)

Reviewer comment (Theory):
Although seemingly making sense, “normalized smoothness”, as measured by s defined in eq.(4) is not valid for interpretation. It disconnects node smoothness from model performance, making it carry no insights into practice and thus it is meaningless to study. For instance, consider a graph with two classes of nodes: one class having feature values of 1, and the other class with feature values of -1. In this case, a linear classifier would have perfect classification performance. If we consistently add one to each node's feature, the differences among node features would not change, and if we apply a classifier again to classify based on the new features, the performance would not change either — we are basically shifting all the node features by the same value and the bias term of a classifier can easily accommodate that. Such a phenomenon is well justified by the unnormalized metrics such as conventional Dirichlet energy because it would remain the same before and after we add the same value to each node. However, the normalized smoothness metric s proposed in this paper would get larger and larger, indicating that the node features are getting "smoother" and "smoother". Given the above concern and the established research on the effects of ReLU and LeakyReLU under unnormalized smoothness [5, 27] (citations provided by the paper), this paper provides very little new theoretical insight.

Author response:
We thank the reviewer for the review. However, **we strongly disagree with your highly unethical and insulting comments, which falsify the truth**. We want to stress that respecting truth is the ethics of being a reviewer. Criticizing another's paper by falsifying the truth is scientific misconduct. It is a severe offense that can damage the reputation of other scientists and undermine public trust in science.

 Before addressing your comments, we would like to point out that: 

- Your criticism that “studying the normalized smoothness of GCN features is meaningless” is rootless, unethical, and insulting.
In particular, studying an existing smoothness notion and an existing important open problem is not a weakness.

- ***Evidence for studying normalized smoothness:*** The correlation between the relative magnitude $||{\bf z}\_{\mathcal{M}^\perp}||/||{\bf z}\_\mathcal{M}||$ -- a quantity that is closely related to the normalized smoothness -- and classification accuracy was first empirically studied in Figure 4 by Oono & Suzuki (ICLR 2020). Moreover, Cai & Wang (arXiv:2006.13318) pointed out that studying normalized smoothness is an open and important problem. Again, ***we believe that studying an open and important problem, pointed out in the pioneering paper, as part of our paper is not a weakness.***

---

Below, we address your comments in detail.

---

**Studying the normalized smoothness is a weakness**

**Reply:** ***We respectfully but strongly disagree that studying an existing smoothness notion and an existing open problem as part of our work is a weakness.*** 

The importance of studying normalized smoothness is supported by evidence. ***Empirical evidence:*** The correlation between the relative magnitude $||{\bf z}\_{\mathcal{M}^\perp}||/||{\bf z}\_\mathcal{M}||$ -- a quantity that is closely related to the normalized smoothness -- and classification accuracy was first empirically studied in Figure 4 by Oono & Suzuki (ICLR 2020). While they demonstrate a strong correlation between classification accuracy and $||{\bf z}\_{\mathcal{M}^\perp}||/||{\bf z}\_{\mathcal{M}}||$, their results neither refute nor conclusively establish that performance improves with an increasing $||{\bf z}\_{\mathcal{M}^\perp}||/||{\bf z}\_\mathcal{M}||$. Despite this, we share the common belief that the ratio $||{\bf z}\_{\mathcal{M}^\perp}||/||{\bf z}\_\mathcal{M}||$ or the normalized smoothness $||{\bf z}\_\mathcal{M}||/||{\bf z}||$ correlates with performance. In contrast to intentionally making features unsmooth, our proposed smoothness control term aims to enable the model to control the smoothness of node features automatically. ***Evidence:*** As the reviewer mentioned, the paper [5] states that **“Finally, analyzing the real over-smoothing effect, i.e., the Rayleigh quotient $\frac{tr(X^T\tilde{\Delta}X)}{|\|X||_2^2}$ for deep GNNs is still an open and important question.”** The Rayleigh quotient is the normalized smoothness. 

In the following three paragraphs, we would like to clarify three substantial mistakes in your review:

First, ***In the footnote of the first page of the paper [5]***, Cai & Wang state that “Strictly speaking, over-smoothing is a misnomer. As we will show, what is decreasing is $tr(X^T\tilde{\Delta}X)$, not the real smoothness $\frac{tr(X^T\tilde{\Delta}X)}{|\|X||_2^2}$ of graph signal $X$.” Since the distance of node features to the eigenspace $\mathcal{M}$ and the Dirichlet energy are two equivalent seminorms, we rephrase that “[5] points out that over-smoothing -- measured by the distance of node features to the eigenspace $\mathcal{M}$ or the Dirichlet energy -- is a misnomer, and the real smoothness of a graph signal should be characterized by a normalized smoothness, e.g., normalizing the Dirichlet energy by the magnitude of the features.”

Reviewer comment (Evaluation):
The improvement over stronger baselines (GCNII and EGNN) in Table 1 is limited in most cases, which raises doubts about the overall effectiveness of SCT.

Author response:
***Improvement over GCNII and EGNN:*** GCNII is a state-of-the-art GCN-style model and EGNN can be considered as GCNII with constraints on the Dirichlet energy of the node features. The proposed smoothness control term (SCT) can consistently improve the state-of-the-art baseline models, especially on heterophilic graphs. For homophilic graphs, the improvement is more significant on larger-scale datasets -- Coauthor-Physics and Ogbn-arxiv -- than on the smaller-scale datasets -- Cora, Citeseer, and PubMed. In particular, Table 1 in our paper shows that SCT can often improve the accuracy of GCNII and EGNN by more than 1% on Coauthor-Physics and Ogbn-arxiv. As far as we are aware, the performance of GCNII-SCT and EGNN-SCT is the state-of-the-art accuracy for GCN-style models; see Section 5.1 and Appendix A.2 in reference [1] listed below for details.

In Table 1, though the accuracy gain is not huge in absolute values for GCNII and EGNN in some tasks, they are significant in almost all cases. To show the statistical significance of the improvement using SCT for smaller-scale datasets, we have included statistical significance tests for accuracy improvement for Cora, PubMed, and Citeseer. In particular, we train each model with 100 different random initializations using the optimal hyperparameters; the mean and standard deviation of the test accuracy are provided in Table 8 of the revision. Moreover, the t-test results on the statistical significance of the accuracy improvement are provided in Table 9 of the revision.

[1] Zhou et al. Dirichlet Energy Constrained Learning for Deep Graph Neural Network, NeurIPS 2021.

Reviewer comment (Evaluation):
Could the authors provide standard deviations for the experimental results in Table 1 (particularly for the baseline methods) and Table 2?

Author response:
**Reply:**
***Standard deviation:*** In our submission, the standard deviations of the models with SCT have been reported when they exist under the benchmark experimental setting. The standard deviation only exists for the experiments of Coauthor-Physics and Ogbn-arxiv in Table 1 and exists for all experiments in Table 2. We stress the benchmark experimental setting here again that we use the fixed public split for the Citation datasets and the particular splits from reference [37] in the revision in the Coauthor-Physics and Ogbn-arxiv datasets. For the baseline models without SCT, the standard deviation results are not reported in the benchmark papers [7,42] in the revision. The code released by the authors of the benchmark papers does not include the hyperparameters used, and we cannot get the exact accuracies by running their released code. For appropriate comparisons, we use the accuracy reported in the benchmark papers.

Reviewer comment (Novelty):
The paper seems to heavily rely on previous works [1,2] for its theoretical results. A more independent theoretical contribution or a clearer delineation of the novel aspects beyond the referenced works would strengthen the paper's originality.

Author response:
**Reply:** Our theoretical results significantly differ from those in [1,2]. Recall that Oono & Suzuki [1] and Cai & Wang [2] focus on showing that over-smoothing is inevitable for GCNs with ReLU or Leaky ReLU activation function. At the same time, Cai & Wang also point out that studying the normalized smoothness is an open and important problem. In contrast, our theory is novel in the following sense: 1) In Section 3, we establish ***a geometric characterization*** of how ReLU and Leaky ReLU affect the smoothness of node features, showing that ***the smoothness of node features are controllable*** by adjusting the projection of node features onto the eigenspace $\mathcal{M}$ for the first time. Our theory also informs a practical smoothness control term. 2) In Section 4, we show that adjusting the projection of node features onto the eigenspace $\mathcal{M}$ results in ***disparate effects*** on the normalized and unnormalized smoothness. 3) We study an open problem proposed by Cai & Wang. These novelties have been listed in Section 1.1 of our paper.

Reviewer comment (Experiments):
The empirical validation could be broadened to enhance the robustness of the findings. Incorporating a more diverse array of datasets and experimenting with various network architectures would be beneficial. Notably, the largest non-homophily graph used is the Squirrel dataset, which consists of 5201 nodes. Exploring larger and more varied graphs could provide more comprehensive insights.

Author response:
**Reply:** The existing experiments in our paper provide a direct comparison against the baseline models by providing a comprehensive study for each task proposed by the baseline papers. The 10 datasets we studied contain a diverse number of nodes, edges, features, and classes as shown in Table 4 in the revision. These tasks sufficiently cover a broad range of applications for GNNs.

To provide broader empirical results of our techniques on large non-homophily graphs, we have included results for the Roman-empire heterophilic graph in Appendix D.5 Table 13 of the revision. The Roman-empire graph has a large diameter which supports using deeper models. We utilize the architectures from Section 6.2 which are equipped with ReLU activation functions and perform hyperparameter tuning following the same procedure. For the reviewer's ease, we have also presented the results below.

| Layers   	| 2             	| 4             	| 16            	| 32            	|
|--------------|-------------------|-------------------|-------------------|-------------------|
| GCN 	 	| 84.48 ± 0.53  	| 84.00 ± 0.71  	| 74.56 ± 0.75  	| 14.32 ± 1.02  	|
| GCN-SCT  	| 85.37 ± 0.56  	| 84.08 ± 0.71  	| 82.58 ± 0.57  	| 79.6 ± 0.49   	|
| GCNII 	| 83.49 ± 0.36  	| 83.43 ± 0.40  	| 80.01 ± 0.50  	| 76.52 ± 0.70  	|
| GCNII-SCT	| 85.44 ± 0.56  	| 85.08 ± 0.24  	| 81.44 ± 0.35  	| 77.28 ± 0.55  	|

Reviewer comment (Presentation):
The presentation of results in Table 1 could be improved for clarity and comprehensiveness. Enhancing the table's presentation could make the findings more accessible and effectively communicate the research outcomes.

Author response:
**Reply:** Thank you for your suggestion, and we have modified the Table 1 in the revision to make it more clear and comprehensive.

Reviewer comment (Evaluation):
The benifit of proposed method is somewhat weak when nerual network is deep.

Author response:
**Reply:** GCNII is a state-of-the-art GCN-style model and EGNN can be considered as GCNII with constraints on the Dirichlet energy of the node features. The proposed smoothness control term (SCT) can consistently improve the state-of-the-art baseline models, especially on heterophilic graphs. For homophilic graphs, the improvement is more significant on larger-scale datasets -- Coauthor-Physics and Ogbn-arxiv -- than on the smaller-scale datasets -- Cora, Citeseer, and PubMed. In particular, Table 1 in our paper shows that SCT can often improve the accuracy of GCNII and EGNN by more than 1% on Coauthor-Physics and Ogbn-arxiv. As far as we are aware, the performance of GCNII-SCT and EGNN-SCT is the state-of-the-art accuracy for GCN-style models; see Section 5.1 and Appendix A.2 in reference [1] listed below for details.

In Table 1, though the accuracy gain is not huge in absolute values for GCNII and EGNN in some tasks, they are significant in almost all cases. To show the statistical significance of the improvement using SCT for smaller-scale datasets, we have included statistical significance tests for accuracy improvement for Cora, PubMed, and Citeseer. In particular, we train each model with 100 different random initializations using the optimal hyperparameters; the mean and standard deviation of the test accuracy are provided in Table 8 of the revision. Moreover, the t-test results on the statistical significance of the accuracy improvement are provided in Table 9 of the revision.

[1] Zhou et al. Dirichlet Energy Constrained Learning for Deep Graph Neural Network, NeurIPS 2021.

Reviewer comment (Writing):
The writing of this paper needs further improvement, as the theoretical part is not very easy to understand. It is recommended to add a summary of notations and optimize the formatting.

Author response:
**Reply:** We appreciate the reviewer’s suggestion. In Appendix A of the revision, we have added a table to summarize all notations -- we did not put the table in the main text due to page limit while Section 1.3 briefly overviews the notations used in our paper.

To make the theoretical part more straightforward to understand, we have further polished Section 1.1 to make it a better navigation of our theoretical results. In particular, our theoretical results are twofold: 1) In Section 3, we establish a geometric characterization of how ReLU or Leaky ReLU affects the smoothness of the node features learned by GCN, showing that the smoothness of node features are controllable by adjusting the projection of input onto the eigenspace $\mathcal{M}$. 2) In Section 4, we comprehensively examine how adjusting the projection of input to $\mathcal{M}$ affects the smoothness of node features measured by two existing smoothness notions.

We have also gone through the paper and done our best to improve the formatting. We are happy to include any further suggestions from the reviewer to improve the paper.

Reviewer comment (Evaluation):
Experiments show that the performance improvement of SCT on deep models like GCNII and EGNN is relatively marginal.

Author response:
**Reply:** GCNII is a state-of-the-art GCN-style model and EGNN can be considered as GCNII with constraints on the Dirichlet energy of the node features. The proposed smoothness control term (SCT) can consistently improve the state-of-the-art baseline models, especially on heterophilic graphs. For homophilic graphs, the improvement is more significant on larger-scale datasets -- Coauthor-Physics and Ogbn-arxiv -- than on the smaller-scale datasets -- Cora, Citeseer, and PubMed. In particular, Table 1 in our paper shows that SCT can often improve the accuracy of GCNII and EGNN by more than 1% on Coauthor-Physics and Ogbn-arxiv. As far as we are aware, the performance of GCNII-SCT and EGNN-SCT is the state-of-the-art accuracy for GCN-style models; see Section 5.1 and Appendix A.2 in reference [1] listed below for details.

In Table 1, though the accuracy gain is not huge in absolute values for GCNII and EGNN in some tasks, they are significant in almost all cases. To show the statistical significance of the improvement using SCT for smaller-scale datasets, we have included statistical significance tests for accuracy improvement for Cora, PubMed, and Citeseer. In particular, we train each model with 100 different random initializations using the optimal hyperparameters; the mean and standard deviation of the test accuracy are provided in Table 8 of the revision. Moreover, the t-test results on the statistical significance of the accuracy improvement are provided in Table 9 of the revision.

[1] Zhou et al. Dirichlet Energy Constrained Learning for Deep Graph Neural Network, NeurIPS 2021.

Reviewer comment (Writing):
Further improvement in writing is needed.

Author response:
**Reply:** Thank you for your comment. We have further improved the writing in the revision to make the paper clearer, and we are happy to include any further suggestions from the reviewer.

Reviewer comment (Theory):
The proof about the contraction property applies only to ReLU and Leaky ReLU. Therefore, this theoretical analysis does not broaden the applicable GNN types.

Author response:
**Reply:** Understanding whether over-smoothing happens or not and how to control the smoothness when other activation functions are used is an interesting problem. However, the over-smoothing has only been theoretically justified for GCN using ReLU and Leaky ReLU activation functions. 

To be more specific, the established proofs of over-smoothing by Oono & Suzuki (ICLR 2020) and Cai & Wang (arXiv:2006.13318) rely on the contraction property of ReLU and Leaky ReLU. In particular, Oono & Suzuki (ICLR 2020) show over-smoothing for GCN with ReLU by using the contraction property of ReLU and point out that it is hard even to extend this result to Leaky ReLU. Cai & Wang prove the contraction property of ReLU or Leaky ReLU by using Dirichlet energy to characterize the smoothness of node features. Importantly, they highlight that other activation functions, such as Sigmoid and Tanh, may not exhibit the contraction property. 

Nevertheless, our work builds a geometric understanding of the contract property of ReLU and Leaky ReLU, which further informs a practical approach to control the smoothness of the learned node features by GCN with ReLU or Leaky ReLU activation function.

Reviewer comment (Writing):
P.5, Definition 4.1: $\|\boldsymbol{z}_{\mathcal{M}}^{(i)}\|$ is undefined.

Author response:
**Reply:** We have defined $||\mathbf{z}_{\mathcal{M}}^{(i)}||$ in Definition 4.1 in the revised paper.

Reviewer comment (Novelty):
P.7: If I understand correctly, the $\beta_l$ parametrization comes from the work of GCNII. If this paper references it, the paper should be cited explicitly.

Author response:
**Reply:** We have cited the GCNII paper right after the $\beta_l$ parameterization in the revision.