Paper: Rethinking the Polynomial Filter of GNNs via Graph Information Activation Theory (ICLR-2024)

Reviewer comment (Theory):
Weakness of Thm. 3.1. Thm 3.1 is the key argument of the equivalence of the "expressive power." However, this is rather weak, since the K linear independent components does not warrant the downstream machine learning algorithms performance. Maybe a set of K eigenvectors has the same expressive power, some random projection onto the K space may have the same expressive power -- but I believe that the "expressive power" we want to know in this context may be more nuanced one.

Author response:
**A1:** Absolutely! In our latest version, we're planning to revamp Theorem 3.1 into a foundational algebraic fact, condensing its content while maintaining its role as an introductory piece that offers a new perspective on problem understanding. The intention behind Theorem 3.1 is to provide a fundamental intuition from an algebraic standpoint. Its primary aim is to challenge the conventional approach of blindly seeking polynomial bases within the existing polynomial field. We aim to illustrate that such selection isn't necessary and might even prove inefficient.

However, our pivotal theorem is Theorem 3.2, which, to the best of our knowledge, holds significant value. Unlike Theorem 3.1, which focuses on expressive ability, Theorem 3.2 delves into the convergence rate of GNNs. This theorem is instrumental in showcasing the efficiency disparities among different networks, offering insights into their convergence rates.

Reviewer comment (Experiments):
Insufficient comparison with exiting filters. In the page 3 of (Defferrard et al, 2016) the complexity of the polynomial filter is discussed. The point of 2) is actually discussed, and also, the Krylov subspace is expected to serve as a better filter, and materialized in [i]. Thus, the authors may want to compare with [i] experimentally and theoretically. Also, since the filter perspective is well-studied in [ii], the authors may want to compare as well. See the questions for the connection between this paper and [ii].

Author response:
**A3:** That sounds like a comprehensive plan! Conducting comparative experiments and delving into theoretical analysis can indeed be time-consuming, especially considering code framework and version compatibility. Your thorough approach will likely yield insightful and well-rounded results. If you need any further assistance or guidance during your experiments or analysis, feel free to reach out!

Reviewer comment (Writing):
The presentation quality of this paper needs to be improved, e.g., there are many symbol errors in the proof (See Questions for detail). Although these errors do not affect the final results, rigidity is one of the most fundamental requirements for scientific papers.

Author response:
**A1:** You are quite right, there are many low-level errors among our proof, which have been corrected, thank you very much! PS: we shall enhance our presentation in the final version and check our paper more rigorously.

Reviewer comment (Theory):
The linear expression ability and convergence rate are not enough to reveal the essential differences between different graph filters. Although authors have shown that different graph polynomial filters have the same expressive ability, their performances may also vary greatly, depending on their implementation approaches (e.g., ChebNet [1] and ChebNetII [2]). It is still unclear what the relation is between this expressive ability and node classification performance.

Author response:
**A2:** Your inquiry is extremely relevant and thought-provoking! We've taken into account all the weaknesses and questions you've highlighted and have incorporated the materials you mentioned as our reference.

Initially, our focus was on establishing general convergence analysis in the polynomial filtering problem. However, we acknowledge a certain gap between this analysis and downstream tasks due to theoretical constraints. We've made efforts to introduce fewer constraints to bridge this gap and genuinely value your input in this regard.

Regarding the jocobiconv article's mention of removing nonlinear activation functions under specific constraints without affecting expressive ability: it highlights the consistency between linear and nonlinear GNNs in terms of expressive ability. While we can impose constraints on weight matrices like W1 and W2 to create a strictly convex optimization problem, this might lead to an excess of constraints deviating from real-world scenarios. We plan to further explore this aspect in our future research endeavors.

Additionally, dynamic analysis within optimization problems is a significant research direction we're actively pursuing. If we make any new discoveries, we'll promptly share them with you! Lastly, we've considered the validity of Theorem 3.1 and plan to reclassify it as an algebraic fact in our latest version, recognizing its current weakness as a theorem.

Reviewer comment (Theory):
In the implementation of these spectral GNNs, the raw features are commonly first fed into an MLP and then transformed by the filters, namely \( Z = \gamma(L)\sigma(\sigma(XW_1)W_2) \). Due to the involvement of two parametric matrices \( W_1, W_2 \) and the non-linear activation function \( \sigma(\cdot) \) in the forward process, the optimization of these spectral GNNs is non-convex, which could not be directly simplified as a convex problem in Eq. (4). Analyzing the training dynamics [3,4] of the model could be a more applicable approach.

Author response:
**A2:** Your inquiry is extremely relevant and thought-provoking! We've taken into account all the weaknesses and questions you've highlighted and have incorporated the materials you mentioned as our reference.

Initially, our focus was on establishing general convergence analysis in the polynomial filtering problem. However, we acknowledge a certain gap between this analysis and downstream tasks due to theoretical constraints. We've made efforts to introduce fewer constraints to bridge this gap and genuinely value your input in this regard.

Regarding the jocobiconv article's mention of removing nonlinear activation functions under specific constraints without affecting expressive ability: it highlights the consistency between linear and nonlinear GNNs in terms of expressive ability. While we can impose constraints on weight matrices like W1 and W2 to create a strictly convex optimization problem, this might lead to an excess of constraints deviating from real-world scenarios. We plan to further explore this aspect in our future research endeavors.

Additionally, dynamic analysis within optimization problems is a significant research direction we're actively pursuing. If we make any new discoveries, we'll promptly share them with you! Lastly, we've considered the validity of Theorem 3.1 and plan to reclassify it as an algebraic fact in our latest version, recognizing its current weakness as a theorem.

Reviewer comment (Experiments):
The optimization approaches (SGD or Adam) also have significant impacts on the performances of spectral GNNs, which should be considered in the analysis.

Author response:
**A2:** Your inquiry is extremely relevant and thought-provoking! We've taken into account all the weaknesses and questions you've highlighted and have incorporated the materials you mentioned as our reference.

Initially, our focus was on establishing general convergence analysis in the polynomial filtering problem. However, we acknowledge a certain gap between this analysis and downstream tasks due to theoretical constraints. We've made efforts to introduce fewer constraints to bridge this gap and genuinely value your input in this regard.

Regarding the jocobiconv article's mention of removing nonlinear activation functions under specific constraints without affecting expressive ability: it highlights the consistency between linear and nonlinear GNNs in terms of expressive ability. While we can impose constraints on weight matrices like W1 and W2 to create a strictly convex optimization problem, this might lead to an excess of constraints deviating from real-world scenarios. We plan to further explore this aspect in our future research endeavors.

Additionally, dynamic analysis within optimization problems is a significant research direction we're actively pursuing. If we make any new discoveries, we'll promptly share them with you! Lastly, we've considered the validity of Theorem 3.1 and plan to reclassify it as an algebraic fact in our latest version, recognizing its current weakness as a theorem.

Reviewer comment (Experiments):
The heterophilic graph datasets used seem to be out-of-date. It has been shown that results obtained on these datasets are not reliable. The authors are encouraged to evaluate on the datasets presented in [5].

Author response:
**A3:**  
You're welcome! Thank you for sharing the experimental data for the heterogeneous graph. Here are the updated results:

    squrriel: 53.51 ± 1.18
    twitch-gamer: 64.47 ± 0.57
     arxiv-year: 46.74±0.32 

Your dedication to incorporating the latest heterogeneous datasets is commendable. If you have any further questions or need additional assistance, feel free to ask.

Reviewer comment (Theory):
There are many symbol errors in the proof of Lemma 3.1 and Lemma 3.2. The denominator of Eq. (16) should be \( \left(\sum_{i=1}^n a_i\right)^2 \). In Eq. (17), which is claimed to be a quadratic function of \( x \), the correct form should be:  
\[
f(x) = \left( \sum_{i=1}^n \frac{a_i}{\lambda_i} \right) x^2 - \frac{\lambda_1+\lambda_n}{\sqrt{\lambda_1 \lambda_n}}x + \left( \sum_{i=1}^n \lambda_i a_i \right).
\]  
In Eq. (19), the third term in the first bracket should be \( \sum_{i=2}^{n-1} \lambda^{-1}_i a_i \). Also, Eq. (39) should be corrected as \( \left(A^{k+1}\right)_{ij} = \sum_{r=1}^n \left(A^k\right)_{ir} A_{rj} \). Although these typos do not affect the final result, they should be corrected to avoid unnecessary misunderstandings for other readers.

Author response:
**A4:** Sincerely admire your patience and careful reading! We will definitely check it more rigorously.

Reviewer comment (Theory):
The motivation and the advantages of the GIA theory are not clear. It is unclear what performance gain the positive and proper activation could bring, whether there is any connection between generalization and positive (or proper) activation, and what extra insights the GIA theory could bring.

Author response:
**A5:** Your problems are really commendable. We realize that our network is not related enough to equation 4, which also leads to the lack of intuitive understanding of this formula. So we add the relevant theoretical part to show how SimpleNet can be written as a form of equation 4. Additionally, the correlation analysis of SimpleNet's condition number in the latest version of our paper promises to illustrate the performance benefits derived from positive and proper activation.

Regarding the GIA's key insight of introducing a new polynomial design framework, while it's a pivotal contribution, we acknowledge the concerns about its solidity and clarity. To address this, our upcoming paper revision will include additional analyses aimed at strengthening and elucidating the framework. The main contents of these analyses are outlined as follows:

Utilizing a unified optimization function framework, we intend to conduct an extensive analysis of the GIA theory. This will involve theoretically substantiating the roles played by positive and negative incentives and their effects on both generalization and the mitigation of over-smoothing.

Referencing the paper Interpreting and Unifying Graph Neural Networks with An Optimization Framework we will employ a unified optimization framework. This framework can effectively analyze various models, represented by the formulation: $O=\min_Z { \zeta \Vert F_1Z-F_2H\Vert_F^2+\xi \text{tr}(Z^\top \hat{L}Z) }$. Furthermore, we will demonstrate that concurrently designing $F_1$ and $F_2$ is equivalent to designing one or the other.

Additionally, we will conduct an in-depth analysis of the SimpleNet model featured in our paper. Specifically, the model equations are described as $F_1=I$ and $F_2=\alpha(I+\hat{A})+\beta(I-\hat{A})$. Notably, the $F_2$ term comprises a weighted sum of positive and non-positive incentives. By combining insights from this analysis with our existing work, we aim to draw the following conclusions:

From the perspective of spectral domain expression, positive activation can capture low-frequency information, while negative activation can acquire high-frequency information. This organic combination in $F2$ allows for the integration of high and low-frequency information, thereby enhancing the model's generalization ability on the broader spectrum.

The flexibility of the weight coefficient influences the network's capacity to alleviate over-smoothing. In our network, the $F2$ term introduces two degrees of freedom—$\alpha$ and $\beta$ (positive and negative)—significantly enhancing our ability to mitigate over-smoothing compared to other networks that lack or possess only one weighted degree of freedom.

The adaptability of the weight coefficient further enhances the model's expressive and generalization capabilities.

Reviewer comment (Novelty):
The proposed fixedMono and learnedMono seem to be variants of JKNet [6], where different hidden features of different neighborhood ranges are combined. The only difference is in the way of combining these features: the authors adopt a linear combination, while Xu et al. use LSTM or max pooling. The authors should clarify this and compare with JKNet.

Author response:
**A6:** Your question holds significant importance. The utilization of linear transformations (weighted sums) is deeply influenced by our interpretation of the problem. We approach problem-solving through the lens of polynomial filtering, employing weighted sums between each term of the polynomial.

Viewed through all kinds of layer-aggregation mechanisms highlighted in the JKnet article, these weighted sums can be likened to a form of basic attention—devoid of any constraints on attention scores ($s_v^{(l)}$). This concept of unbounded attention holds a fundamental unity. The diverse aggregation methodologies proposed by JKnet are highly insightful, motivating us to further refine our model for experimentation. We aim to conduct a more comprehensive analysis of this issue from an aggregation perspective, particularly in comparison to JKNet. The outcomes of our analytical and experimental endeavors will be detailed in the forthcoming version of our paper.

Reviewer comment (Theory):
The authors assert that GNNs can be conceptualized as optimizers, and can be mathematically formulated in a uniform optimization form as depicted in Equation 4. However, this claim appears to be unfounded. As elucidated in [1], only PPNP and APPNP align with the representation provided by Equation 4.

Author response:
**A1:** Thank you for addressing the query about Eq.(4), which is originally proposed in [1] article, our paper and [2], provides a more comprehensive analysis on this formula. Furthermore, [3] offers a theoretical analysis of the implementation of this formula in BernNet.

Acknowledging that our network was initially not adequately linked to this formula, resulting in a lack of intuitive understanding, we've incorporated additional theoretical insights. These additions elucidate how SimpleNet can be represented in the form of Eq.(4). Additionally, the revision of this manuscript will include a correlation analysis of the condition number of SimpleNet. This analysis aims to establish a clearer connection and understanding between our network architecture and Eq.(4).

[1]Thomas Navin Lal Jason Weston Bernhard Sch¨olkopf Dengyong Zhou, Olivier Bousquet. Learning with local and global consistency. Neural Information Processing Systems, 2003.

[2] Meiqi Zhu, Xiao Wang, Chuan Shi, Houye Ji, and Peng Cui. Interpreting and unifying graph neural
networks with an optimization framework. In Proceedings of the Web Conference 2021, pp.1215–1226, 2021.

[3] Mingguo He, Zhewei Wei, Hongteng Xu, et al. Bernnet: Learning arbitrary graph spectral filters via
bernstein approximation. Advances in Neural Information Processing Systems, 34:14239–14251,
2021.

Reviewer comment (Novelty):
The so-called Graph Information Activation theory posited by the authors is essentially a reintroduction of graph coloring.

Author response:
**A2:** You've brought up an intriguing point regarding this theory! While graph coloring represents a general information aggregation framework, its direct application to polynomial filter design in GNNs requires more refined and specific theories for a comprehensive study of related problems. If your assertion holds, it suggests that not all message passing frameworks undergo a reintroduction akin to GIA's approach.

The crux lies in GIA's definition of a succinct message passing framework within the field. GIA introduces the concept of positive, non-positive, and proper components within this framework. These concepts serve as a foundation for a systematic analysis of existing polynomial filters. Additionally, leveraging this theory, we propose a model that demonstrates exceptional performance and computational advantages. Hence, we firmly believe in the merit and significance of this theory.

Reviewer comment (Experiments):
The test datasets comprising Cora, Citeseer, Pubmed, Computers, and Photos are too small, thus rendering the assertion that GNN FixedMono outperforms BernNet less convincing. I recommend that the authors evaluate GNN FixedMono and BernNet using the Open Graph Benchmark.

Author response:
**A3:** Thank you for your valuable feedback on the dataset. However, the FixedMono network isn't our final proposal. Its purpose was mainly to empirically demonstrate that designing the filter solely based on polynomial properties leads to limited gains. This motivated the development of the GIA framework and the subsequent proposal of the SimpleNet model below. Regarding the experiment you requested, here are the results:

    Squirrel: 52.11 ± 0.78

    Twitch-gamer: 64.07 ± 0.77

    Arxiv-year: 46.34 ± 0.52

These results are comparable to the Bernet dataset.

Reviewer comment (Evaluation):
This paper omits an analysis of SimpleNet concerning the over-smoothing issue.

Author response:
**A4:** It's a meaningful question! In our upcoming paper revision, we aim to delve deeper into this inquiry using a unified optimization framework. The primary objectives of this analysis are as follows:

Utilizing a unified optimization function framework, we intend to conduct an extensive analysis of the GIA theory. This will involve theoretically substantiating the roles played by positive and negative incentives and their effects on both generalization and the mitigation of over-smoothing.

Referencing the paper Interpreting and Unifying Graph Neural Networks with An Optimization Framework we will employ a unified optimization framework. This framework can effectively analyze various models, represented by the formulation: $O=\min_Z { \zeta \Vert F_1Z-F_2H\Vert_F^2+\xi \text{tr}(Z^\top \hat{L}Z) }$. Furthermore, we will demonstrate that concurrently designing $F_1$ and $F_2$ is equivalent to designing one or the other.

Additionally, we will conduct an in-depth analysis of the SimpleNet model featured in our paper. Specifically, the model equations are described as $F_1=I$ and $F_2=\alpha(I+\hat{A})+\beta(I-\hat{A})$. Notably, the $F_2$ term comprises a weighted sum of positive and non-positive incentives. By combining insights from this analysis with our existing work, we aim to draw the following conclusions:

From the perspective of spectral domain expression, positive activation can capture low-frequency information, while negative activation can acquire high-frequency information. This organic combination in $F2$ allows for the integration of high and low-frequency information, thereby enhancing the model's generalization ability on the broader spectrum.

The flexibility of the weight coefficient influences the network's capacity to alleviate over-smoothing. In our network, the $F2$ term introduces two degrees of freedom—$\alpha$ and $\beta$ (positive and negative)—significantly enhancing our ability to mitigate over-smoothing compared to other networks that lack or possess only one weighted degree of freedom.

The adaptability of the weight coefficient further enhances the model's expressive and generalization capabilities.

Reviewer comment (Experiments):
Why consider adding the term $\sum^{K\_{1}}\_{i=0}\alpha\_{i}(2\mathbf{I}-\mathbf{L})^{i}$ and the term $\sum^{K\_{2}}\_{j=0}\beta\_{j}\mathbf{L}^{j}$ instead of concatenating them?

Author response:
**A5:** Thank you for your very constructive question. In specific segments of GNN research, there's a departure from utilizing the message passing framework.  Instead, researchers directly combine the Laplacian matrix and feature matrix. For instance, when utilizing the superimposed network as input, it might be feasible to directly concatenate matrices such as L and 2I-L in these domains.

However, when it comes to the polynomial domain, implementing the concatenation method you mentioned isn't clear to us. In polynomial filtering, an n×n polynomial matrix is required, and we're uncertain about how concatenation would apply in this context. Our decision to use the addition operation is elaborated on in section 3.4 of our work. The addition operator allows for the independent decoupling of positive and non-positive components, thereby enhancing training performance.