Paper: PolyFormer: Scalable Graph Transformer via Polynomial Attention (ICLR-2024)

Reviewer comment (Theory):
My understanding is that this method uses intra-node attention, where attention is calculated for a set of tokens associated with *each node* and not between nodes. The only cross-node information is from the tokens when they are constructed with A and L. This does not seem like a very efficient way of passing information between nodes, as the cross-node information is hardcoded in the tokens.

Author response:
Our method employs intra-node attention. This choice is based on the following considerations:

**Scalability**: In computer vision image classification, Transformer-based models usually calculate attention between patches of an image, rather than between entire images. Similarly, in natural language processing sentence classification, these models typically focus on attention between nodes within a sentence. This approach facilitates training and inference on mini-batches, thus avoiding scalability issues when dealing with a large number of images or sentences. Inspired by this, we design polynomial tokens for each node and compute attention between these tokens within a node for node classification tasks, thereby enhancing the model's scalability. Conversely, cross-node attention entails considerable computational costs, resulting in a complexity of $O(n^2d)$, where $n$ is the number of nodes, and $d$ is the dimensionality of the node representations. Such complexity is impractical for large-scale graph node classification tasks. Even with efficient attention mechanisms like [1], the complexity remains at least $O(nd^2)$, which is prohibitive for large-scale graphs with, for example, 100 million nodes. In short, intra-node attention enables Graph Transformer models with great scalability.

**Model Performance**: While cross-node information can provide flexibility in learning node-pair relationships for graph Transformers, this flexibility can sometimes detract from practical performance, as noted in [2]. To incorporate structural information and boost performance, NAGphormer and [3, 4] have introduced various positional encodings. However, these additions incur extra costs, limiting scalability to large graph datasets and potentially failing to achieve ideal model performance. Although cross-node attention seems more efficient in transmitting information between nodes, most real-world graph data resemble small-world networks. In such networks, any node can be reached from any other node within a limited number of steps. Therefore, even with a truncated order of polynomial $h_k(\mathbf{A}, \mathbf{L})$, the polynomial tokens contain sufficient information for intra-node attention. Besides, our intra-node attention mechanism, PolyAttn, is theoretically equivalent to a node-wise polynomial filter in the spectral domain, as demonstrated in Theorem 3.1:
$$
\mathbf{Z}\_{i,:} = \sum\_{k=0}^K \mathbf{H}'^{(i)}\_{k,:} = \sum\_{k=0}^K \alpha\_k^{(i)}(g\_k(\mathbf{P})\mathbf{X})\_{i,:} = \left(\mathbf{U}h^{(i)}(\mathbf{\Lambda})\mathbf{U}^\top \mathbf{X}\right)_{i,:}
$$
This kind of filter [5, 6, 7, 8, 9], which hardcode node information via polynomial, perform well in node classification tasks and outperforms some Graph Transformer models which employ cross-node attention. In fact, our node classification experiments (Tables 4 and 5) demonstrate that our model using hardcore encoding delivers outstanding results in most cases as well.

In summary, our intra-node attention mechanism, termed "PolyAttn", is designed considering both scalability and model performance.

Reviewer comment (Evaluation):
The experiments do not show a large improvement over baseline methods. For example, in Table 2, the ChebNetII baseline performs very well and is not that far off PolyAttn (Cheb). In fact, it outperforms PolyAttn (Mono) on some tasks. This is a bit disappointing since PolyAttn (Mono) is touted as being interpretable, but PolyAttn (Cheb) is the method that outperforms baselines.

Author response:
(1) ''ChebNetII performs well and is not far off polyattn(cheb).''

Below are the $R^2$ scores / the sum of squared error (denoted as SE) results from Table 2. A higher $R^2$ score indicates better performance, while a lower SE indicates better performance as well. Both the $R^2$ score and SE of PolyAttn (Cheb) significantly outperform those of ChebNetII. Specifically, comparing ChebNetII, PolyAttn (Cheb) reduced SE by up to **3500 times** in the Low-high pass setting.

| Model (5k para) | Mixed low-pass    | Mixed high-pass   | Mixed band-pass   | Mixed rejection-pass | Low&high-pass     | Band&rejection-pass |
| --------------- | ----------------- | ----------------- | ----------------- | -------------------- | ----------------- | ------------------- |
| GPRGNN          | 0.9978/0.9784     | 0.9806/0.7846     | 0.9088/1.2977     | 0.9962/1.8374        | 0.8499/35.3719    | 0.9876/13.4890      |
| PolyAttn (Mono) | *0.9994/0.2550*   | *0.9935/0.2631*   | 0.9030/1.3798     | 0.9971/1.4025        | *0.9997/0.0696*   | *0.9992/0.8763*     |
| ChebNetII       | 0.9980/0.8991     | 0.9811/0.7615     | *0.9492/0.7229*   | *0.9982/0.8610*      | 0.8494/35.4702    | 0.9870/14.1149      |
| PolyAttn (Cheb) | **0.9997/0.1467** | **0.9960/0.0148** | **0.9945/0.0782** | **0.9996/0.1949**    | **0.9999/0.0118** | **0.9999/0.0416**   |

(2) ''CheNetII outperforms PolyAttn (Mono) on some tasks (in table 2).''

We conducted experiments on synthetic datasets with GRPGNN and PolyAttn (Mono), which uses the Monomial basis, as well as ChebNetII and PolyAttn (Cheb), which uses the Chebyshev basis. It is observed that, **under the same basis, PolyAttn shows a significant improvement** over polynomial graph neural networks (PolyAttn (Mono) outperforms GPRGNN in 5 out of 6 datasets; PolyAttn (Cheb) outperforms ChebNetII in all 6 datasets). 

Next, we explain why ChebNetII outperforms PolyAttn (Mono) on some tasks. As indicated by [7], the Chebyshev basis is theoretically superior to the Monomial basis in fitting filter functions. Consequently, due to the inherent limitations of the Monomial basis, PolyAttn (Mono) exhibits a slight disadvantage when fitting complex filter functions such as Mixed band-pass filters and Mixed rejection-pass filters. **Nevertheless, PolyAttn (Mono) still significantly outperforms ChebNetII in the remaining four tasks.**

Reviewer comment (Experiments):
Some experimental settings are not reasonable.

Author response:
The comparison of model complexity, given a similar parameter count, **was conducted in line with the settings used in prior works** [1, 2, 10]. However, evaluating the cost-effectiveness at optimal performance settings is also a critical aspect. To address this, we extended complexity comparison experiments under the optimal conditions on the dataset Roman-empire. The results are summarized as follows:

| Model       | Acc.       | Prep. Time(s) | Train. (ms/epoch) | Infer. (ms/epoch) | Max GPU Mem. (MB) |
| ----------- | ---------- | ------------- | ----------------- | ----------------- | ----------------- |
| Transformer | 65.29±0.47 | 0             | 1721              | 739               | 11457             |
| NAGphormer  | 74.45±0.48 | 42.12         | 426               | 214               | 3063              |
| Specformer  | 69.94±0.34 | 1120.73       | 3931              | 7233              | 42813             |
| PolyFormer  | 78.89±0.39 | 0.24          | 1079              | 733               | 3319              |

In optimal settings, PolyFormer exhibits a marginally slower training and inference time compared to NAGphormer, which may be attributable to design choices such as order-wise MLPs. However, given that PolyFormer **significantly outperforms** NAGphormer in preprocessing speed and demonstrates considerable performance improvements, this trade-off is deemed acceptable. Regarding GPU memory overhead, PolyFormer and NAGphormer are virtually identical. It is important to note that our evaluations were conducted under full batch settings. Theoretically, a reduced GPU overhead is attainable with minibatch-based processing. Overall, PolyFormer stands as a highly competitive option, considering accuracy, preprocessing speed, training and inference times, and GPU memory requirements.

Reviewer comment (Novelty):
The proposed PolyFormer actually combines many existing techniques. The Polynomial Token could be regarded as an extension of Hop2Token[1] with different propagation strategies. Moreover, utilizing tanh() to compute the attention score and the node-shared attention bias also have been proposed and successfully implemented in previous works [2] and [3] respectively.

Author response:
Our work is **motivated** by polynomial filters, a method that conveys both theoretical guarantees and good empirical performance in graph neural networks, particularly performs well in tasks like node classification. Despite their strong theoretical and empirical performance, polynomial filters are usually node-unified [1, 2, 3, 4, 5, 6], limiting their expressive power. It is challenging to extend these to node-wise filters directly. Fortunately, with our specially designed model, we achieve an equivalence to more expressive node-wise filters in the spectral domain. On the other hand, most existing graph transformers, including NAGphormer, add spectral information via Laplacian eigendecomposition into the spatial domain, which is costly. However, as our model arises directly from the spectral domain, it eliminates the need for this high-cost addition of spectral information, while still achieving outstanding performance due to the powerful expressive power of spectral filters. 

More specifically, our model differs from NAGphormer in various aspects:

1. **Node Token Design Perspective.** 
   As highlighted in our Introduction, our polynomial tokens are conceptualized from a spectral perspective, designed to convey spectral information through various polynomial bases, such as the Monomial basis, Chebyshev basis (used in our paper), and even the Bernstein basis (in response to Reviewer tq9c). In contrast, NAGphormer's spatially-oriented Hop2Token generates node tokens by aggregating information from different hops in the spatial domain. Even with the Monomial basis, our PolyAttn enables the model to capture information directly in the spectral domain, avoiding the high-cost process of adding additional spectral information, a step necessary in NAGphormer and other popular Graph Transformers [7, 8, 9].
2. **Elaborately Designed PolyAttn vs. Vanilla Self-Attention.**
   NAGphormer employs vanilla self-attention for encoding tokens derived from Hop2Token. In contrast, we introduce specially designed PolyAttn for our proposed polynomial tokens. **While some design elements (e.g., tanh(), node-shared attention bias) have been used in prior work, it is important to note that our primary focus is not on innovating new activation functions for attention scores or creating novel attention biases.** Instead, our focus is on developing an attention mechanism specifically tailored for polynomial tokens, establishing a direct connection with polynomial filters in the spectral domain.

With the polynomial tokens in the spectral domain and the tailored PolyAttn, our model effectively addresses two significant challenges in previous works:

1. **Graph Transformer models** that add spectral information (e.g., NAGphormer and  [7, 8, 9]) enhance expressive power but require extensive time and space. In contrast, our model, which originates from the spectral domain, already possesses powerful expressive capabilities. Without the need for additional spectral information, which incurs significant costs, our model demonstrates great scalability and performance, capable of handling graphs with up to **100 million nodes**, as illustrated in Experiment 4.2.
2. **Polynomial filters** (e.g., [1, 2, 3, 4, 5, 6]) are typically node-unified, learning the same filter for all nodes and thereby limiting their expressive power. Our model operates as a **node-wise** filter, adaptively learning distinct filters for different nodes, as theoretically substantiated in Theorem 3.1 and Proposition 3.1, and empirically demonstrated in Experiment 4.1.

In summary, inspired by the concept of spectral filters, our approach uniquely crafts node tokens and attention mechanisms from a spectral viewpoint, contrasting with NAGphormer's spatial focus. Leveraging polynomial tokens and PolyAttn, our model not only theoretically aligns with but also empirically aligns the expressive node-wise filters. These properties help to address challenges in both Graph Transformers and polynomial filters. Based on this elaborate design, our model offers a desirable balance between scalability and expressiveness.

Reviewer comment (Experiments):
More challenging datasets need to be added into the experiments to validate the effectiveness of the proposed PolyFormer, including Actor, Squirrel, Chameleon and ogb-products. The first three are challenging heterophilic datasets and the last one is the representative large-scale graph dataset. Note that, Squirrel and Chameleon should consider the filtered versions proposed in [4].

Author response:
For the Actor dataset, we employed a 60%/20%/20% split for training, validation, and testing, respectively. For the other three datasets, we utilized the provided splits. Below are the corresponding results on 10 runs.

|                   | Actor      | Chameleon (Filtered) | Squirrel (Filtered) | ogbn-products |
| :---------------- | ---------- | -------------------- | ------------------- | :------------ |
| GPRGNN            | 40.41±0.72 | 42.28±2.87           | 41.09±1.18          | 76.00±0.16    |
| ChebNetII         | 42.05±0.97 | 42.67±1.43           | 41.22±0.37          | 75.05±0.14    |
| Specformer        | 42.24±0.93 | 42.82±2.54           | 40.60±0.53          | -             |
| NAGphormer        | 40.88±1.37 | 40.36±1.77           | 39.79±0.84          | -             |
| PolyFormer (Mono) | 42.29±0.74 | 46.86±1.61           | 42.56±0.96          | 76.97±0.16    |
| PolyFormer (Cheb) | 43.29±0.80 | 45.35±2.97           | 41.83±1.18          | 76.93±0.40    |

According to the table above, PolyFormer outperforms the baseline models on these challenging heterophilic datasets and large-scale datasets.

Reviewer comment (Experiments):
The authors conduct the complexity comparison in Section 4.3. They keep the total number of parameters approximately the same for each model. However, the settings of this comparison are not reasonable since it is not the true parameter setting for each model to achieve the best performance on the dataset. Compared to NAGphormer, the proposed PolyFormer introduces order-wise MLP to initialize the query and key matrix, which inevitably increases the training cost. There is a question about the true training cost of each model on its optimal parameter setting.

Author response:
The comparison of model complexity, given a similar parameter count, **was conducted in line with the settings used in prior works** [1, 2, 10]. However, evaluating the cost-effectiveness at optimal performance settings is also a critical aspect. To address this, we extended complexity comparison experiments under the optimal conditions on the dataset Roman-empire. The results are summarized as follows:

| Model       | Acc.       | Prep. Time(s) | Train. (ms/epoch) | Infer. (ms/epoch) | Max GPU Mem. (MB) |
| ----------- | ---------- | ------------- | ----------------- | ----------------- | ----------------- |
| Transformer | 65.29±0.47 | 0             | 1721              | 739               | 11457             |
| NAGphormer  | 74.45±0.48 | 42.12         | 426               | 214               | 3063              |
| Specformer  | 69.94±0.34 | 1120.73       | 3931              | 7233              | 42813             |
| PolyFormer  | 78.89±0.39 | 0.24          | 1079              | 733               | 3319              |

In optimal settings, PolyFormer exhibits a marginally slower training and inference time compared to NAGphormer, which may be attributable to design choices such as order-wise MLPs. However, given that PolyFormer **significantly outperforms** NAGphormer in preprocessing speed and demonstrates considerable performance improvements, this trade-off is deemed acceptable. Regarding GPU memory overhead, PolyFormer and NAGphormer are virtually identical. It is important to note that our evaluations were conducted under full batch settings. Theoretically, a reduced GPU overhead is attainable with minibatch-based processing. Overall, PolyFormer stands as a highly competitive option, considering accuracy, preprocessing speed, training and inference times, and GPU memory requirements.

Reviewer comment (Experiments):
The polynomial bases are somewhat limited to Monomial and Chebyshev. More advanced bases could be explored.

Author response:
Your suggestion is constructive. In fact, our model framework can be extended to other polynomial bases. Accordingly, we have included results for PolyFormer with the Bernstein basis [1] and the optimal basis [2]. The specific results are as follows:

|              | GPRGNN     | ChebNetII  | NAGphormer | Specformer | PolyFormer (Bern) | PolyFormer (Opt) | PolyFormer (Mono) | PolyFormer (Cheb) |
| ------------ | ---------- | ---------- | ---------- | ---------- | ----------------- | ---------------- | ----------------- | ----------------- |
| Tolokers     | 77.25±0.61 | 76.83±0.53 | 81.57±0.44 | 80.42±0.55 | 84.32±0.59        | 83.15±0.49       | 84.00±0.45        | 83.88±0.33        |
| Roman-empire | 74.08±0.54 | 72.70±0.30 | 74.45±0.48 | 69.94±0.34 | 77.64±0.33        | 77.15±0.33       | 78.89±0.39        | 80.27±0.39        |

Compared to the baseline models, PolyFormer models based on different basis all perform well.

Reviewer comment (Experiments):
Comparisons to very recent graph neural network methods are missing.

Author response:
We have supplemented with the latest Graph Transformer model GOAT [3] to enrich our baseline. The specific results are as follows:

|                   | CS         | Physics    | Tolokers   | Roman-empire |
| ----------------- | ---------- | ---------- | ---------- | ------------ |
| GPRGNN            | 95.26±0.15 | 97.84±0.59 | 77.25±0.61 | 74.08±0.54   |
| ChebNetII         | 96.33±0.12 | 97.25±0.78 | 79.23±0.43 | 74.64±0.39   |
| Specformer        | 96.07±0.10 | 97.70±0.60 | 80.42±0.55 | 69.94±0.34   |
| NAGphormer        | 94.41±0.35 | 97.18±0.36 | 81.57±0.44 | 74.45±0.48   |
| GOAT              | 95.12±0.21 | 97.09±0.24 | 83.26±0.52 | 72.30±0.48   |
| PolyFormer (Mono) | 96.30±0.14 | 98.16±0.28 | 83.66±0.63 | 78.47±0.38   |
| PolyFormer (Cheb) | 96.47±0.14 | 98.01±0.37 | 83.41±0.47 | 79.62±0.22   |

Reviewer comment (Experiments):
For large graphs, is recomputing the polynomial tokens for each minibatch really efficient?

Author response:
During the preprocessing stage before training and inference, our model precomputes and stores polynomial tokens for all nodes. 

Once training and inference begin, these precomputed polynomial tokens are directly loaded for each minibatch, **eliminating the need for their recomputation．** 

From the preprocessing stage to the phases of training and inference, **polynomial tokens for all nodes are computed only once.** Consequently, this approach is efficient for large graphs in practical applications.

Reviewer comment (Novelty):
The novelty is incremental compared with NAGphormer, PolyFormer adds MLP on each hop aggregation of NAGphormer, to some extent.

Author response:
**A2:** We distinguish our work from NAGphormer based on motivation, model design, and theory and empirical performance.

**(1) Motivation.** Inspired by the strong performance and theoretical guarantees of spectral filters, we aim to build a bridge between Graph Transformers and expressive spectral filters, a feat not achieved by NAGphormer. Our design of polynomial tokens and PolyAttn even establishes an equivalence to more expressive node-wise filters.

**(2) Detailed Design Differences between PolyFormer and NAGphormer.**

*a. Spectral-based Node Tokens.* NAGphormer's Hop2Token aggregates information from different hops in the spatial domain. In implementation, NAGphormer necessitates additional structural encoding based on Laplacian eigendecomposition to supplement spectral information, leading to high costs. Conversely, our polynomial tokens, designed in the spectral domain, inherently contain spectral information, obviating the need for additional positional encoding.

*b. Specialized PolyAttn.* Unlike NAGphormer, which uses vanilla self-attention for tokens from Hop2Token, we have tailored PolyAttn for polynomial tokens. This customization in our approach goes beyond incorporating order-wise MLPs. We intricately design PolyAttn, including order-wise MLPs, tanh() activation to enable both positive and negative attention scores, and aligning the Value matrix to the polynomial basis. This establishes an equivalence between our model and polynomial filters in the spectral domain, endowing the Polyformer with advanced expressive capabilities compared to NAGphormer, which is spatially oriented.

**(3) Theory and Empirical Performance.** In Section 3.3, we theoretically demonstrate that our model acts as an expressive node-wise filter. Furthermore, we analyze how multi-head PolyAttn functions as multi-channel filters, which are more expressive. These properties are not possessed by NAGphormer. In addition, we analyze how the vanilla attention in NAGphormer could limit the expressive power. In Section 4.1, we empirically prove that our model can learn node-wise filters on both synthetic and real-world datasets, outperforming NAGphormer. Specifically, the proposed Graph Transformer PolyFormer outperforms NAGphormer on homophilic, heterophilic, and large-scale graphs.

In conclusion, motivated by spectral filters, we directly design node tokens and attention from a spectral perspective, whereas NAGphormer is mainly spatially oriented. With the help of polynomial tokens and PolyAttn, we both theoretically and empirically demonstrate that our model achieves equivalence to expressive node-wise filters. Without the need for additional spectral supplementation, which is implemented in NAGphormer, our model outperforms baselines on various datasets.

Reviewer comment (Theory):
Theorem 1 seems not informative somehow.

Author response:
**A3:** As mentioned in A2, we established a connection between our model and polynomial filters based on the proposed polynomial tokens and PolyAttn. It's important to note that most existing polynomial GNNs [1, 2, 3, 4, 5] learn a **node-unified** filter, i.e., all nodes share the same filter, limiting the model's expressiveness. For instance, when nodes follow different filter patterns (e.g., high-pass for some nodes and low-pass for others), node-unified filters struggle to match such patterns. In contrast, our model acts as a **node-wise** filter. Specifically, Theorem 3.1 **theoretically constructs the bridge between PolyAttn** ($\sum\_{k=0}^{K} \mathbf{H}'^{(i)}\_{k,:}$ in Theorem 3.1) **and the node-wise filter** ($\sum\_{k=0}^K \alpha^{(i)}\_k (g\_k{(\mathbf{P})} \mathbf{X})\_{i,:}$ in Theorem 3.1).

Reviewer comment (Theory):
Extending the first bullet point in the weaknesses section, could the authors explain why the proposed "polynomial token" with PolyAttn is a spectral method, not a spatial method, based on the polynomial type listed in Table 1?

Author response:
**A4:** We further elaborate based on Theorem 3.1. Combining equations (8) and (2) from the paper, we obtain: 

$\mathbf{Z}\_{i,:} = \sum\_{k=0}^K \mathbf{H}'^{(i)}\_{k,:} =\sum_{k=0}^K \alpha_k^{(i)}(g\_k(\mathbf{P})\mathbf{X})\_{i,:} = \left(\mathbf{U}h^{(i)}(\mathbf{\Lambda})\mathbf{U}^\top \mathbf{X}\right)\_{i,:}$.

Here, $\mathbf{Z}\_{i,:}$ is the representation of the node, $\mathbf{H}'^{(i)}\_{k,:}$ is the representation obtained by polynomial token via PolyAttn, $\mathbf{U}$ and $\mathbf{\Lambda}$ are the eigenvectors and eigenvalues corresponding to the Laplacian matrix respectively, $\mathbf{X}$ is the original representation of the node. This illustrates that the node representation learned with polynomial tokens via PolyAttn is derived in the spectral domain, i.e. $\mathbf{Z}\_{i,:} = \left(\mathbf{U}h^{(i)}(\mathbf{\Lambda})\mathbf{U}^\top \mathbf{X}\right)\_{i,:}$.

For different types of polynomial tokens used, as shown in Table 1, the expression of $ h(\mathbf{\Lambda}) $ varies. However, the learning of node representation can consistently be expressed in the form of the formula above [1, 3], indicating learning in the spectral domain.

Reviewer comment (Theory):
How to understand \(h(\lambda)\) in Figure 1 with \(\alpha\) in Eq. 8? Moreover, during the learning process, how to obtain and interpret the low-pass and high-pass?

Author response:
**A5:** We answer the question from two aspects. 

**(1) The relationship between $ h(\lambda) $ in Figure 1 and $\alpha$ in Eq. 8.** 

In Fig. 1, $ h(\lambda) $ denotes a mapping function applied to the eigenvalues of the normalized Laplacian matrix, where $\lambda \in [0,2]$. We define $ h^{(i)}(\lambda) $ as the function corresponding to node $ i $. In Eq. 8, $\alpha_k^{(i)}$, for $ k=\{0,\cdots, K\} $, represents the set of polynomial coefficients for node $ i $. The relationship between $ h^{(i)}(\lambda) $ and $\alpha_k^{(i)}$ is determined by the choice of polynomial tokens:

* Monomial basis: $ h^{(i)}(\lambda) = \sum_{k=0}^{K}\alpha_k^{(i)} \cdot (1-\lambda)^k $
* Chebyshev basis: $ h^{(i)}(\lambda) = \sum_{k=0}^{K} \alpha_k^{(i)} \cdot T_k(\lambda) $, where $ T_k(\lambda) $ denotes the Chebyshev polynomial of $ T_0(\lambda) = 1, T_1(\lambda) = \lambda - 1 $, and $ T_k(\lambda) = 2(\lambda - 1) \cdot T_{k-1}(\lambda) - T_{k-2}(\lambda) $ for $ k \geq 2 $

**(2) Interpreting and obtaining the low-pass and high-pass filters.** 

In simpler terms, a low-pass filter $ h(\lambda) $ preserves smaller eigenvalues $\lambda$ of the Laplacian and attenuates larger ones, while a high-pass filter does the opposite. For instance, $ h(\lambda) = 1 - \frac{\lambda}{2}, \lambda \in [0,2] $ is an example of a linear low-pass filter, while $ h(\lambda) = \frac{\lambda}{2} $ exemplifies a linear high-pass filter. Low-pass filtering tends to smooth the node signals in the graph, whereas high-pass filtering emphasizes the differences in signals [6].

During the learning process, filters are obtained by learning the trainable parameters in PolyAttn (e.g. $\mathbf{W}_Q$ and $\mathbf{W}_K$). This leads to the generation of appropriate attention scores and, subsequently the acquisition of polynomial filter coefficients. The detailed relationship between the attention scores in PolyAttn and the polynomial filter coefficients is illustrated in the proof of Theorem 3.1. With the acquisition of polynomial filter coefficients $\alpha_k^{(i)}$ for node $i$, we have the corresponding filter $h^{(i)}(\lambda)$ .

Furthermore, we verify that our model successfully learns effective filters on both synthetic and real-world datasets, as discussed in sections 4.1.1 and 4.1.2. Notably, our model is capable of learning distinct filters for different nodes, effectively acting as a node-wise filter. For example, Figure 2(b) illustrates that our model learns **low-pass** filters for certain nodes and **high-pass** filters for others in a synthetic dataset, yielding optimal results as indicated in Table 2. Similarly, on real-world datasets, our model adapts by learning diverse filters for each node, demonstrating more expressive capabilities compared to node-unified filters, as presented in Table 3.

In summary, our model is capable of both theoretically and empirically learning diverse filters for nodes (e.g., low-pass filter, high-pass filter).