Paper: Polynormer: Polynomial-Expressive Graph Transformer in Linear Time (ICLR-2024)

Reviewer comment (Theory):
It is inappropriate to claim that GTs and GNNs has limited polynomial expressivity (in section 3.1 and appendix C), since the non-linearity layers are not negligible. In [1] it is shown that without softmax GTs cannot represent GNNs. And in [2] Transformers are proved to be universal approximators on sequences with the softmax layer as key component. Can you discuss the polynomial expressivity of GTs and GNNs with non-linearity layers? And since [2] proves that Transformers are universal approximators, do GTs have $\infty$-polynomial expressivity?

Author response:
Firstly, we would like to note that it is important to analyze the polynomial expressivity of prior graph models without nonlinear functions, as it emphasizes the need for the activation functions in those models to achieve reasonable accuracy. This highlights Polynormer’s advantage in polynomial expressivity, which allows Polynormer to achieve promising results without the use of activation functions. 

While integrating nonlinear functions into prior graph models implicitly represents a high-degree polynomial, the polynomial coefficients are fixed since those nonlinear functions are typically not learnable. As a result, these models still cannot be parametrized to adaptively learn a polynomial that consists of specific monomials. In contrast, Polynormer learns the high-degree polynomial with coefficients controlled by trainable attention scores, enabling it to expressively represent various polynomial functions.

While it is true that GTs with positional encoding are universal approximators, they require unbounded depth to achieve this property. In practice, however, GTs typically have fixed and small depths (e.g., 2~3) and thus do not have $\infty$-polynomial expressivity.

Reviewer comment (Theory):
The concept of graph is defined by edge connections. And the definition of polynomial expressivity is completely ignorant of graph structure, comparing to WL-test expressivity. From my opinion, polynomial expressivity defined here should be used to model expressivity on sets, not graphs. What is the motivation of modeling the polynomial expressivity of graph models?

Author response:
Graph models essentially learn node representations by aggregating features from a set of nodes, which can be viewed as a multiset pooling problem with auxiliary information about the graph structure [Hua-NeurIPS22; Baek-ICLR21]. Thus, the polynomial expressivity measures how well a graph model incorporates the auxiliary structural information. For instance, given a node $i$ and its $1$-hop neighbors $j$ and $k$, a model A that learns $x_i+x_j+x_k+x_ix_jx_k$ is more expressive than a model B that only learns $x_i+x_j+x_k$, since A incorporates the third-degree monomial $x_ix_jx_k$ that captures third-order structures (either a triangle or a path formed by nodes $i$, $j$, $k$).

Reviewer comment (Experiments):
The O(N+E) complexity claim should be supported by more experiments, like a training time (VRAM) – graph size plot on synthetic random graphs with different sizes.

Author response:
Thanks for the suggestion. In our revision, we added the training time and memory usage of Polynormer on synthetic graphs with various sizes in Appendix J.1,  which confirms that Polynormer has linear complexity with respect to the number of nodes/edges in a graph. Moreover, we further provide the complete profiling results of Polynormer and GT baselines in terms of training time and memory usage on three large graphs in Appendix J.2. The results showcase that Polynormer consistently ranks the top 3 fastest models with relatively low memory usage. This demonstrates the scalability advantage of Polynormer over prior GT models.

Reviewer comment (Theory):
Can authors discuss more about the relationship between WL-test expressivity and polynomial expressivity? For example, is the proposed Polynormer strictly more powerful than 1-WL-GNNs?

Author response:
This is a great question. Indeed, it is interesting to see if Polynormer can be more powerful than 1-WL GNNs under the WL hierarchy. In response to this question, we have included Appendix K in our revision and made reference to it in Section 3.3 (marked in blue). 

Concretely, we show that by properly designing the weight matrix B in Equation 4, Polynormer can be more expressive than 1-WL GNNs. Initially, we set $B=\mathbf{1} \beta^T$ to build our Polynormer model. Note that the constant vector $\mathbf{1}$ here is essentially the eigenvector $v_1$ that corresponds to the smallest graph Laplacian eigenvalue. This motivates us to design $B = v_2 \beta^T$, where $v_2$ denotes the eigenvector corresponding to the second smallest (i.e., first non-trivial) Laplacian eigenvalue. To differentiate between these two choices of designing $B$, we denote the Polynormer of using $v_1$ and $v_2$ by Polynormer-v1 and Polynormer-v2, respectively. We then show that Polynormer-v2 is able to distinguish non-isomorphic graphs such as circular skip link graphs, which are known to be indistinguishable by the 1-WL test as well as 1-WL GNNs. 

Through empirical comparison of Polynormer-v2 with Polynormer-v1 in Table 8 (Appendix K), however, we observe similar accuracy achieved by both models on realistic node classification datasets. Hence, we implement Polynormer based on Polynormer-v1 in practice to avoid computing the Laplacian eigenvector, and leave the improvement of Polynormer-v2 to our future work.

Reviewer comment (Theory):
Why and how could polynomial expressivity improve model performance? The point was not clear.

Author response:
A2: High polynomial expressivity indicates that the model is capable of learning high-order interactions among node features. For instance, given a node $i$ and its $1$-hop neighbors $j$ and $k$. Conventional GNNs capture first-order (linear) feature interactions such as $x_i, x_j, x_j+x_k$ during neighbor aggregation. In contrast, a polynomial-expressive model captures additional higher-order interactions such as $x_i^2x_j, x_ix_jx_k$, etc. Notably, the high-order interactions implicitly correspond to high-order graph structures (e.g., $x_ix_jx_k$ corresponds to either a triangle or a path formed by nodes $i, j, k$.), which are known to be crucial for many graph problems.

Reviewer comment (Writing):
How is the degree of polynomial defined?

Author response:
A3: The polynomial degree refers to the highest degree among monomial terms in the polynomial. In this work, the degree of a monomial term represents the number of node features for the Hadamard product, as defined in Equation 10 (Appendix A).

Reviewer comment (Theory):
The theoretical expressivity claims in Section 3.1 may be overclaiming capabilities, as proofs make simplifying assumptions about scalar features that differ from real graph data (Section 4). Can this be justified further?

Author response:
We would like to clarify that our proof makes no assumption about node features being scalars. Specifically, given a node feature matrix $X \in R^{n \times d}$, we first define a monomial term representing the Hadamard product of a series of $d$-dimensional node features $X_i \in R^d$, as illustrated in Equation 10 (Appendix A). Subsequently, we prove by induction in Equations 12 and 13 that Polynormer learns a polynomial which is a linear combination of the monomial terms defined in Equation 10. Thus, our proof considers high-dimensional node features.

In our revision, we further highlight the shape of $X$ and $X_i$ in Appendix A (marked in blue) to indicate that our proof is not restricted to scalar node features.

Reviewer comment (Experiments):
While complexity is analyzed, runtime and memory usage are not empirically compared to baselines in Section 4.2 to demonstrate scalability.

Author response:
Thanks for the suggestion. In our revision, we have added the complete profiling results of Polynormer and GT baselines in terms of training time and GPU memory usage on three large graphs in Appendix J.2, which we reference in Section 4.2 (marked in blue). The results showcase that Polynormer consistently ranks the top 3 fastest models with relatively low memory usage. This demonstrates the scalability advantage of Polynormer over prior GT models. Moreover, we also show the profiling results of Polynormer on synthetic graphs with various sizes in Appendix J.1,  which confirms that Polynormer has linear complexity with respect to the number of nodes/edges in a graph.