Original: From Local to Global: Spectral-Inspired Graph Neural Networks
Revised: From Local to Global: Spectral-Inspired Graph Neural Networks

Original: Abstract
Revised: Abstract

Original: Graph Neural Networks (GNNs) are powerful deep learning methods for NonEuclidean data.
Revised: Graph Neural Networks (GNNs) are powerful deep learning methods for NonEuclidean data.

Original: Popular GNNs are message-passing algorithms (MPNNs) that aggregate and combine signals in a local graph neighborhood.
Revised: Popular GNNs are message-passing algorithms (MPNNs) that aggregate and combine signals in a local graph neighborhood.

Original: However, shallow
Revised: However, shallow

Original: MPNNs tend to miss long-range signals and perform poorly on some heterophilous graphs, while deep MPNNs can suffer from issues like over-smoothing or oversquashing.
Revised: MPNNs tend to miss long-range signals and perform poorly on some heterophilous graphs, while deep MPNNs can suffer from issues like over-smoothing or oversquashing.

Original: To mitigate such issues, existing works typically borrow normalization techniques from training neural networks on Euclidean data or modify the graph structures.
Revised: To mitigate such issues, existing works typically borrow normalization techniques from training neural networks on Euclidean data or modify the graph structures.

Original: Yet these approaches are not well-understood theoretically and could increase the overall computational complexity.
Revised: Yet these approaches are not well-understood theoretically and could increase the overall computational complexity.

Original: In this work, we draw inspirations from spectral graph embedding and propose PowerEmbed — a simple layer-wise normalization technique to boost MPNNs.
Revised: In this work, we draw inspirations from spectral graph embedding and propose PowerEmbed — a simple layer-wise normalization technique to boost MPNNs.

Original: We show PowerEmbed can provably express the top- k leading eigenvectors of the graph operator, which prevents oversmoothing and is agnostic to the graph topology; meanwhile, it produces a list of representations ranging from local features to global signals, which avoids oversquashing.
Revised: We show PowerEmbed can provably express the top- k leading eigenvectors of the graph operator, which prevents oversmoothing and is agnostic to the graph topology; meanwhile, it produces a list of representations ranging from local features to global signals, which avoids oversquashing.

Original: We apply PowerEmbed in a wide range of simulated and real graphs and demonstrate its competitive performance, particularly for heterophilous graphs.
Revised: We apply PowerEmbed in a wide range of simulated and real graphs and demonstrate its competitive performance, particularly for heterophilous graphs.

Original:  Introduction
Revised:  Introduction

Original: We consider the graph representation learning problem.
Revised: We consider the graph representation learning problem.

Original: Given the graph G of n nodes with adjacency matrix A ∈ R n × n and node features X ∈ R n × p , a function f embeds the graph G by mapping ( A,X ) to h ∈ R n × k , where each node is associated with an embedding vector h i ∈
Revised: Given the graph G of n nodes with adjacency matrix A ∈ R n × n and node features X ∈ R n × p , a function f embeds the graph G by mapping ( A,X ) to h ∈ R n × k , where each node is associated with an embedding vector h i ∈

Original: R k .
Revised: R k .

Original: The task is to ﬁnd a permutation equivariant function f
Revised: The task is to ﬁnd a permutation equivariant function f

Original: (i e. f (Π A Π (cid:62) , Π X ) =
Revised: (i e. f (Π A Π (cid:62) , Π X ) =

Original: Π f
Revised: Π f

Original: ( A, X ) for all permutation matrices Π ), while preserving the graph structure: similar nodes in G should be close to each other in the embedding space R k .
Revised: ( A, X ) for all permutation matrices Π ), while preserving the graph structure: similar nodes in G should be close to each other in the embedding space R k .

Original: The quality of the embedding h is typically evaluated in the subsequent statistical inference task, such as node classiﬁcation or link prediction.
Revised: The quality of the embedding h is typically evaluated in the subsequent statistical inference task, such as node classiﬁcation or link prediction.

Original: Spectral embeddings choose f based on matrix factorization.
Revised: Spectral embeddings choose f based on matrix factorization.

Original: For example, f could output the top- k eigenvectors of A .
Revised: For example, f could output the top- k eigenvectors of A .

Original: Spectral embeddings are well-established statistical methods that enjoy nice theoretical properties on random graphs, e.g. stochastic block models [1], latent position random graphs [2].
Revised: Spectral embeddings are well-established statistical methods that enjoy nice theoretical properties on random graphs, e.g. stochastic block models [1], latent position random graphs [2].

Original: Nonetheless, spectral methods typically employ a global spectral decomposition, which can be computationally prohibitive on large-scale graphs; they are also brittle on sparse graphs due to the bias in graph spectra estimation [3],
Revised: Nonetheless, spectral methods typically employ a global spectral decomposition, which can be computationally prohibitive on large-scale graphs; they are also brittle on sparse graphs due to the bias in graph spectra estimation [3],

Original: Graph neural networks (GNNs) parameterize f as a neural network, and typically optimize f with label supervision from the subsequent inference task.
Revised: Graph neural networks (GNNs) parameterize f as a neural network, and typically optimize f with label supervision from the subsequent inference task.

Original: Popular GNNs are message-passing algorithms (MPNNs) that aggregate and combine signals in a local graph neighborhood.
Revised: Popular GNNs are message-passing algorithms (MPNNs) that aggregate and combine signals in a local graph neighborhood.

Original: Despite their computational efﬁciency and scalability, recent works have shown that shallow MPNNs tend to miss long-range signals or global information [4], while deep MPNNs can suffer from issues like over-smoothing (i.e., losing information for the inference task)
Revised: Despite their computational efﬁciency and scalability, recent works have shown that shallow MPNNs tend to miss long-range signals or global information [4], while deep MPNNs can suffer from issues like over-smoothing (i.e., losing information for the inference task)

Original: [5, 6] or over-squashing (i.e. distortion of information from distant nodes) [4, 7].
Revised: [5, 6] or over-squashing (i.e. distortion of information from distant nodes) [4, 7].

Original: Moreover, MPNNs seem to perform poorly on several heterophilous graphs (i.e., nodes from different classes are more likely to connect, as opposed homophilous graphs) [8–10].
Revised: Moreover, MPNNs seem to perform poorly on several heterophilous graphs (i.e., nodes from different classes are more likely to connect, as opposed homophilous graphs) [8–10].

Original: To mitigate such issues, existing MPNNs typically borrow optimization techniques from training standard neural networks [11, 12] or modify the graph structures [13, 14] with increasingly complicated architectures [15, 16].
Revised: To mitigate such issues, existing MPNNs typically borrow optimization techniques from training standard neural networks [11, 12] or modify the graph structures [13, 14] with increasingly complicated architectures [15, 16].

Original: Nevertheless, these approaches do not fully exploit the nature of the graph learning problem.
Revised: Nevertheless, these approaches do not fully exploit the nature of the graph learning problem.

Original: To encode the right kind of inductive bias, there are two major classes of theoretically-driven GNNs: (1) higher-order GNNs that are based on the
Revised: To encode the right kind of inductive bias, there are two major classes of theoretically-driven GNNs: (1) higher-order GNNs that are based on the

Original: Weisfeiler-Lehman hierarchy [17–26]; (2) spectral GNNs that are motivated from graph Fourier transform [27–31].
Revised: Weisfeiler-Lehman hierarchy [17–26]; (2) spectral GNNs that are motivated from graph Fourier transform [27–31].

Original: Despite their nice theoretical properties, they typically incur higher computational costs (e.g., due to operations on higher-order tensors in higher-order GNNs, or additional computations to parameterize the spectral ﬁlter functions in spectral GNNs).
Revised: Despite their nice theoretical properties, they typically incur higher computational costs (e.g., due to operations on higher-order tensors in higher-order GNNs, or additional computations to parameterize the spectral ﬁlter functions in spectral GNNs).

Original: 1.1 Our Contributions
Revised: 1.1 Our Contributions

Original: Inspired by spectral embeddings, we propose a simple normalization technique in MPNNs to encode global spectra information.
Revised: Inspired by spectral embeddings, we propose a simple normalization technique in MPNNs to encode global spectra information.

Original: Speciﬁcally:
Revised: Speciﬁcally:

Original: We propose an unsupervised representation learning method PowerEmbed (Algorithm 1) by augmenting the message-passing layer with a simple normalization step; PowerEmbed can express the top- k eigenvectors of the graph operator, which is agnostic to the graph topology and works well for graphs with homophily or heterophily.
Revised: We propose an unsupervised representation learning method PowerEmbed (Algorithm 1) by augmenting the message-passing layer with a simple normalization step; PowerEmbed can express the top- k eigenvectors of the graph operator, which is agnostic to the graph topology and works well for graphs with homophily or heterophily.

Original: We couple PowerEmbed with an inception network (Algorithm 2) to learn the rich representations that interpolate from local message-passing features to global spectral information, which provably avoids over-smoothing and over-squashing.
Revised: We couple PowerEmbed with an inception network (Algorithm 2) to learn the rich representations that interpolate from local message-passing features to global spectral information, which provably avoids over-smoothing and over-squashing.

Original: We demonstrate numerically that our simple techniques achieve competitive performance for node classiﬁcation in a wide range of simulated and real-world graphs.
Revised: We demonstrate numerically that our simple techniques achieve competitive performance for node classiﬁcation in a wide range of simulated and real-world graphs.

Original:  Related Work
Revised:  Related Work

Original: Normalization techniques in MPNNs.
Revised: Normalization techniques in MPNNs.

Original: Many normalization techniques have been proposed to design deeper MPNNs:
Revised: Many normalization techniques have been proposed to design deeper MPNNs:

Original: [32, 33] proposed a layer-wise mean subtraction and rescaling to maintain pairwise node embedding distances; [34] proposed a layer-wise node normalization depending on their (predicted) clusters; [12, 35] adapted normalization techniques from deep learning and propose node-wise, batch-wise, and graph-wise normalization methods.
Revised: [32, 33] proposed a layer-wise mean subtraction and rescaling to maintain pairwise node embedding distances; [34] proposed a layer-wise node normalization depending on their (predicted) clusters; [12, 35] adapted normalization techniques from deep learning and propose node-wise, batch-wise, and graph-wise normalization methods.

Original: Injecting global information in MPNNs.
Revised: Injecting global information in MPNNs.

Original: Global properties of the graph can be encoded as inputs to MPNNs, such as using spectral embeddings as node features [16, 36], sampling anchor nodes [37], or using other low-pass geometric features [38].
Revised: Global properties of the graph can be encoded as inputs to MPNNs, such as using spectral embeddings as node features [16, 36], sampling anchor nodes [37], or using other low-pass geometric features [38].

Original: On the other hand, the global information may be learned by using speciﬁc architectural choices, such as residual connections [11, 39], attention mechanisms [14, 40–42], or transformers [15, 16, 43].
Revised: On the other hand, the global information may be learned by using speciﬁc architectural choices, such as residual connections [11, 39], attention mechanisms [14, 40–42], or transformers [15, 16, 43].

Original: Furthermore, to speed up the long-range information ﬂow, the original graph can be modiﬁed, such as graph sparsiﬁcation [44–46], graph sampling [47, 48], or localized subgraph extraction [49].
Revised: Furthermore, to speed up the long-range information ﬂow, the original graph can be modiﬁed, such as graph sparsiﬁcation [44–46], graph sampling [47, 48], or localized subgraph extraction [49].

Original: Finally, multiscale graph representation learning can be used, such as scattering transform [50, 29], or hierarchical approaches [51, 52].
Revised: Finally, multiscale graph representation learning can be used, such as scattering transform [50, 29], or hierarchical approaches [51, 52].

Original: GNNs for community detection.
Revised: GNNs for community detection.

Original: Several GNNs have been proposed to tackle community detection in stochastic block models.
Revised: Several GNNs have been proposed to tackle community detection in stochastic block models.

Original: These include semi-supervised approaches such as line-GNNs [53], diff-pooling [52], and GNNs combined with Markov random ﬁeld models [54].
Revised: These include semi-supervised approaches such as line-GNNs [53], diff-pooling [52], and GNNs combined with Markov random ﬁeld models [54].

Original: There are also unsupervised approaches such as graph-pooling [55].
Revised: There are also unsupervised approaches such as graph-pooling [55].

Original: To the best of our knowledge, the statistical properties of the aforementioned MPNNs are not wellunderstood.
Revised: To the best of our knowledge, the statistical properties of the aforementioned MPNNs are not wellunderstood.

Original: Moreover, many of them incur high computational costs.
Revised: Moreover, many of them incur high computational costs.

Original: This motivates our work to propose simple techniques that enjoy provable statistical guarantees and computational efﬁciency.
Revised: This motivates our work to propose simple techniques that enjoy provable statistical guarantees and computational efﬁciency.

Original:  Preliminaries
Revised:  Preliminaries

Original: Notations.
Revised: Notations.

Original: Let the graph G =
Revised: Let the graph G =

Original: ( V, E, X, Y ) with node set V , edge set E , node input features
Revised: ( V, E, X, Y ) with node set V , edge set E , node input features

Original: X ∈ R n × p and node labels Y ∈ R n , where | V |
Revised: X ∈ R n × p and node labels Y ∈ R n , where | V |

Original: = n .
Revised: = n .

Original: Let A, D be the adjacency matrix and the degree matrix of G .
Revised: Let A, D be the adjacency matrix and the degree matrix of G .

Original: Let ¯ A
Revised: Let ¯ A

Original: =
Revised: =

Original: ˜
Revised: ˜

Original: D − 0 .
Revised: D − 0 .

Original: 5 ˜
Revised: 5 ˜

Original: A ˜ D − 0 .
Revised: A ˜ D − 0 .

Original: 5 be the symmetric graph Laplacian,
Revised: 5 be the symmetric graph Laplacian,

Original: where ˜ A =
Revised: where ˜ A =

Original: A + I, ˜ D
Revised: A + I, ˜ D

Original: = D
Revised: = D

Original: + I .
Revised: + I .

Original: Let A rw =
Revised: Let A rw =

Original: ˜ D − 1 ˜
Revised: ˜ D − 1 ˜

Original: A be the random walk graph Laplacian.
Revised: A be the random walk graph Laplacian.

Original: Let
Revised: Let

Original: U (cid:107) U
Revised: U (cid:107) U

Original: [: ,k ] (cid:107) be the column normalized form of matrix U .
Revised: [: ,k ] (cid:107) be the column normalized form of matrix U .

Original: Inference problem setup.
Revised: Inference problem setup.

Original: To evaluate the graph embedding h
Revised: To evaluate the graph embedding h

Original: = f
Revised: = f

Original: ( A, X ) , we consider the node classiﬁcation task:
Revised: ( A, X ) , we consider the node classiﬁcation task:

Original: Given a graph G of n nodes with n t < n training labels, the goal is to predict the remaining n − n t test labels.
Revised: Given a graph G of n nodes with n t < n training labels, the goal is to predict the remaining n − n t test labels.

Original: Let Ω and Ω ⊥ the indices corresponding to training nodes and test nodes respectively ( V = V Ω ∪ V Ω ⊥ ).
Revised: Let Ω and Ω ⊥ the indices corresponding to training nodes and test nodes respectively ( V = V Ω ∪ V Ω ⊥ ).

Original: Then the classiﬁer is trained on the training set ( h Ω , Y Ω ) and evaluated on the test set ( h Ω ⊥ , Y Ω ⊥ ) .
Revised: Then the classiﬁer is trained on the training set ( h Ω , Y Ω ) and evaluated on the test set ( h Ω ⊥ , Y Ω ⊥ ) .

Original: If the graph embedding is a list of features P =
Revised: If the graph embedding is a list of features P =

Original: [ X, h (1) . . .
Revised: [ X, h (1) . . .

Original: , h ( L ) ] , then we let P Ω , P Ω ⊥ denote the training and test set features, where
Revised: , h ( L ) ] , then we let P Ω , P Ω ⊥ denote the training and test set features, where

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition 3.1 (Stochastic Block Model (SBM)) .
Revised: Deﬁnition 3.1 (Stochastic Block Model (SBM)) .

Original: A graph A with n nodes is a random SBM graph if it is sampled as
Revised: A graph A with n nodes is a random SBM graph if it is sampled as

Original: [Equation]
Revised: [Equation]

Original: where Z ∈ R n × K is a membership matrix such that Z i,k is 1 if the i -th node belongs to the k -th class, (cid:107)
Revised: where Z ∈ R n × K is a membership matrix such that Z i,k is 1 if the i -th node belongs to the k -th class, (cid:107)

Original: Z i, · (cid:107) 

Revised: Z i, · (cid:107)

Original: = (cid:80)
Revised: = (cid:80)

Original: Kk =1 | Z i,k | = 1 , and B ∈ [0 , 1] K × K is a full-rank matrix representing the block connection probability.
Revised: Kk =1 | Z i,k | = 1 , and B ∈ [0 , 1] K × K is a full-rank matrix representing the block connection probability.

Original: Deﬁnition 3.2 (2B-SBM with Gaussian node features) .
Revised: Deﬁnition 3.2 (2B-SBM with Gaussian node features) .

Original: A two-block symmetric SBM (2B-SBM) is given by:
Revised: A two-block symmetric SBM (2B-SBM) is given by:

Original: Z i, · = (cid:26) [1 , 0] if i ∈ [ n/ 2] [0 , 1] otherwise.
Revised: Z i, · = (cid:26) [1 , 0] if i ∈ [ n/ 2] [0 , 1] otherwise.

Original: , B = (cid:20) p q q p
Revised: , B = (cid:20) p q q p

Original: [Equation]
Revised: [Equation]

Original: where p, q ∈ (0 , 1) , p (cid:54) = q .
Revised: where p, q ∈ (0 , 1) , p (cid:54) = q .

Original: The node features in block k ∈ { 0 , 1 } are sampled from a m -dimensional multivariate Gaussian N ( µ k , Σ k ) and stored in a node feature matrix X ∈ R n × m .
Revised: The node features in block k ∈ { 0 , 1 } are sampled from a m -dimensional multivariate Gaussian N ( µ k , Σ k ) and stored in a node feature matrix X ∈ R n × m .

Original: The SBM model is a canonical random graph model with planted clusters, which has been widely studied in the context of community detection [3, 2, 1, 56].
Revised: The SBM model is a canonical random graph model with planted clusters, which has been widely studied in the context of community detection [3, 2, 1, 56].

Original: In particular, the leading eigenvectors of the graph adjacency A or the graph Laplacians A rw , ¯ A provably encode community membership, provided that the graphs are sufﬁciently dense [57, 1].
Revised: In particular, the leading eigenvectors of the graph adjacency A or the graph Laplacians A rw , ¯ A provably encode community membership, provided that the graphs are sufﬁciently dense [57, 1].

Original: To see this, consider the 2B-SBM model in (3) without node features,
Revised: To see this, consider the 2B-SBM model in (3) without node features,

Original: then the matrix P =
Revised: then the matrix P =

Original: E ( A ) has three eigenvalues, ranked by magnitude as ( p + q ) / 2 , ( p − q ) / 2 , 0 , where 0 has multiplicity n − 2 .
Revised: E ( A ) has three eigenvalues, ranked by magnitude as ( p + q ) / 2 , ( p − q ) / 2 , 0 , where 0 has multiplicity n − 2 .

Original: The leading eigenvector is a constant vector, whereas the second eigenvector u 2 ( P ) =
Revised: The leading eigenvector is a constant vector, whereas the second eigenvector u 2 ( P ) =

Original: [ 1 n/ 2 ; − 1 n/ 2 ] reveals the community structure.
Revised: [ 1 n/ 2 ; − 1 n/ 2 ] reveals the community structure.

Original: Now, the random graph A can be viewed as a perturbation of its expectation P .
Revised: Now, the random graph A can be viewed as a perturbation of its expectation P .

Original: Standard results in concentration of measure show that the eigenvalues and eigenvectors of A are close to those of P for sufﬁciently dense graphs (the average node degree needs to be of the order Ω( n log n ) ).
Revised: Standard results in concentration of measure show that the eigenvalues and eigenvectors of A are close to those of P for sufﬁciently dense graphs (the average node degree needs to be of the order Ω( n log n ) ).

Original: However, spectral methods fail to detect communities when the graphs are very sparse ( p = q
Revised: However, spectral methods fail to detect communities when the graphs are very sparse ( p = q

Original: = O (1 /n ) ), and there are known information-theoretical thresholds for certain stochastic block models (see [3] for a recent survey).
Revised: = O (1 /n ) ), and there are known information-theoretical thresholds for certain stochastic block models (see [3] for a recent survey).

Original: In this work, we consider classical spectral embedding methods deﬁned as follows:
Revised: In this work, we consider classical spectral embedding methods deﬁned as follows:

Original: Let A =
Revised: Let A =

Original: USU (cid:62) , XX (cid:62) =
Revised: USU (cid:62) , XX (cid:62) =

Original: ˜ U Σ ˜ U (cid:62) be the spectral decomposition of the graph A and the covariance matrix
Revised: ˜ U Σ ˜ U (cid:62) be the spectral decomposition of the graph A and the covariance matrix

Original: XX (cid:62) .
Revised: XX (cid:62) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Crucially, spectral methods that are based on the top- k eigenvectors ranked by absolute magnitude have no assumption on the graph being homophilous (e.g., p > q ) or heterophilous (e.g., p < q );
Revised: Crucially, spectral methods that are based on the top- k eigenvectors ranked by absolute magnitude have no assumption on the graph being homophilous (e.g., p > q ) or heterophilous (e.g., p < q );

Original: They perform consistently well on both cases by “looking globally" at the graph structure.
Revised: They perform consistently well on both cases by “looking globally” at the graph structure.

Original: In contrast, message-passing methods are local algorithms that work well in homophilous graphs, but they tend to overlook global signals and struggle to learn in heterophilous graphs.
Revised: In contrast, message-passing methods are local algorithms that work well in homophilous graphs, but they tend to overlook global signals and struggle to learn in heterophilous graphs.

Original: Deﬁnition 3.3 (Message-Passing Neural Network (MPNN)) .
Revised: Deﬁnition 3.3 (Message-Passing Neural Network (MPNN)) .

Original: A L -layer MPNN initializes the embedding h (0) =
Revised: A L -layer MPNN initializes the embedding h (0) =

Original: X .
Revised: X .

Original: At each iteration l , the embedding of node i is updated as
Revised: At each iteration l , the embedding of node i is updated as

Original: [Equation]
Revised: [Equation]

Original: where φ, ψ are the update and message functions, and N ( i ) denotes the neighbors of node i .
Revised: where φ, ψ are the update and message functions, and N ( i ) denotes the neighbors of node i .

Original: For example, graph convolutional networks (GCN [58]) choose φ as a nonlinear activation function σ and ψ ( i,j ) =
Revised: For example, graph convolutional networks (GCN [58]) choose φ as a nonlinear activation function σ and ψ ( i,j ) =

Original: [Equation]
Revised: [Equation]

Original: , where W ( l − 1) is a learnable weight matrix.
Revised: , where W ( l − 1) is a learnable weight matrix.

Original: In other words:
Revised: In other words:

Original: [Equation]
Revised: [Equation]

Original: GCN can be further simpliﬁed by removing the nonlinearity σ , which is known as simple graph convolution (SGC [59]):
Revised: GCN can be further simpliﬁed by removing the nonlinearity σ , which is known as simple graph convolution (SGC [59]):

Original: [Equation]
Revised: [Equation]

Original: To train a classiﬁer on top of the SGC embedding in (7), we can absorb the weight matrices W ( l − 1) to the downstream classiﬁer.
Revised: To train a classiﬁer on top of the SGC embedding in (7), we can absorb the weight matrices W ( l − 1) to the downstream classiﬁer.

Original: In other words, SGC can be viewed as producing an unsupervised representation ¯
Revised: In other words, SGC can be viewed as producing an unsupervised representation ¯

Original: A L X followed by learning a classiﬁer.
Revised: A L X followed by learning a classiﬁer.

Original: However, such representation fails to distinguish nodes from different classes when L is sufﬁciently large, known as the over-smoothing issue of deep
Revised: However, such representation fails to distinguish nodes from different classes when L is sufﬁciently large, known as the over-smoothing issue of deep

Original: GNNs.
Revised: GNNs.

Original: Deﬁnition 3.4 (Over-smoothing [5]) .
Revised: Deﬁnition 3.4 (Over-smoothing [5]) .

Original: Assume the graph has one connected component.
Revised: Assume the graph has one connected component.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Over-smoothing describes that for a sufﬁciently deep SGC model, its node embedding converges to the largest eigenvector which only encodes connected component and degree information, but tends to miss the community structure present in subsequent eigenvectors.
Revised: Over-smoothing describes that for a sufﬁciently deep SGC model, its node embedding converges to the largest eigenvector which only encodes connected component and degree information, but tends to miss the community structure present in subsequent eigenvectors.

Original: It has also been extended and deﬁned for nonlinear GCNs [6, 60] and empirically observed in deep MPNNs [32, 44, 39].
Revised: It has also been extended and deﬁned for nonlinear GCNs [6, 60] and empirically observed in deep MPNNs [32, 44, 39].

Original: On the other hand, over-squashing refers to the distortion of node features in the learned graph representation in a deep GNN model [4, 7], which can also hurt subsequent inference when the node features carry useful signals.
Revised: On the other hand, over-squashing refers to the distortion of node features in the learned graph representation in a deep GNN model [4, 7], which can also hurt subsequent inference when the node features carry useful signals.

Original: Deﬁnition 3.5 (Over-squashing [7]) .
Revised: Deﬁnition 3.5 (Over-squashing [7]) .

Original: Let h ( L ) i
Revised: Let h ( L ) i

Original: = h ( L ) i ( x 1 , . . .
Revised: = h ( L ) i ( x 1 , . . .

Original: , x n ) be the output for node i of a
Revised: , x n ) be the output for node i of a

Original: L -layer MPNN with input features { x i } ni =1 .
Revised: L -layer MPNN with input features { x i } ni =1 .

Original: Then the over-squashing effect (for node i with respect to node s ) is measured by the Jacobian ∂h ( L ) i /∂x s .
Revised: Then the over-squashing effect (for node i with respect to node s ) is measured by the Jacobian ∂h ( L ) i /∂x s .

Original: Thus, the smaller the Jacobian value, the more node feature information is “squashed" out in the embedding.
Revised: Thus, the smaller the Jacobian value, the more node feature information is “squashed” out in the embedding.

Original:  PowerEmbed
Revised:  PowerEmbed

Original: In this section, we ﬁrst show how to express the top- k eigenvectors of a graph operator using message-passing power iteration ( PowerEmbed in Algorithm 1).
Revised: In this section, we ﬁrst show how to express the top- k eigenvectors of a graph operator using message-passing power iteration ( PowerEmbed in Algorithm 1).

Original: Next, we describe how to exploit the rich representations produced by PowerEmbed by using the inception neural network architecture (Algorithm 2).
Revised: Next, we describe how to exploit the rich representations produced by PowerEmbed by using the inception neural network architecture (Algorithm 2).

Original: Our framework is summarized in Figure 1.
Revised: Our framework is summarized in Figure 1.

Original: Algorithm 1 PowerEmbed
Revised: Algorithm 1 PowerEmbed

Original: Require: a graph operator S ∈ R n × n , node features X ∈ R n × k , a list P =
Revised: Require: a graph operator S ∈ R n × n , node features X ∈ R n × k , a list P =

Original: [ X ] .
Revised: [ X ] .

Original: Initialize:
Revised: Initialize:

Original: U ( t ) =
Revised: U ( t ) =

Original: X for t = 0 to L-1 do ˜ U
Revised: X for t = 0 to L-1 do ˜ U

Original: ( t + 1) =
Revised: ( t + 1) =

Original: S U ( t ) [message-passing]
Revised: S U ( t ) [message-passing]

Original: U ( t + 1) =
Revised: U ( t + 1) =

Original: ˜
Revised: ˜

Original: U ( t + 1)[ ˜ U ( t + 1) (cid:62) ˜
Revised: U ( t + 1)[ ˜ U ( t + 1) (cid:62) ˜

Original: U ( t + 1)] − 1 [Normalization]
Revised: U ( t + 1)] − 1 [Normalization]

Original: Append
Revised: Append

Original: U ( t +1) (cid:107) U
Revised: U ( t +1) (cid:107) U

Original: ( t +1)
Revised: ( t +1)

Original: [: ,k ] (cid:107) to P end for return P
Revised: [: ,k ] (cid:107) to P end for return P

Original: Algorithm 1 is motivated from the natural power iteration [61], which provably returns the top- k eigenvectors of a symmetric matrix S (up to an orthogonal transformation) upon convergence, with mild assumptions that the input features X has full column rank and the S has an eigen-gap, i.e., λ k (cid:54) =
Revised: Algorithm 1 is motivated from the natural power iteration [61], which provably returns the top- k eigenvectors of a symmetric matrix S (up to an orthogonal transformation) upon convergence, with mild assumptions that the input features X has full column rank and the S has an eigen-gap, i.e., λ k (cid:54) =

Original: λ k
Revised: λ k

Original: +1 where λ k denotes the k -th eigenvalue of S (ranked by magnitude).
Revised: +1 where λ k denotes the k -th eigenvalue of S (ranked by magnitude).

Original: Crucially, the
Revised: Crucially, the

Original: In practice, if the original node features live in high-dimensional space (e.g., more features than the number of nodes), we ﬁrst perform a dimensionality reduction to obtain the best rank- k approximation of X before running Algorithm 1 to speed up convergence and reduce memory complexity.
Revised: In practice, if the original node features live in high-dimensional space (e.g., more features than the number of nodes), we ﬁrst perform a dimensionality reduction to obtain the best rank- k approximation of X before running Algorithm 1 to speed up convergence and reduce memory complexity.

Original: Unlike existing work that requires pre-computed eigenvectors as input [16, 36, 62], our approach implicitly computes eigenvectors using power iteration to improve efﬁciency while maintaining expressivity.
Revised: Unlike existing work that requires pre-computed eigenvectors as input [16, 36, 62], our approach implicitly computes eigenvectors using power iteration to improve efﬁciency while maintaining expressivity.

Original: By embedding the original graph to a list of Euclidean features, PowerEmbed allows fast training and inference with computational complexity independent of the graph topology.
Revised: By embedding the original graph to a list of Euclidean features, PowerEmbed allows fast training and inference with computational complexity independent of the graph topology.

Original: PowerEmbed extracts both local features from the ﬁrst few iterations and global information from the last few iterations (i.e., the top- k eigenvectors).
Revised: PowerEmbed extracts both local features from the ﬁrst few iterations and global information from the last few iterations (i.e., the top- k eigenvectors).

Original: These rich features are jointly learned using the inception network (Algorithm 2).
Revised: These rich features are jointly learned using the inception network (Algorithm 2).

Original: Therefore, the presence of local features (including the input features) can avoid over-squashing in Deﬁnition 3.5; the ability to express top- k eigenvectors of a graph operator alleviates over-smoothing in Deﬁnition 3.4 and makes it agnostic to the graph topology.
Revised: Therefore, the presence of local features (including the input features) can avoid over-squashing in Deﬁnition 3.5; the ability to express top- k eigenvectors of a graph operator alleviates over-smoothing in Deﬁnition 3.4 and makes it agnostic to the graph topology.

Original: Such expressivity is necessary to avoid over-smoothing/over-squashing considered in this work, but may not be sufﬁcient depending on the data and the optimization procedure.
Revised: Such expressivity is necessary to avoid over-smoothing/over-squashing considered in this work, but may not be sufﬁcient depending on the data and the optimization procedure.

Original: Algorithm
Revised: Algorithm

Original: Require:
Revised: Require:

Original: Training features P Ω (1), training labels Y Ω , loss function (cid:96) , step size η , hidden dimension k (cid:48) , number of classes K , epochs T .
Revised: Training features P Ω (1), training labels Y Ω , loss function (cid:96) , step size η , hidden dimension k (cid:48) , number of classes K , epochs T .

Original: Initialize:
Revised: Initialize:

Original: A list of Multilayer Perceptions MLPs =
Revised: A list of Multilayer Perceptions MLPs =

Original: [ f θ 0 , . .
Revised: [ f θ 0 , . .

Original: , f θ L ]
Revised: , f θ L ]

Original: where f θ i : R k
Revised: where f θ i : R k

Original: → R k (cid:48) , a classiﬁer g θ (cid:48) : R ( L +1) k (cid:48)
Revised: → R k (cid:48) , a classiﬁer g θ (cid:48) : R ( L +1) k (cid:48)

Original: → R K .
Revised: → R K .

Original: for t = 0 to T do
Revised: for t = 0 to T do

Original: H =
Revised: H =

Original: [Equation]
Revised: [Equation]

Original: H
Revised: H

Original: [Equation]
Revised: [Equation]

Original: = concat ( H
Revised: = concat ( H

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ˆ Y =
Revised: ˆ Y =

Original: g θ (cid:48)
Revised: g θ (cid:48)

Original: ( H (cid:48) )
Revised: ( H (cid:48) )

Original: L = (cid:96) ( ˆ Y , Y Ω ) θ = θ − η ∇ θ L for all θ ∈ { θ 0 , . . .
Revised: L = (cid:96) ( ˆ Y , Y Ω ) θ = θ − η ∇ θ L for all θ ∈ { θ 0 , . . .

Original: , θ L , θ (cid:48) } end for return g θ (cid:48) ◦ MLPs
Revised: , θ L , θ (cid:48) } end for return g θ (cid:48) ◦ MLPs

Original: The following propositions, proven in Appendix A are basic facts of graph theory.
Revised: The following propositions, proven in Appendix A are basic facts of graph theory.

Original: Proposition 4. applies the convergence results from [61] to the graph operators used in Algorithm 1, and it shows that PowerEmbed can express the top- k eigenvectors of the common graph operators.
Revised: Proposition 4. applies the convergence results from [61] to the graph operators used in Algorithm 1, and it shows that PowerEmbed can express the top- k eigenvectors of the common graph operators.

Original: Proposition 4.
Revised: Proposition 4.

Original: shows that our graph representation learning approach is permutation equivariant [63].
Revised: shows that our graph representation learning approach is permutation equivariant [63].

Original: Proposition 4.1.
Revised: Proposition 4.1.

Original: Consider a graph G with graph operator S ∈
Revised: Consider a graph G with graph operator S ∈

Original: { A, ¯ A, A rw } and feature matrix
Revised: { A, ¯ A, A rw } and feature matrix

Original: X ∈ R n × k .
Revised: X ∈ R n × k .

Original: If X is full (column) rank and the k -th and ( k + 1) -th eigenvalues of S are distinct, then the last iterate U ( t + 1) from PowerEmbed converges to the top k eigenvectors of S when t → ∞ (up to an orthogonal transformation in O ( k ) ).
Revised: If X is full (column) rank and the k -th and ( k + 1) -th eigenvalues of S are distinct, then the last iterate U ( t + 1) from PowerEmbed converges to the top k eigenvectors of S when t → ∞ (up to an orthogonal transformation in O ( k ) ).

Original: Proposition 4.2.
Revised: Proposition 4.2.

Original: The method PowerEmbed (Algorithm 1) followed by the inception neural network g θ (cid:48) ◦ MLPs (from Algorithm 2) is permutation equivariant.
Revised: The method PowerEmbed (Algorithm 1) followed by the inception neural network g θ (cid:48) ◦ MLPs (from Algorithm 2) is permutation equivariant.

Original:  Numerical Experiments 5.1 Baselines
Revised:  Numerical Experiments 5.1 Baselines

Original: To evaluate the beneﬁts of injecting global spectra information into MPNNs, We compare
Revised: To evaluate the beneﬁts of injecting global spectra information into MPNNs, We compare

Original: PowerEmbed with three classes of baselines.
Revised: PowerEmbed with three classes of baselines.

Original: Unnormalized counterparts.
Revised: Unnormalized counterparts.

Original: Unsupervised MPNNs without layer-wise normalization using the same inception network architecture (Algorithm 2) : SIGN [64] that uses A rw , and a modiﬁed version of
Revised: Unsupervised MPNNs without layer-wise normalization using the same inception network architecture (Algorithm 2) : SIGN [64] that uses A rw , and a modiﬁed version of

Original: SGC [59] that uses ¯ A which we call “SGC(Incep)”.
Revised: SGC [59] that uses ¯ A which we call “SGC(Incep)”.

Original: These baselines illustrate the effects of adding the proposed layer-wise normalization in Algorithm 1.
Revised: These baselines illustrate the effects of adding the proposed layer-wise normalization in Algorithm 1.

Original: Spectral methods.
Revised: Spectral methods.

Original: Standard spectral embedding methods described in (4) that use purely global (spectral) information.
Revised: Standard spectral embedding methods described in (4) that use purely global (spectral) information.

Original: These global baselines measure the effects of adding local information (from neighbor-aggregated features) in PowerEmbed .
Revised: These global baselines measure the effects of adding local information (from neighbor-aggregated features) in PowerEmbed .

Original: Semi-supervised MPNNs.
Revised: Semi-supervised MPNNs.

Original: Popular models including graph convolutional networks (GCN) [58] and graph attention networks (GAT) [65]) for benchmark purpose; MPNNs that encode long-range information including geometric GCN (GEOM-GCN) [8], generalized page-rank GNN (GPR-GNN) [66], GCN via initial residual and identity mapping (GCNII)
Revised: Popular models including graph convolutional networks (GCN) [58] and graph attention networks (GAT) [65]) for benchmark purpose; MPNNs that encode long-range information including geometric GCN (GEOM-GCN) [8], generalized page-rank GNN (GPR-GNN) [66], GCN via initial residual and identity mapping (GCNII)

Original: [67], and Jumping Knowledge Networks with layer concatenation (JK-Concat) [14]; These baselines are chosen to gauge the effects of injecting spectral information as opposed to long-range spatial information.
Revised: [67], and Jumping Knowledge Networks with layer concatenation (JK-Concat) [14]; These baselines are chosen to gauge the effects of injecting spectral information as opposed to long-range spatial information.

Original: We remark that all unsupervised methods (i.e., PowerEmbed , unnormalized counterparts, spectral methods) produce an embedding matrix h ∈ R n × k (or a list of features P ) without label supervision, followed by ﬁtting a neural network classiﬁer:
Revised: We remark that all unsupervised methods (i.e., PowerEmbed , unnormalized counterparts, spectral methods) produce an embedding matrix h ∈ R n × k (or a list of features P ) without label supervision, followed by ﬁtting a neural network classiﬁer:

Original: For a single embedding matrix h , we ﬁt a 2-layer MLP with ReLU activation; for the list P =
Revised: For a single embedding matrix h , we ﬁt a 2-layer MLP with ReLU activation; for the list P =

Original: [ X, h 1 , . . .
Revised: [ X, h 1 , . . .

Original: , h L ] , we ﬁt the inception network (Algorithm 2).
Revised: , h L ] , we ﬁt the inception network (Algorithm 2).

Original: On the other hand, all semi-supervised MPNNs optimize the network end-to-end.
Revised: On the other hand, all semi-supervised MPNNs optimize the network end-to-end.

Original: All the simulations and code are available in this repository 1 .
Revised: All the simulations and code are available in this repository 1 .

Original: 5.2 Synthetic graphs
Revised: 5.2 Synthetic graphs

Original: In this section, we simulate random graphs following the stochastic block models deﬁned in (3), with node features X sampled from a mixture of two Gaussians with mean µ 0 =
Revised: In this section, we simulate random graphs following the stochastic block models deﬁned in (3), with node features X sampled from a mixture of two Gaussians with mean µ 0 =

Original: [1 , 1] , µ 1 = − µ 0 .
Revised: [1 , 1] , µ 1 = − µ 0 .

Original: We consider the (binary) node classiﬁcation setting described in Section 3 with n = 500 nodes and a 10/90 train/test split.
Revised: We consider the (binary) node classiﬁcation setting described in Section 3 with n = 500 nodes and a 10/90 train/test split.

Original: We choose k = 2 (i.e., use X without dimensionality reduction).
Revised: We choose k = 2 (i.e., use X without dimensionality reduction).

Original: Figure 2 illustrates the convergence of the last iterate of PowerEmbed to the top- 2 eigenvectors of a 2B-SBM expected adjacency matrix, with different choices of the parameters p,q that controls the block probability matrix B deﬁned in (3).
Revised: Figure 2 illustrates the convergence of the last iterate of PowerEmbed to the top- 2 eigenvectors of a 2B-SBM expected adjacency matrix, with different choices of the parameters p,q that controls the block probability matrix B deﬁned in (3).

Original: The convergence is fast in the dense case where λ i ( A )
Revised: The convergence is fast in the dense case where λ i ( A )

Original: ≈
Revised: ≈

Original: λ
Revised: λ

Original: i ( P ) , i ∈ { 1 , 2 } and are well-separated from the remaining n − 2 eigenvalues.
Revised: i ( P ) , i ∈ { 1 , 2 } and are well-separated from the remaining n − 2 eigenvalues.

Original: In the sparse case, the eigen-gap λ 2 ( A ) − λ 3 ( A ) is smaller due to higher (random) noise in the sparse random graph A , and thus the convergence of ˆ u 2 is slower.
Revised: In the sparse case, the eigen-gap λ 2 ( A ) − λ 3 ( A ) is smaller due to higher (random) noise in the sparse random graph A , and thus the convergence of ˆ u 2 is slower.

Original: Figure 3 shows that PowerEmbed is able to recover the spectral embeddings in dense graphs (left column), and outperforms spectral embedding in sparse graphs (right column).
Revised: Figure 3 shows that PowerEmbed is able to recover the spectral embeddings in dense graphs (left column), and outperforms spectral embedding in sparse graphs (right column).

Original: This can be explained by PowerEmbed ’s ability to leverage both local node features and global structural information.
Revised: This can be explained by PowerEmbed ’s ability to leverage both local node features and global structural information.

Original: Moreover, PowerEmbed and spectral methods perform consistently well on graphs with homophily (top row) and heterophily (bottom row).
Revised: Moreover, PowerEmbed and spectral methods perform consistently well on graphs with homophily (top row) and heterophily (bottom row).

Original: This is in contrast to standard MPNNs that suffer from oversmoothing (e.g., “GCN-10”) and fail in sparse heterophilous graphs (e.g., “GCN-5” performs much worse in heterophilous graphs in (d) compared to homophilous graph in (b) while “Power(Lap)-10” performs steadily well).
Revised: This is in contrast to standard MPNNs that suffer from oversmoothing (e.g., “GCN-10”) and fail in sparse heterophilous graphs (e.g., “GCN-5” performs much worse in heterophilous graphs in (d) compared to homophilous graph in (b) while “Power(Lap)-10” performs steadily well).

Original: Comprehensive simulation results including other baselines can be found in
Revised: Comprehensive simulation results including other baselines can be found in

Original: Appendix C.1.
Revised: Appendix C.1.

Original: 5.3 Real-world graphs
Revised: 5.3 Real-world graphs

Original: We report node classiﬁcation results of PowerEmbed on 10 graph benchmark datasets, includingheterophily graphs and 5 homophily graphs with varying densities.
Revised: We report node classiﬁcation results of PowerEmbed on 10 graph benchmark datasets, includingheterophily graphs and 5 homophily graphs with varying densities.

Original: We summarize key statistics of each graph in Table 1, Table 2 and describe their details in Appendix B.2.
Revised: We summarize key statistics of each graph in Table 1, Table 2 and describe their details in Appendix B.2.

Original: For most graphs except Computers and Photo, we use the same data splits (48/32/20 for train/validation/test) from [8] released in Pytorch Geometric [68]; for Computers and Photo that were not studied in [8], we use the same data split (60/20/20) as in [69].
Revised: For most graphs except Computers and Photo, we use the same data splits (48/32/20 for train/validation/test) from [8] released in Pytorch Geometric [68]; for Computers and Photo that were not studied in [8], we use the same data split (60/20/20) as in [69].

Original: Note that most of the node features are high-dimensional bag-of-words vectors.
Revised: Note that most of the node features are high-dimensional bag-of-words vectors.

Original: Thus, for unsupervised embedding methods
Revised: Thus, for unsupervised embedding methods

Original: [Table]
Revised: [Table]

Original: and JK-Concat, we use the top- k eigenvector of the feature covariance as the input feature, where k
Revised: and JK-Concat, we use the top- k eigenvector of the feature covariance as the input feature, where k

Original: = 10 for small-size graphs including Wisconsin, Texas, and Cornell, and k = 100 elsewhere; for semi-supervised methods (GCN-II, GPR-GNN), we use the original node features following the original model architectures.
Revised: = 10 for small-size graphs including Wisconsin, Texas, and Cornell, and k = 100 elsewhere; for semi-supervised methods (GCN-II, GPR-GNN), we use the original node features following the original model architectures.

Original: Self-loops are used for all graphs.
Revised: Self-loops are used for all graphs.

Original: We train all models using ADAM optimizer [70] and full-batch gradient descent with the same hyperparameters: 100 epochs, 0.
Revised: We train all models using ADAM optimizer [70] and full-batch gradient descent with the same hyperparameters: 100 epochs, 0.

Original: learning rate, 0.5 dropout rate, and no weight decay.
Revised: learning rate, 0.5 dropout rate, and no weight decay.

Original: Further details can be found in Appendix B.1.
Revised: Further details can be found in Appendix B.1.

Original: Table 1 summarizes the experimental results on heterophilous graph benchmarks: PowerEmbed outperforms other baselines on graphs with heterophily, particularly on dense heterophilous graphs.
Revised: Table 1 summarizes the experimental results on heterophilous graph benchmarks: PowerEmbed outperforms other baselines on graphs with heterophily, particularly on dense heterophilous graphs.

Original: Figure 4 visualize the effect of increasing the number of message-passing layers in Squirrel and
Revised: Figure 4 visualize the effect of increasing the number of message-passing layers in Squirrel and

Original: Chameleon, two dense graphs with heterophily: PowerEmbed exhibits consistent performance in deeper models and avoids over-smoothing; The ability to express global spectra information via simple normalization signiﬁcantly boosts performance over unnormalized MPNN counterparts in heterophilous graphs.
Revised: Chameleon, two dense graphs with heterophily: PowerEmbed exhibits consistent performance in deeper models and avoids over-smoothing; The ability to express global spectra information via simple normalization signiﬁcantly boosts performance over unnormalized MPNN counterparts in heterophilous graphs.

Original: Across all 5 heterophilous graphs, we observe that even simple spectral method
Revised: Across all 5 heterophilous graphs, we observe that even simple spectral method

Original: A_X concatenating leading eigenvectors of graph spectra and node covariance achieves competitive performance, drastically outperforming unnormalized MPNNs and semi-supervised MPNNs.
Revised: A_X concatenating leading eigenvectors of graph spectra and node covariance achieves competitive performance, drastically outperforming unnormalized MPNNs and semi-supervised MPNNs.

Original: This highlights the utility of learning graph representation from ﬁrst principle.
Revised: This highlights the utility of learning graph representation from ﬁrst principle.

Original: Further ablation studies are shown in Appendix C.2.
Revised: Further ablation studies are shown in Appendix C.2.

Original: Table 2 summarizes the results on homophilous graph benchmarks: PowerEmbed achieves competitive performance as other MPNN baselines.
Revised: Table 2 summarizes the results on homophilous graph benchmarks: PowerEmbed achieves competitive performance as other MPNN baselines.

Original: As discussed in Section 3, MPNNs tend to work well in graphs with homophily due to the inductive bias of aggregating local neighbors’ information, while spectral methods could perform sub-optimally, especially in sparse graphs.
Revised: As discussed in Section 3, MPNNs tend to work well in graphs with homophily due to the inductive bias of aggregating local neighbors’ information, while spectral methods could perform sub-optimally, especially in sparse graphs.

Original: Together with Table 1,
Revised: Together with Table 1,

Original: PowerEmbed is shown to be agnostic to the graph topology and can ﬂexibly combine local and global information.
Revised: PowerEmbed is shown to be agnostic to the graph topology and can ﬂexibly combine local and global information.

Original:  Discussion
Revised:  Discussion

Original: In this work, we propose a simple technique PowerEmbed that exploits the advantages of global spectral methods and local message-passing algorithms for graph representation learning.
Revised: In this work, we propose a simple technique PowerEmbed that exploits the advantages of global spectral methods and local message-passing algorithms for graph representation learning.

Original: Specifically, PowerEmbed performs a layer-wise normalization that allows MPNNs to express the top- k
Revised: Specifically, PowerEmbed performs a layer-wise normalization that allows MPNNs to express the top- k

Original: [Table]
Revised: [Table]

Original: eigenvectors of a graph and capture global spectral information.
Revised: eigenvectors of a graph and capture global spectral information.

Original: We demonstrate the advantages of our techniques theoretically and empirically on both synthetic and real-world graphs.
Revised: We demonstrate the advantages of our techniques theoretically and empirically on both synthetic and real-world graphs.

Original: Our spectral-inspired GNN opens the door to many interesting future directions.
Revised: Our spectral-inspired GNN opens the door to many interesting future directions.

Original: Firstly, we can extend
Revised: Firstly, we can extend

Original: PowerEmbed to semi-supervised graph learning tasks allowing learnable weights in the messagepassing layers: While PowerEmbed can still express the top- k eigenvectors of the graph operators, it may converge to a function of eigenvectors driven by label supervision.
Revised: PowerEmbed to semi-supervised graph learning tasks allowing learnable weights in the messagepassing layers: While PowerEmbed can still express the top- k eigenvectors of the graph operators, it may converge to a function of eigenvectors driven by label supervision.

Original: We conjecture that the semi-supervised versions may be helpful in certain sparse graphs, where the graph eigenvectors are suboptimal in estimating community structure and the label signals can improve the inference performance (e.g., partially-labeled SBM [72, 73]).
Revised: We conjecture that the semi-supervised versions may be helpful in certain sparse graphs, where the graph eigenvectors are suboptimal in estimating community structure and the label signals can improve the inference performance (e.g., partially-labeled SBM [72, 73]).

Original: Furthermore, since PowerEmbed combines global spectral information with local MPNNs, it is natural to consider more powerful versions of
Revised: Furthermore, since PowerEmbed combines global spectral information with local MPNNs, it is natural to consider more powerful versions of

Original: PowerEmbed that are based on higher-order MPNNs.
Revised: PowerEmbed that are based on higher-order MPNNs.

Original: Yet it remains open to fully understand the relations between graph spatial information (i.e., symmetries) and graph spectral information (e.g., eigenvalues and eigenvectors)
Revised: Yet it remains open to fully understand the relations between graph spatial information (i.e., symmetries) and graph spectral information (e.g., eigenvalues and eigenvectors)

Original: We hope that our approach will inspire future research on principled combinations of global and local methods for graph representation learning.
Revised: We hope that our approach will inspire future research on principled combinations of global and local methods for graph representation learning.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: We thank Rene Vidal for motivating this work, and Youngser Park for his valuable comments to this paper.
Revised: We thank Rene Vidal for motivating this work, and Youngser Park for his valuable comments to this paper.

Original: We also thank the anonymous reviewers for giving us constructive feedback.
Revised: We also thank the anonymous reviewers for giving us constructive feedback.

Original: SV is partially supported by ONR N00014-22-1-2126, NSF CISE 2212457, an AI2AI Amazon research award, and the NSF–Simons Research Collaboration on the Mathematical and Scientiﬁc Foundations of Deep
Revised: SV is partially supported by ONR N00014-22-1-2126, NSF CISE 2212457, an AI2AI Amazon research award, and the NSF–Simons Research Collaboration on the Mathematical and Scientiﬁc Foundations of Deep

Original: Learning (MoDL) (NSF DMS 2031985).
Revised: Learning (MoDL) (NSF DMS 2031985).

Original: A Proofs of PowerEmbed properties
Revised: A Proofs of PowerEmbed properties

Original: Proof of Proposition 4.1.
Revised: Proof of Proposition 4.1.

Original: Recall S ∈ R n × n and U ( t ) ∈ R n × k for all t ≥ 0 .
Revised: Recall S ∈ R n × n and U ( t ) ∈ R n × k for all t ≥ 0 .

Original: Since S ∈ { A, ¯ A, A rw } ,
Revised: Since S ∈ { A, ¯ A, A rw } ,

Original: S admits a spectral decomposition.
Revised: S admits a spectral decomposition.

Original: This is obvious for the symmetric matrices A, ¯ A .
Revised: This is obvious for the symmetric matrices A, ¯ A .

Original: For nonsymmetric A rw , if λ is an eigenvalue of ¯ A with eigenvector w , then λ is also an eigenvalue of A rw with eigenvector ˜ D − 0 .
Revised: For nonsymmetric A rw , if λ is an eigenvalue of ¯ A with eigenvector w , then λ is also an eigenvalue of A rw with eigenvector ˜ D − 0 .

Original: 5 w [76, Proposition.3.3].
Revised: 5 w [76, Proposition.3.3].

Original: Assume the initialization U 0 is full rank, and the k -th and ( k + 1) -th eigenvalue of S are distinct.
Revised: Assume the initialization U 0 is full rank, and the k -th and ( k + 1) -th eigenvalue of S are distinct.

Original: Following the argument from Appendix A in [61] we observe
Revised: Following the argument from Appendix A in [61] we observe

Original: [Equation]
Revised: [Equation]

Original: where U ( t ) ∈ R n × k .
Revised: where U ( t ) ∈ R n × k .

Original: Consider the spectral decomposition of S :
Revised: Consider the spectral decomposition of S :

Original: [Equation]
Revised: [Equation]

Original: where V 1 ∈ R n × k , V 2 ∈ R n × ( n − k ) , and Λ 1 = diag( λ 1 , .
Revised: where V 1 ∈ R n × k , V 2 ∈ R n × ( n − k ) , and Λ 1 = diag( λ 1 , .

Original: λ k ) , Λ 2 = diag( λ k +1 , . . .
Revised: λ k ) , Λ 2 = diag( λ k +1 , . . .

Original: , λ n ) and
Revised: , λ n ) and

Original: Let Φ( t )
Revised: Let Φ( t )

Original: =
Revised: =

Original: V (cid:62) 1 U ( t ) ∈
Revised: V (cid:62) 1 U ( t ) ∈

Original: R k × k and Ω( t ) =
Revised: R k × k and Ω( t ) =

Original: V (cid:62) 2 U ( t ) ∈
Revised: V (cid:62) 2 U ( t ) ∈

Original: R ( n − k ) × k for t ≥ 0 .
Revised: R ( n − k ) × k for t ≥ 0 .

Original: Multiplying [ V 1 V 2 ] (cid:62) on both sides of (9) and using (10), we have
Revised: Multiplying [ V 1 V 2 ] (cid:62) on both sides of (9) and using (10), we have

Original: [Equation]
Revised: [Equation]

Original: 11) where Z ( t ) =
Revised: 11) where Z ( t ) =

Original: { Φ( t ) (cid:62) Λ 21 Φ( t )
Revised: { Φ( t ) (cid:62) Λ 21 Φ( t )

Original: + Ω( t ) (cid:62)
Revised: + Ω( t ) (cid:62)

Original: Λ 22 Ω( t ) } − 1 .
Revised: Λ 22 Ω( t ) } − 1 .

Original: Since Φ(0)
Revised: Since Φ(0)

Original: =
Revised: =

Original: V (cid:62) 1 U (0) is full rank, there exists L such that Ω(0)
Revised: V (cid:62) 1 U (0) is full rank, there exists L such that Ω(0)

Original: =
Revised: =

Original: L Φ(0) .
Revised: L Φ(0) .

Original: A simple induction argument shows that
Revised: A simple induction argument shows that

Original: [Equation]
Revised: [Equation]

Original: Using the assumption that λ k > λ k +1 , (12) implies that lim t →∞ Ω( t ) → 0 .
Revised: Using the assumption that λ k > λ k +1 , (12) implies that lim t →∞ Ω( t ) → 0 .

Original: Therefore lim t →∞
Revised: Therefore lim t →∞

Original: Z
Revised: Z

Original: ( t ) =
Revised: ( t ) =

Original: { Φ( t ) (cid:62) Λ 21 Φ( t ) } − 1 .
Revised: { Φ( t ) (cid:62) Λ 21 Φ( t ) } − 1 .

Original: Then (11) implies
Revised: Then (11) implies

Original: [Equation]
Revised: [Equation]

Original: Thus, upon convergence at t =
Revised: Thus, upon convergence at t =

Original: L , Φ( L )Φ( L ) (cid:62) =
Revised: L , Φ( L )Φ( L ) (cid:62) =

Original: Λ − 1 1 which implies U ( L ) =
Revised: Λ − 1 1 which implies U ( L ) =

Original: V 1 W Λ − 0 .
Revised: V 1 W Λ − 0 .

Original: 5where W is a unitary matrix.
Revised: 5where W is a unitary matrix.

Original: This proves that Algorithm 1 returns the top- k eigenvectors of S up to orthogonal transformation in O ( k ) .
Revised: This proves that Algorithm 1 returns the top- k eigenvectors of S up to orthogonal transformation in O ( k ) .

Original: Proof of Proposition 4.2.
Revised: Proof of Proposition 4.2.

Original: Algorithm 1 is permutation equivariant.
Revised: Algorithm 1 is permutation equivariant.

Original: Namely, for all Π ∈ S n permutation matrix and all A ∈ R n × n and all X ∈ R n × k we have that
Revised: Namely, for all Π ∈ S n permutation matrix and all A ∈ R n × n and all X ∈ R n × k we have that

Original: PowerEmbed
Revised: PowerEmbed

Original: [Equation]
Revised: [Equation]

Original: It sufﬁces to observe that Π A Π (cid:62) Π X = Π
Revised: It sufﬁces to observe that Π A Π (cid:62) Π X = Π

Original: A X
Revised: A X

Original: and (Π A X ) (cid:62) (Π A X ) =
Revised: and (Π A X ) (cid:62) (Π A X ) =

Original: ( A X ) (cid:62) ( A X ) , making ˜ U
Revised: ( A X ) (cid:62) ( A X ) , making ˜ U

Original: ( t +1) (cid:62)
Revised: ( t +1) (cid:62)

Original: ˜ U
Revised: ˜ U

Original: ( t +1) permutation invariant, and therefore [ T u ( ˜ U ( t +1) (cid:62)
Revised: ( t +1) permutation invariant, and therefore [ T u ( ˜ U ( t +1) (cid:62)

Original: ˜ U
Revised: ˜ U

Original: ( t +1))]
Revised: ( t +1))]

Original: − 1 is also permutation invariant.
Revised: − 1 is also permutation invariant.

Original: Therefore at each step ˜ U ( t +1)(Π A Π (cid:62) , Π X ) =
Revised: Therefore at each step ˜ U ( t +1)(Π A Π (cid:62) , Π X ) =

Original: Π ˜ U
Revised: Π ˜ U

Original: ( t +1)
Revised: ( t +1)

Original: ( A, X ) , making PowerEmbed permutation equivariant.
Revised: ( A, X ) , making PowerEmbed permutation equivariant.

Original: The inception networks g θ (cid:48) ◦ MLPs are permutation equivariant because the same function is applied to all the nodes.
Revised: The inception networks g θ (cid:48) ◦ MLPs are permutation equivariant because the same function is applied to all the nodes.

Original: It implements the equivariance as weight sharing (similar to message-passing methods).
Revised: It implements the equivariance as weight sharing (similar to message-passing methods).

Original: B Experiment details B.1 Baselines
Revised: B Experiment details B.1 Baselines

Original: We perform PowerEmbed with different graph operators: A (indicated by “Power”), A rw (indicated by “Power(RW)”) and ¯ A (indicated by “Power(Lap)”).
Revised: We perform PowerEmbed with different graph operators: A (indicated by “Power”), A rw (indicated by “Power(RW)”) and ¯ A (indicated by “Power(Lap)”).

Original: For semi-supervised MPNNs including GCNII, GPR-GNN, we use hyperparameters that are are comparable with PowerEmbed : epochs = 100 , learning rate = 0 .
Revised: For semi-supervised MPNNs including GCNII, GPR-GNN, we use hyperparameters that are are comparable with PowerEmbed : epochs = 100 , learning rate = 0 .

Original: 01 , dropout rate = 0 .
Revised: 01 , dropout rate = 0 .

Original: 5 , weight decay = 0 , number of layers = 10 .
Revised: 5 , weight decay = 0 , number of layers = 10 .

Original: Speciﬁcally for GCNII, we use α = 0 .
Revised: Speciﬁcally for GCNII, we use α = 0 .

Original: We remark that JK-Concat [14] can be viewed as the semi-supervised version of SIGN [64] that uses learnable weights and nonlinearity in the message-passing layers.
Revised: We remark that JK-Concat [14] can be viewed as the semi-supervised version of SIGN [64] that uses learnable weights and nonlinearity in the message-passing layers.

Original: Thus, JK-Concat could produce layer-wise embeddings more adaptive to the inference task by learning end-to-end.
Revised: Thus, JK-Concat could produce layer-wise embeddings more adaptive to the inference task by learning end-to-end.

Original: Nonetheless, similar to other semi-supervised MPNNs, JK-Concat requires storing the graph at training and inference time, whereas unsupervised methods like PowerEmbed and SIGN ﬁrst embed the graph as Euclidean features, which allows fast training and inference with computational complexity independent of the graph topology [64].
Revised: Nonetheless, similar to other semi-supervised MPNNs, JK-Concat requires storing the graph at training and inference time, whereas unsupervised methods like PowerEmbed and SIGN ﬁrst embed the graph as Euclidean features, which allows fast training and inference with computational complexity independent of the graph topology [64].

Original: Due to its similarity to PowerEmbed and SIGN, we adopt the same input feature dimensionality reduction strategy (see details in Appendix B.3) and 10 -layer concatenation for better comparison.
Revised: Due to its similarity to PowerEmbed and SIGN, we adopt the same input feature dimensionality reduction strategy (see details in Appendix B.3) and 10 -layer concatenation for better comparison.

Original: B.2 Datasets
Revised: B.2 Datasets

Original: To evaluate PowerEmbed , we consider 5 dense graphs and 5 sparse graphs with different sizes and homophily ratios (see Tables 1, 2 for summary statistics).
Revised: To evaluate PowerEmbed , we consider 5 dense graphs and 5 sparse graphs with different sizes and homophily ratios (see Tables 1, 2 for summary statistics).

Original: Webpage networks.
Revised: Webpage networks.

Original: In these networks, nodes represent webpages and edges are hyperlinks between them.
Revised: In these networks, nodes represent webpages and edges are hyperlinks between them.

Original: Squirrel and Chameleon are created based on speciﬁc topics from Wikipedia [77], where node features are informative nouns in the Wikipedia pages and the node labels are created in [8] which reﬂect the average monthly trafﬁc of the webpage.
Revised: Squirrel and Chameleon are created based on speciﬁc topics from Wikipedia [77], where node features are informative nouns in the Wikipedia pages and the node labels are created in [8] which reﬂect the average monthly trafﬁc of the webpage.

Original: Wisconsin, Texas and Cornell are introduced in [8], where node features are bag-of-words representation of the webpages, and node labels are the webpage category.
Revised: Wisconsin, Texas and Cornell are introduced in [8], where node features are bag-of-words representation of the webpages, and node labels are the webpage category.

Original: All webpage networks we considered exhibit heterophily structure, where node with different class labels tend to be connected.
Revised: All webpage networks we considered exhibit heterophily structure, where node with different class labels tend to be connected.

Original: Co-purchase networks.
Revised: Co-purchase networks.

Original: Computers and Photo are Amazon co-purchase networks introduced in [78], where nodes represent goods, edges represent tendency of being bought together, node features are bag-of-words vector of product reviews, and node labels are product category.
Revised: Computers and Photo are Amazon co-purchase networks introduced in [78], where nodes represent goods, edges represent tendency of being bought together, node features are bag-of-words vector of product reviews, and node labels are product category.

Original: Citation networks.
Revised: Citation networks.

Original: Cora and Citeseer are standard citation network benchmarks [79] where nodes represent documents, edges represent citation links, node features are bag-of-words vector from the dictionary, and node labels are the document category.
Revised: Cora and Citeseer are standard citation network benchmarks [79] where nodes represent documents, edges represent citation links, node features are bag-of-words vector from the dictionary, and node labels are the document category.

Original: Co-authorship network.
Revised: Co-authorship network.

Original: Coauthor(CS) is introduced in [78] where nodes represent researchers, edges represent coauthorship, node features are bag-of-words vector from the paper keyword dictionary, and node labels are the research ﬁelds.
Revised: Coauthor(CS) is introduced in [78] where nodes represent researchers, edges represent coauthorship, node features are bag-of-words vector from the paper keyword dictionary, and node labels are the research ﬁelds.

Original: B.3 The Choice of the Embedding dimension k
Revised: B.3 The Choice of the Embedding dimension k

Original: PowerEmbed , similar to spectral embedding methods, crucially depends on the choice of embedding dimension k : choosing k too small introduces large bias while choosing k too large increases variance.
Revised: PowerEmbed , similar to spectral embedding methods, crucially depends on the choice of embedding dimension k : choosing k too small introduces large bias while choosing k too large increases variance.

Original: We refer the interested reader to [2, 80] for in-depth discussions.
Revised: We refer the interested reader to [2, 80] for in-depth discussions.

Original: In our experiments on real graphs, we choose k = 10 for small graphs including Wisconsin, Texas, and Cornell; k = 100 for other larger graphs.
Revised: In our experiments on real graphs, we choose k = 10 for small graphs including Wisconsin, Texas, and Cornell; k = 100 for other larger graphs.

Original: The same embedding dimension is used across all unsupervised methods (i.e., PowerEmbed , unnormalized counterparts, spectral methods) and JKConcat that take the top- k eigenvectors of the feature covariance matrix as input.
Revised: The same embedding dimension is used across all unsupervised methods (i.e., PowerEmbed , unnormalized counterparts, spectral methods) and JKConcat that take the top- k eigenvectors of the feature covariance matrix as input.

Original: The embedding dimension is chosen based on balancing the bias and variance as well as reducing memory complexity.
Revised: The embedding dimension is chosen based on balancing the bias and variance as well as reducing memory complexity.

Original: On the other hand, we follow the original experiment design in semi-supervised MPNNs including
Revised: On the other hand, we follow the original experiment design in semi-supervised MPNNs including

Original: GCNII and GPR-GNN, which use original node features and set k the same as the feature dimension.
Revised: GCNII and GPR-GNN, which use original node features and set k the same as the feature dimension.

Original: We remark that the optimal choice of k depends on each dataset and each embedding method, which is beyond the scope of current work but an interesting future research direction.
Revised: We remark that the optimal choice of k depends on each dataset and each embedding method, which is beyond the scope of current work but an interesting future research direction.

Original: C Ablation study C.1 Synthetic graphs
Revised: C Ablation study C.1 Synthetic graphs

Original: We report comprehensive simulation results on 2B-SBM models.
Revised: We report comprehensive simulation results on 2B-SBM models.

Original: Graph Laplacian operator ¯ A is used in all message-passing methods including PowerEmbed , SGC, GCN, GCNII, GPR-GNN.
Revised: Graph Laplacian operator ¯ A is used in all message-passing methods including PowerEmbed , SGC, GCN, GCNII, GPR-GNN.

Original: All experiments are done using n = 500 nodes and repeated for 30 random runs.
Revised: All experiments are done using n = 500 nodes and repeated for 30 random runs.

Original: Figure 5 supplements
Revised: Figure 5 supplements

Original: Figure 3 with other baselines, and Figure 6 shows a more ﬁne-grained picture as the graph density decreases.
Revised: Figure 3 with other baselines, and Figure 6 shows a more ﬁne-grained picture as the graph density decreases.

Original: We make the following observations.
Revised: We make the following observations.

Original: Performance in sparse graphs.
Revised: Performance in sparse graphs.

Original: In sparse graphs (e.g., Fig 5 (b), Fig 6), spectral methods and the last iterate of PowerEmbed perform poorly whereas PowerEmbed using all intermediate representations (including initial node features) performs well.
Revised: In sparse graphs (e.g., Fig 5 (b), Fig 6), spectral methods and the last iterate of PowerEmbed perform poorly whereas PowerEmbed using all intermediate representations (including initial node features) performs well.

Original: Shallow MPNNs achieve strong performance (e.g., ‘SGC-2”, “GCN-2”), including heterophilous graphs (Fig 5 (B)): as analyzed in [10], when nodes from the same class exhibit similar associativity pattern with other classes (as in the case of SBM graphs), heterophilous graphs are not necessarily hard to learn for shallows MPNNs.
Revised: Shallow MPNNs achieve strong performance (e.g., ‘SGC-2”, “GCN-2”), including heterophilous graphs (Fig 5 (B)): as analyzed in [10], when nodes from the same class exhibit similar associativity pattern with other classes (as in the case of SBM graphs), heterophilous graphs are not necessarily hard to learn for shallows MPNNs.

Original: Over-smoothing.
Revised: Over-smoothing.

Original: Fig 6 shows that Standard MPNNs (e.g., ‘SGC-10”, “GCN-10”) suffer from oversmoothing: increasing the number of layers degrades the performance.
Revised: Fig 6 shows that Standard MPNNs (e.g., ‘SGC-10”, “GCN-10”) suffer from oversmoothing: increasing the number of layers degrades the performance.

Original: Moreover, the denser the graph, the worse of the performance degradation (e.g., ‘SGC-5”, “GCN-5’ perform better in sparse graphs than dense graphs), a phenomenon discussed in [6, 44].
Revised: Moreover, the denser the graph, the worse of the performance degradation (e.g., ‘SGC-5”, “GCN-5’ perform better in sparse graphs than dense graphs), a phenomenon discussed in [6, 44].

Original: On the other hand, the last iterate of
Revised: On the other hand, the last iterate of

Original: PowerEmbed (right-most columns in Fig 5) prevents over-smoothing in dense graphs.
Revised: PowerEmbed (right-most columns in Fig 5) prevents over-smoothing in dense graphs.

Original: MPNNs that are speciﬁcally designed to avoid over-smoothing (e.g., GCNII, GPR-GNN) perform more stable as the number of layers increases, but suffer from high variance issues.
Revised: MPNNs that are speciﬁcally designed to avoid over-smoothing (e.g., GCNII, GPR-GNN) perform more stable as the number of layers increases, but suffer from high variance issues.

Original: PowerEmbed also prevents over-smoothing while achieving smaller variance.
Revised: PowerEmbed also prevents over-smoothing while achieving smaller variance.

Original: This illustrates the beneﬁts of expressing global spectral information over long-range spatial information under the SBM models, which we also empirically observe in some real-world graphs (see Table 1, Table 2).
Revised: This illustrates the beneﬁts of expressing global spectral information over long-range spatial information under the SBM models, which we also empirically observe in some real-world graphs (see Table 1, Table 2).

Original: , “GPR-GNN”), albeit with higher variance.
Revised: , “GPR-GNN”), albeit with higher variance.

Original: C.2 Real-world graphs
Revised: C.2 Real-world graphs

Original: We compare the performance of PowerEmbed on real benchmarks using (1) all iterations; (2) last iteration; (3) the input feature and the last iteration.
Revised: We compare the performance of PowerEmbed on real benchmarks using (1) all iterations; (2) last iteration; (3) the input feature and the last iteration.

Original: Figure 7 shows that on dense graphs, even without layer-wise concatenation, PowerEmbed outperforms its unnormalized counterparts when the number of layers increases.
Revised: Figure 7 shows that on dense graphs, even without layer-wise concatenation, PowerEmbed outperforms its unnormalized counterparts when the number of layers increases.

Original: This empirically demonstrates its ability to avoid over-smoothing.
Revised: This empirically demonstrates its ability to avoid over-smoothing.